sparkler
Web Crawler
A high-performance web crawler built on Apache Spark that fetches and analyzes web resources in real-time.
Spark-Crawler: Apache Nutch-like crawler that runs on Apache Spark.
410 stars
45 watching
143 forks
Language: Java
last commit: over 1 year ago
Linked from 1 awesome list
big-datadistributed-systemsinformation-retrievalnutchsearchsearch-enginesolrsparktikaweb-crawler
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
 internetarchive/sparkling | A data processing library built on top of Apache Spark to handle temporal web data | 11 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 apache/incubator-stormcrawler | A collection of resources for building web crawlers on Apache Storm using Java | 891 |
 vida-nyu/ache | A web crawler designed to efficiently collect and prioritize relevant content from the web | 454 |
| internetarchive/brozzler | A distributed web crawler that fetches and extracts links from websites using a real browser. | 671 |
| apache/spark | An analytics engine designed to handle large-scale data processing and analysis | 39,916 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 378 |
 archiveteam/grab-site | A web crawler designed to backup websites by recursively crawling and writing WARC files. | 1,398 |
 tweag/sparkle | A tool for creating resilient, scalable analytics applications with Haskell on top of Apache Spark | 447 |
 hightman/pspider | A parallel web crawler framework built using PHP and MySQLi | 266 |
 c-sto/recursebuster | A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content | 242 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 806 |
 a11ywatch/crawler | Performs web page crawling at high performance. | 49 |
 webrecorder/browsertrix-crawler | A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner. | 652 |
 spider-rs/spider | A web crawler and scraper built on top of Rust, designed to extract data from the web in a flexible and configurable manner. | 1,140 |