incubator-stormcrawler
by apache
A scalable, mature and versatile web crawler based on Apache Storm
AI summary
Crawler framework
A scalable and versatile web crawling framework based on Apache Storm
- stars
- 895
- forks
- 260
- watching
- 66
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
Web Crawler
A high-performance web crawler built on Apache Spark that fetches and analyzes web resources in real-time.
Crawler
A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Crawler framework
A distributed crawler framework that simplifies the process of building crawlers using Spring Boot and Redis
Web crawler
A web crawler designed to efficiently collect and prioritize relevant content from the web
Crawler
A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus.
Web crawler
A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content
Crawler framework
A Java framework for building web-based crawlers with features like distributed crawling and proxy support.
Crawler
A Web crawler that creates custom archives in WARC/CDX format
Web crawler
A Node.js module for crawling web sites and scraping their content
Crawler library
A web crawling library for .NET that allows customizable crawling and throttling of websites.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions incubator-stormcrawler.
