supercrawler
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
A web crawler. Supercrawler automatically crawls websites. Define custom handlers to parse content. Obeys robots.txt, rate limits and concurrency limits.
378 stars
11 watching
62 forks
Language: JavaScript
last commit: almost 2 years ago
Linked from 1 awesome list
crawlerdistributed-crawlerrobotsitemapweb-crawler
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
 cocrawler/cocrawler | A versatile web crawler built with modern tools and concurrency to handle various crawl tasks | 187 |
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,038 |
 turnersoftware/infinitycrawler | A web crawling library for .NET that allows customizable crawling and throttling of websites. | 248 |
 apache/incubator-stormcrawler | A collection of resources for building web crawlers on Apache Storm using Java | 891 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 internetarchive/brozzler | A distributed web crawler that fetches and extracts links from websites using a real browser. | 671 |
 rivermont/spidy | A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling | 340 |
| puerkitobio/fetchbot | A flexible web crawler that follows robots.txt policies and crawl delays. | 786 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 806 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,826 |
 spider-rs/spider | A web crawler and scraper built on top of Rust, designed to extract data from the web in a flexible and configurable manner. | 1,140 |
 c-sto/recursebuster | A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content | 242 |
 fmpwizard/owlcrawler | A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus. | 55 |
 webrecorder/browsertrix-crawler | A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner. | 652 |
 bplawler/crawler | A Scala-based DSL for programmatically accessing and interacting with web pages | 148 |