cobweb

rivermont/spidy340

Web crawler

A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling

brendonboshell/supercrawler380

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

cocrawler/cocrawler188

Web Crawler

A versatile web crawler built with modern tools and concurrency to handle various crawl tasks

apache/incubator-stormcrawler895

Crawler framework

A scalable and versatile web crawling framework based on Apache Storm

jmg/crawley188

Crawler

A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.

webrecorder/browsertrix-crawler677

Crawler

A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.

hominee/dyer135

Web crawler

A fast and flexible web crawling tool with features like asynchronous I/O and event-driven design.

uscdatascience/sparkler411

Web Crawler

A high-performance web crawler built on Apache Spark that fetches and analyzes web resources in real-time.

internetarchive/brozzler678

Web Crawler

A distributed web crawler that fetches and extracts links from websites using a real browser.

amoilanen/js-crawler254

Web crawler

A Node.js module for crawling web sites and scraping their content

vida-nyu/ache459

Web crawler

A web crawler designed to efficiently collect and prioritize relevant content from the web

archiveteam/grab-site1.4K

Web crawler

A web crawler designed to backup websites by recursively crawling and writing WARC files.

dyweb/scrala113

Web crawler framework

A web crawling framework written in Scala that allows users to define the start URL and parse response from it

twiny/spidy151

Domain crawler

Tools to crawl websites and collect domain names with availability status

c-sto/recursebuster243

Web crawler

A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content