spidy
by rivermont
The simple, easy to use command line web crawler.
AI summary
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
- stars
- 340
- forks
- 69
- watching
- 23
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
jmg/crawley188
Crawler
A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.
Web Crawler
A Python-based tool for web crawling and data collection from various websites
twiny/spidy151
Domain crawler
Tools to crawl websites and collect domain names with availability status
spider-rs/spider1.2K
Web scraper
A tool for web data extraction and processing using Rust
Crawler
A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.
Web Crawler
A flexible PHP web crawler with configurable traversal algorithms and filters.
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
s0rg/crawley268
Web URL extractor
A utility for systematically extracting URLs from web pages and printing them to the console.
Web crawler
A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
hominee/dyer135
Web crawler
A fast and flexible web crawling tool with features like asynchronous I/O and event-driven design.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions spidy.
