rubyretriever
by joenorton
Asynchronous Web Crawler & Scraper
AI summary
Web crawler
A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space.
- stars
- 143
- forks
- 26
- watching
- 7
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
Web scraper
A Ruby gem for web scraping and extracting metadata from web pages.
spider-rs/spider1.2K
Web scraper
A tool for web data extraction and processing using Rust
Crawler
A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
felipecsl/wombat1.3K
Web scraper library
A Ruby-based web crawler and data extraction tool with an elegant DSL.
crawler
Performs web page crawling at high performance.
Link crawler
A tool to crawl websites by exploring links recursively with support for JavaScript rendering.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Web crawler
A Node.js module for crawling web sites and scraping their content
Crawler meta tag
A middleware that adds a meta tag to HTTP responses to instruct search engines on how to crawl the content.
Crawler
A high-performance web crawling and scraping solution with customizable settings and worker pooling.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions rubyretriever.
