spidr
by postmodern
A versatile Ruby web spidering library that can spider a site, multiple domains, certain links or infinitely. Spidr is designed to be fast and easy to use.
AI summary
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
- stars
- 809
- forks
- 106
- watching
- 28
- awesome lists
- 3
Similar projects
Found by comparing what the projects do, not just their names.
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
spider-rs/spider1.2K
Web scraper
A tool for web data extraction and processing using Rust
felipecsl/wombat1.3K
Web scraper library
A Ruby-based web crawler and data extraction tool with an elegant DSL.
twiny/spidy151
Domain crawler
Tools to crawl websites and collect domain names with availability status
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
Crawler library
A high-level web crawling and scraping framework for Elixir.
Crawler
A modular, concurrent web crawler framework written in Go.
Web crawler
A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space.
Crawler library
A Rust-based library for building and managing cryptocurrency crawlers
Web scraper
A Scala-based DSL for programmatically accessing and interacting with web pages
dyweb/scrala113
Web crawler framework
A web crawling framework written in Scala that allows users to define the start URL and parse response from it
Crawler library
A web crawling library for .NET that allows customizable crawling and throttling of websites.
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
Spider manager
An app that allows you to manage Scrapy spiders through a Django admin interface.
Web scraper
A Ruby gem for web scraping and extracting metadata from web pages.
Featured in 3 awesome lists
Each link jumps to the spot where the list mentions spidr.
