ebot

feng19/spider_man23

Crawler library

A high-level web crawling and scraping framework for Elixir.

puerkitobio/fetchbot787

Crawler

A flexible web crawler that follows robots.txt policies and crawl delays.

brendonboshell/supercrawler380

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

elixir-crawly/crawly994

Spider

A framework for extracting structured data from websites

fredwu/crawler945

Crawler

A high-performance web crawling and scraping solution with customizable settings and worker pooling.

elliotgao2/gain2K

crawler

A Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites.

postmodern/spidr809

Crawler library

A Ruby web crawling library that provides flexible and customizable methods to crawl websites

rndinfosecguy/scavenger634

Crawler

An OSINT bot that crawls pastebin sites to search for sensitive data leaks

puerkitobio/gocrawl2K

Crawler

A concurrent web crawler written in Go that allows flexible and polite crawling of websites.

hu17889/go_spider1.8K

Crawler

A modular, concurrent web crawler framework written in Go.

felipecsl/wombat1.3K

Web scraper library

A Ruby-based web crawler and data extraction tool with an elegant DSL.

fmpwizard/owlcrawler55

Crawler

A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus.

turnersoftware/infinitycrawler248

Crawler library

A web crawling library for .NET that allows customizable crawling and throttling of websites.

vida-nyu/ache459

Web crawler

A web crawler designed to efficiently collect and prioritize relevant content from the web

joenorton/rubyretriever143

Web crawler

A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space.