go_spider

puerkitobio/gocrawl2K

Crawler

A concurrent web crawler written in Go that allows flexible and polite crawling of websites.

wspl/creeper780

Crawler framework

A framework for building cross-platform web crawlers using Go

antchfx/antch261

Web crawler

A framework for building fast and efficient web crawlers and scrapers in Go.

feng19/spider_man23

Crawler library

A high-level web crawling and scraping framework for Elixir.

chenjiandongx/github-spider264

Crawler

A Python-based web crawler for scraping Github user and repository data.

iamstoxe/urlgrab331

Link crawler

A tool to crawl websites by exploring links recursively with support for JavaScript rendering.

fmpwizard/owlcrawler55

Crawler

A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus.

brendonboshell/supercrawler380

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

postmodern/spidr809

Crawler library

A Ruby web crawling library that provides flexible and customizable methods to crawl websites

elixir-crawly/crawly994

Spider

A framework for extracting structured data from websites

dyweb/scrala113

Web crawler framework

A web crawling framework written in Scala that allows users to define the start URL and parse response from it

jmg/crawley188

Crawler

A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.

puerkitobio/fetchbot787

Crawler

A flexible web crawler that follows robots.txt policies and crawl delays.

3nock/spidersuite614

Web Spider Crawler

A cross-platform web spider/crawler tool for analyzing and mapping attack surfaces

mvdbos/php-spider1.3K

Web Crawler

A flexible PHP web crawler with configurable traversal algorithms and filters.