urlgrab

hu17889/go_spider1.8K

Crawler

A modular, concurrent web crawler framework written in Go.

jmg/crawley188

Crawler

A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.

antchfx/antch261

Web crawler

A framework for building fast and efficient web crawlers and scrapers in Go.

s0rg/crawley268

Web URL extractor

A utility for systematically extracting URLs from web pages and printing them to the console.

puerkitobio/gocrawl2K

Crawler

A concurrent web crawler written in Go that allows flexible and polite crawling of websites.

amoilanen/js-crawler254

Web crawler

A Node.js module for crawling web sites and scraping their content

dwisiswant0/galer255

URL extractor

A tool to extract URLs from HTML attributes by crawling in and evaluating JavaScript

internetarchive/brozzler678

Web Crawler

A distributed web crawler that fetches and extracts links from websites using a real browser.

brendonboshell/supercrawler380

Web Crawler

A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.

wspl/creeper780

Crawler framework

A framework for building cross-platform web crawlers using Go

stewartmckee/cobweb226

Web Crawler

A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner

rivermont/spidy340

Web crawler

A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling

vida-nyu/ache459

Web crawler

A web crawler designed to efficiently collect and prioritize relevant content from the web

twiny/spidy151

Domain crawler

Tools to crawl websites and collect domain names with availability status

fmpwizard/owlcrawler55

Crawler

A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus.