gocrawl
Crawler
A concurrent web crawler written in Go that allows flexible and polite crawling of websites.
Polite, slim and concurrent web crawler.
2k stars
108 watching
193 forks
Language: Go
last commit: over 5 years agoLinked from 1 awesome list
crawlerrobots-txt
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
| puerkitobio/fetchbot | A flexible web crawler that follows robots.txt policies and crawl delays. | 787 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,827 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 fmpwizard/owlcrawler | A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus. | 55 |
 c-sto/recursebuster | A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content | 243 |
 cocrawler/cocrawler | A versatile web crawler built with modern tools and concurrency to handle various crawl tasks | 188 |
 wspl/creeper | A framework for building cross-platform web crawlers using Go | 780 |
 turnersoftware/infinitycrawler | A web crawling library for .NET that allows customizable crawling and throttling of websites. | 248 |
 chenjiandongx/github-spider | A Python-based web crawler for scraping Github user and repository data. | 264 |
 antchfx/antch | A framework for building fast and efficient web crawlers and scrapers in Go. | 261 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 188 |
 internetarchive/brozzler | A distributed web crawler that fetches and extracts links from websites using a real browser. | 678 |
 iamstoxe/urlgrab | A tool to crawl websites by exploring links recursively with support for JavaScript rendering. | 331 |
 s0rg/crawley | A utility for systematically extracting URLs from web pages and printing them to the console. | 268 |
 fredwu/crawler | A high-performance web crawling and scraping solution with customizable settings and worker pooling. | 945 |