creeper
Crawler framework
A framework for building cross-platform web crawlers using Go
Creeper - The Next Generation Crawler Framework (Go)
780 stars
47 watching
57 forks
Language: Go
last commit: over 9 years agoLinked from 1 awesome list
crawlercross-platformframeworkgolanglanguagescriptspider
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,827 |
 antchfx/antch | A framework for building fast and efficient web crawlers and scrapers in Go. | 261 |
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,036 |
 zhegexiaohuozi/seimicrawler | A distributed crawler framework that simplifies the process of building crawlers using Spring Boot and Redis | 1,980 |
 dyweb/scrala | A web crawling framework written in Scala that allows users to define the start URL and parse response from it | 113 |
 fmpwizard/owlcrawler | A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus. | 55 |
 codesofun/web-bee | A Java framework for building web-based crawlers with features like distributed crawling and proxy support. | 189 |
 turnersoftware/infinitycrawler | A web crawling library for .NET that allows customizable crawling and throttling of websites. | 248 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 188 |
 iamstoxe/urlgrab | A tool to crawl websites by exploring links recursively with support for JavaScript rendering. | 331 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 809 |
 crawlzone/crawlzone | A PHP framework for asynchronous internet crawling and web scraping | 78 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 feng19/spider_man | A high-level web crawling and scraping framework for Elixir. | 23 |
 archiveteam/grab-site | A web crawler designed to backup websites by recursively crawling and writing WARC files. | 1,406 |