colly
Website scraper
A framework for extracting structured data from websites in a fast and elegant way
Elegant Scraper and Crawler Framework for Golang
23k stars
334 watching
2k forks
Language: Go
last commit: 4 months ago
Linked from 3 awesome lists
crawlercrawlingframeworkgogolangscraperscrapingspider
avelino/awesome-go
brucedone/awesome-crawler
xwjdsh/awesome-go-extraRelated projects:
| Repository | Description | Stars |
|---|---|---|
 geziyor/geziyor | A fast and flexible web crawling and scraping framework for extracting structured data from websites. | 2,629 |
 jaeles-project/gospider | A tool for web crawling and exploitation written in Go. | 2,578 |
 unclecode/crawl4ai | A tool for web crawling and data extraction, designed to work with large language models. | 16,180 |
 gohugoio/hugo | A fast and flexible tool for generating static websites with built-in support for various content formats. | 75,938 |
 yujiosaka/headless-chrome-crawler | A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites | 5,527 |
 apify/crawlee | A tool for building reliable web scraping and browser automation pipelines in Node.js. | 15,740 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,826 |
 hakluke/hakrawler | A tool for automatically discovering and crawling web application endpoints and assets | 4,502 |
 ionicabizau/scrape-it | A Node.js library and CLI tool for automating web page scraping and parsing | 4,012 |
 s0md3v/photon | A fast and flexible web crawler designed to gather information from the internet | 11,067 |
 needmorecowbell/giggity | A tool to scrape and store hierarchical data about GitHub organizations, users, or repositories. | 126 |
 golangci/golangci-lint | A tool that runs multiple Go linters in parallel to check code quality and catch errors | 15,737 |
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,038 |
 spatie/crawler | A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently. | 2,537 |
 my8100/scrapydweb | A web application for managing Scrapyd clusters, analyzing Scrapy logs, and visualizing results. | 3,161 |