gain
by elliotgao2
Web crawling framework based on asyncio.
AI summary
crawler
A Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites.
- stars
- 2K
- forks
- 208
- watching
- 75
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
howie6879/ruia1.8K
Crawler framework
An async web scraping micro-framework built with asyncio and aiohttp to simplify URL crawling
jmg/crawley188
Crawler
A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.
Crawler
A Python-based web crawler for scraping Github user and repository data.
Crawler library
A high-level web crawling and scraping framework for Elixir.
xianhu/pspider1.8K
Web Crawler Framework
A Python web crawler framework with support for multi-threading and proxy usage.
Crawler
A flexible web crawler that follows robots.txt policies and crawl delays.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Crawler
A modular, concurrent web crawler framework written in Go.
crawler
Performs web page crawling at high performance.
Web Crawler Framework
A minimalist web crawler framework built on top of miners and structure-based data extraction
Crawler
A high-performance web crawling and scraping solution with customizable settings and worker pooling.
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
Crawler
A concurrent web crawler written in Go that allows flexible and polite crawling of websites.
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
qinxuye/cola1.5K
Crawler library
A high-level framework for building distributed data extractors from web pages
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions gain.
