scrapy-cluster
by istresearch
This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.
AI summary
Crawler cluster
A distributed scraping framework that scales crawling and prioritizes sites, utilizing Redis and Kafka for coordination.
- stars
- 1.2K
- forks
- 323
- watching
- 108
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
scrapy/scrapely1.9K
HTML parser
A pure-python library for extracting structured data from HTML pages.
dyweb/scrala113
Web crawler framework
A web crawling framework written in Scala that allows users to define the start URL and parse response from it
Crawler meta tag
A middleware that adds a meta tag to HTTP responses to instruct search engines on how to crawl the content.
Redis cluster client
A client library for managing Redis clusters using Lua scripts in an OpenResty configuration.
Map Clustering Library
A map annotation clustering library that efficiently groups and displays geographic pins on an iOS map view.
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
Crawler
An OSINT bot that crawls pastebin sites to search for sensitive data leaks
Graph clustering library
A PyTorch extension library providing optimized graph cluster algorithms
Spider
A framework for extracting structured data from websites
Spider manager
An app that allows you to manage Scrapy spiders through a Django admin interface.
howie6879/ruia1.8K
Crawler framework
An async web scraping micro-framework built with asyncio and aiohttp to simplify URL crawling
Scraper
A tool to scrape and store hierarchical data about GitHub organizations, users, or repositories.
Document scraper
A tool to fetch information about public Google documents from various services
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
tidyverse/rvest1.5K
Web scraper
A package for extracting data from web pages using HTML parsing and CSS/XPath selectors.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions scrapy-cluster.
