distribute_crawler
by gnemoug
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现
AI summary
Crawler framework
A distributed web crawler framework using Scrapy, Redis, MongoDB, and Graphite for efficient crawling and data storage.
- stars
- 3.2K
- forks
- 1.6K
- watching
- 367
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
mongodb/mongo26.5K
NoSQL database
A high-performance, NoSQL document-oriented database system written in C++
cnodejs/nodeclub9.3K
Community forum
A Node.js-based community platform with MongoDB and Redis integration
MongoDB mapper
An Object-Document Mapper framework for interacting with MongoDB databases in Ruby.
Data pipeline
A MongoDB pipeline extension for Scrapy spiders that enables real-time data insertion and buffering options.
Document soft deletion
Provides a 'soft delete' functionality for Mongoid documents
MongoDB Cluster
A Docker Compose setup for a sharded MongoDB cluster with replication.
Sharder
A Redis-based sharding solution for Phalcon applications
Market feeder
Automates real-time market data retrieval and storage from Huobi exchange, publishing updates to Redis for use in backtesting and analysis.
MongoDB cleaner
A Ruby gem that provides a clean and efficient way to delete data from MongoDB databases.
MongoDB driver
A MongoDB driver written in Crystal that provides a high-performance interface to the MongoDB database
nkrode/redislive3.1K
Redis viewer
An application that visualizes Redis instances and analyzes query patterns and spikes.
DB seeder
Tools and libraries for importing data into MongoDB databases
Data generator
Generates randomized data for testing MongoDB databases
Mongo DB mapper
A Python library providing an object-document mapper for MongoDB with support for synchronous and asynchronous operations.
Hash extractor
Extracts hashes from MongoDB databases in SCRAM-SHA-1 or SCRAM-SHA-256 format for use with hashcat
Featured in 1 awesome list
Each link jumps to the spot where the list mentions distribute_crawler.
