MSpider
Web Crawler
A Python-based tool for web crawling and data collection from various websites
Spider
348 stars
55 watching
192 forks
Language: Python
last commit: over 2 years ago
Linked from 1 awesome list
mspider
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
 rivermont/spidy | A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling | 340 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 186 |
 xianhu/pspider | A Python web crawler framework with support for multi-threading and proxy usage. | 1,827 |
 hightman/pspider | A parallel web crawler framework built using PHP and MySQLi | 266 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 medialab/minet | A command line tool and Python library for extracting data from various web sources. | 286 |
 mvdbos/php-spider | A flexible PHP web crawler with configurable traversal algorithms and filters. | 1,332 |
 chenjiandongx/github-spider | A Python-based web crawler for scraping Github user and repository data. | 264 |
 feng19/spider_man | A high-level web crawling and scraping framework for Elixir. | 23 |
 elliotgao2/gain | A Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites. | 2,035 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 806 |
 spider-rs/spider | A web crawler and scraper built on top of Rust, designed to extract data from the web in a flexible and configurable manner. | 1,140 |
 fmpwizard/owlcrawler | A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus. | 55 |
 3nock/spidersuite | A cross-platform web spider/crawler tool for analyzing and mapping attack surfaces | 601 |
 holgerd77/django-dynamic-scraper | An app that allows you to manage Scrapy spiders through a Django admin interface. | 1,153 |