php-spider
Web Crawler
A flexible PHP web crawler with configurable traversal algorithms and filters.
A configurable and extensible PHP web spider
1k stars
87 watching
232 forks
Language: PHP
last commit: 9 months ago
Linked from 2 awesome lists
ziadoz/awesome-php
brucedone/awesome-crawlerRelated projects:
| Repository | Description | Stars |
|---|---|---|
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 spider-rs/spider | A tool for web data extraction and processing using Rust | 1,234 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,827 |
 spider/spider | A flexible graph database abstraction for PHP | 23 |
 rivermont/spidy | A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling | 340 |
 hightman/pspider | A parallel web crawler framework built using PHP and MySQLi | 266 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 amoilanen/js-crawler | A Node.js module for crawling web sites and scraping their content | 254 |
 3nock/spidersuite | A cross-platform web spider/crawler tool for analyzing and mapping attack surfaces | 614 |
 manning23/mspider | A Python-based tool for web crawling and data collection from various websites | 348 |
 crawlzone/crawlzone | A PHP framework for asynchronous internet crawling and web scraping | 78 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 809 |
 fmpwizard/owlcrawler | A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus. | 55 |
 feng19/spider_man | A high-level web crawling and scraping framework for Elixir. | 23 |
 holgerd77/django-dynamic-scraper | An app that allows you to manage Scrapy spiders through a Django admin interface. | 1,155 |