php-spider

Web Crawler

A flexible PHP web crawler with configurable traversal algorithms and filters.

A configurable and extensible PHP web spider

1k stars

87 watching

232 forks

Language: PHP

last commit: over 2 years ago

Linked from 2 awesome lists

Backlinks from these awesome lists:

Related projects:

Repository	Description	Stars
brendonboshell/supercrawler	A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.	380
spider-rs/spider	A tool for web data extraction and processing using Rust	1,234
hu17889/go_spider	A modular, concurrent web crawler framework written in Go.	1,827
spider/spider	A flexible graph database abstraction for PHP	23
rivermont/spidy	A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling	340
hightman/pspider	A parallel web crawler framework built using PHP and MySQLi	266
stewartmckee/cobweb	A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner	226
amoilanen/js-crawler	A Node.js module for crawling web sites and scraping their content	254
3nock/spidersuite	A cross-platform web spider/crawler tool for analyzing and mapping attack surfaces	614
manning23/mspider	A Python-based tool for web crawling and data collection from various websites	348
crawlzone/crawlzone	A PHP framework for asynchronous internet crawling and web scraping	78
postmodern/spidr	A Ruby web crawling library that provides flexible and customizable methods to crawl websites	809
fmpwizard/owlcrawler	A distributed web crawler that coordinates crawling tasks across multiple worker processes using a message bus.	55
feng19/spider_man	A high-level web crawling and scraping framework for Elixir.	23
holgerd77/django-dynamic-scraper	An app that allows you to manage Scrapy spiders through a Django admin interface.	1,155