spiderable-middleware
crawler proxy
intercepts requests from web crawlers and proxies them to a prerendering service for rendering HTML
🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites based on top of front-end JavaScript frameworks
39 stars
3 watching
4 forks
Language: JavaScript
last commit: 4 months ago
Linked from 1 awesome list
crawlermeteormeteor-packagemiddlewarenodejsnpmnpm-packageseoseo-optimizationspiderable
urigo/awesome-meteorRelated projects:
| Repository | Description | Stars |
|---|---|---|
 dyweb/scrala | A web crawling framework written in Scala that allows users to define the start URL and parse response from it | 113 |
 feng19/spider_man | A high-level web crawling and scraping framework for Elixir. | 23 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,827 |
 genkio/spider-less | A web spider service built on AWS Lambda and other cloud services to monitor websites for specific content changes. | 185 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 mvdbos/php-spider | A flexible PHP web crawler with configurable traversal algorithms and filters. | 1,336 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 elixir-crawly/crawly | A framework for extracting structured data from websites | 994 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 809 |
 twiny/spidy | Tools to crawl websites and collect domain names with availability status | 151 |
 3nock/spidersuite | A cross-platform web spider/crawler tool for analyzing and mapping attack surfaces | 614 |
 joncanning/skyscraper | A framework for building asynchronous web scrapers and crawlers using async/await and Reactive Extensions. | 59 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 188 |
 tjatse/node-readability | Automates web page scraping and text extraction to make any webpage readable | 343 |
 amoilanen/js-crawler | A Node.js module for crawling web sites and scraping their content | 254 |