robocop
Crawler meta tag
A middleware that adds a meta tag to HTTP responses to instruct search engines on how to crawl the content.
Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses
3 stars
1 watching
0 forks
Language: Ruby
last commit: almost 12 years ago
Linked from 1 awesome list
coopermaa/awesome-rackRelated projects:
| Repository | Description | Stars |
|---|---|---|
 puerkitobio/fetchbot | A flexible web crawler that follows robots.txt policies and crawl delays. | 787 |
 xavriley/rack-seo | Generates meta tags based on page content using Rack Middleware | 2 |
 lassebunk/metamagic | A Ruby on Rails plugin for generating and managing meta tags. | 440 |
| puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,036 |
 jekyll/jekyll-seo-tag | Generates standardized metadata tags for search engines and social networks to improve website indexing and display. | 1,663 |
 jaimeiniesta/metainspector | A Ruby gem for web scraping and extracting metadata from web pages. | 1,038 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 crypto-crawler/crypto-crawler-rs | A Rust-based library for building and managing cryptocurrency crawlers | 235 |
 rndinfosecguy/scavenger | An OSINT bot that crawls pastebin sites to search for sensitive data leaks | 631 |
 laramies/metagoofil | Extracts metadata from public documents found on websites, useful for brute-force attacks. | 1,050 |
 joenorton/rubyretriever | A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space. | 143 |
 jetrockets/metka | A Rails gem for managing tags with PostgreSQL array columns in a flexible and efficient way | 53 |
 postmodern/spidr | A Ruby web crawling library that provides flexible and customizable methods to crawl websites | 809 |
 a11ywatch/crawler | Performs web page crawling at high performance. | 51 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 188 |