robocop
by pjkelly
Robocop is a simple Rack middleware that inserts the X-Robots-Tag into the headers of all your responses
AI summary
Crawler meta tag
A middleware that adds a meta tag to HTTP responses to instruct search engines on how to crawl the content.
- stars
- 3
- forks
- 0
- watching
- 1
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Crawler
A flexible web crawler that follows robots.txt policies and crawl delays.
Meta tag generator
Generates meta tags based on page content using Rack Middleware
Meta tag generator
A Ruby on Rails plugin for generating and managing meta tags.
Crawler
A concurrent web crawler written in Go that allows flexible and polite crawling of websites.
Meta tag generator
Generates standardized metadata tags for search engines and social networks to improve website indexing and display.
Web scraper
A Ruby gem for web scraping and extracting metadata from web pages.
Web Crawler
A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages.
Crawler library
A Rust-based library for building and managing cryptocurrency crawlers
Crawler
An OSINT bot that crawls pastebin sites to search for sensitive data leaks
Document Extractor
Extracts metadata from public documents found on websites, useful for brute-force attacks.
Web crawler
A Ruby-based tool for web crawling and data extraction, aiming to be a replacement for paid software in the SEO space.
Tag manager
A Rails gem for managing tags with PostgreSQL array columns in a flexible and efficient way
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
crawler
Performs web page crawling at high performance.
jmg/crawley188
Crawler
A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions robocop.
