recursebuster
Web crawler
A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content
rapid content discovery tool for recursively querying webservers, handy in pentesting and web application assessments
242 stars
9 watching
33 forks
Language: Go
last commit: about 5 years ago
Linked from 2 awesome lists
content-discoverygobusterrecursive
enaqx/awesome-pentest
vavkamil/awesome-bugbounty-toolsRelated projects:
| Repository | Description | Stars |
|---|---|---|
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,038 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 378 |
 hu17889/go_spider | A modular, concurrent web crawler framework written in Go. | 1,826 |
 webrecorder/browsertrix-crawler | A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner. | 652 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 norconex/crawlers | A suite of tools for gathering and processing data from the web and file systems | 183 |
 helgeho/web2warc | A Web crawler that creates custom archives in WARC/CDX format | 24 |
 internetarchive/brozzler | A distributed web crawler that fetches and extracts links from websites using a real browser. | 671 |
| puerkitobio/fetchbot | A flexible web crawler that follows robots.txt policies and crawl delays. | 786 |
 s0rg/crawley | A utility for systematically extracting URLs from web pages and printing them to the console. | 263 |
 apache/incubator-stormcrawler | A collection of resources for building web crawlers on Apache Storm using Java | 891 |
 uscdatascience/sparkler | A high-performance web crawler built on Apache Spark that fetches and analyzes web resources in real-time. | 410 |
 cocrawler/cocrawler | A versatile web crawler built with modern tools and concurrency to handle various crawl tasks | 187 |
 archiveteam/grab-site | A web crawler designed to backup websites by recursively crawling and writing WARC files. | 1,398 |
 vida-nyu/ache | A web crawler designed to efficiently collect and prioritize relevant content from the web | 454 |