hakrawler
Crawler
A tool for automatically discovering and crawling web application endpoints and assets
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
5k stars
61 watching
497 forks
Language: Go
last commit: 10 months ago
Linked from 2 awesome lists
bugbountycrawlinghackingosintpentestingreconreconnaissance
brucedone/awesome-crawler
vavkamil/awesome-bugbounty-toolsRelated projects:
| Repository | Description | Stars |
|---|---|---|
 jaeles-project/gospider | A tool for web crawling and exploitation written in Go. | 2,578 |
 iamstoxe/urlgrab | A tool to crawl websites by exploring links recursively with support for JavaScript rendering. | 330 |
 s0rg/crawley | A utility for systematically extracting URLs from web pages and printing them to the console. | 263 |
 geziyor/geziyor | A fast and flexible web crawling and scraping framework for extracting structured data from websites. | 2,629 |
| hakluke/haktrails | A Golang client for querying SecurityTrails API data to gather information on domains and their associated networks. | 539 |
 yasserg/crawler4j | A Java-based web crawler for extracting and processing web page content | 4,555 |
 gocolly/colly | A framework for extracting structured data from websites in a fast and elegant way | 23,317 |
 s0md3v/photon | A fast and flexible web crawler designed to gather information from the internet | 11,067 |
 spatie/crawler | A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently. | 2,537 |
 yujiosaka/headless-chrome-crawler | A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites | 5,527 |
 apify/crawlee | A tool for building reliable web scraping and browser automation pipelines in Node.js. | 15,604 |
 unclecode/crawl4ai | A tool for web crawling and data extraction, designed to work with large language models. | 16,180 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 378 |
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,038 |
 code4craft/webmagic | A scalable framework for building web crawlers in Java. | 11,432 |