hakrawler
Crawler
A tool for automatically discovering and crawling web application endpoints and assets
Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application
5k stars
63 watching
498 forks
Language: Go
last commit: almost 2 years ago
Linked from 2 awesome lists
bugbountycrawlinghackingosintpentestingreconreconnaissance
brucedone/awesome-crawler
vavkamil/awesome-bugbounty-toolsRelated projects:
| Repository | Description | Stars |
|---|---|---|
 jaeles-project/gospider | A tool for web crawling and exploitation written in Go. | 2,598 |
 iamstoxe/urlgrab | A tool to crawl websites by exploring links recursively with support for JavaScript rendering. | 331 |
 s0rg/crawley | A utility for systematically extracting URLs from web pages and printing them to the console. | 268 |
 geziyor/geziyor | A fast and flexible web crawling and scraping framework for extracting structured data from websites. | 2,646 |
| hakluke/haktrails | A Golang client for querying SecurityTrails API data to gather information on domains and their associated networks. | 540 |
 yasserg/crawler4j | A Java-based web crawler for extracting and processing web page content | 4,563 |
 gocolly/colly | A framework for extracting structured data from websites in a fast and elegant way | 23,444 |
 s0md3v/photon | A fast and flexible web crawler designed to gather information from the internet | 11,122 |
 spatie/crawler | A powerful web crawler written in PHP that can execute JavaScript and crawl multiple URLs concurrently. | 2,552 |
 yujiosaka/headless-chrome-crawler | A distributed crawling framework that leverages Headless Chrome to scrape dynamic websites | 5,534 |
 apify/crawlee | A tool for building reliable web scraping and browser automation pipelines in Node.js. | 16,081 |
 unclecode/crawl4ai | A web crawling tool designed to extract structured data from the web for use in AI applications | 18,541 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 380 |
 puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,036 |
 code4craft/webmagic | A framework for building scalable web crawlers in Java. | 11,456 |