crawley
by s0rg
The unix-way web crawler
AI summary
Web URL extractor
A utility for systematically extracting URLs from web pages and printing them to the console.
- stars
- 268
- forks
- 14
- watching
- 2
- awesome lists
- 4
Similar projects
Found by comparing what the projects do, not just their names.
URL extractor
A tool to extract URLs from HTML attributes by crawling in and evaluating JavaScript
mvdan/xurls1.2K
URL extractor
A tool to extract URLs from text using regular expressions in the Go programming language.
Archive extractor
A tool and package for extracting web archive data from popular sources like Wayback Machine and Common Crawl using the Go programming language.
JS extractor
A tool to extract JavaScript sources from URLs and web pages efficiently
HTML extractor
A tool for automatically extracting structured data from HTML pages
Web scraper
A tool to simplify web scraping of list-like structured data from web pages
URL Extractor
A tool to extract and dump URLs from Chrome's stored databases.
Web archiver
Archives a web page as a single HTML file with embedded resources.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Web scraper
A framework for extracting structured data from web pages using CSS selectors.
Link crawler
A tool to crawl websites by exploring links recursively with support for JavaScript rendering.
Website scraper
Downloads and crawls web pages, allowing for the archiving of websites.
Crawler
A concurrent web crawler written in Go that allows flexible and polite crawling of websites.
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
Featured in 4 awesome lists
Each link jumps to the spot where the list mentions crawley.
