spidy
by twiny
Domain names collector - Crawl websites and collect domain names along with their availability status.
AI summary
Domain crawler
Tools to crawl websites and collect domain names with availability status
- stars
- 151
- forks
- 27
- watching
- 6
Similar projects
Found by comparing what the projects do, not just their names.
Crawler library
A Ruby web crawling library that provides flexible and customizable methods to crawl websites
Web crawler
A simple command-line web crawler that automatically extracts links from web pages and can be run in parallel for efficient crawling
Crawler
An OSINT bot that crawls pastebin sites to search for sensitive data leaks
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Crawler
A modular, concurrent web crawler framework written in Go.
Search indexer
A package to create a private search index by crawling and indexing a website
Domain scraper
A tool that extracts domain names from SSL certificates of arbitrary hosts during TLS handshakes
Spider
A framework for extracting structured data from websites
Domain scanner
A tool to find identical domain names with SOA DNS records under different TLDs
Web scanner
A tool for crawling and scanning websites for sensitive information such as endpoints, secrets, and tokens.
Link crawler
A tool to crawl websites by exploring links recursively with support for JavaScript rendering.
Web crawler
A Node.js module for crawling web sites and scraping their content
s0rg/crawley268
Web URL extractor
A utility for systematically extracting URLs from web pages and printing them to the console.
dyweb/scrala113
Web crawler framework
A web crawling framework written in Scala that allows users to define the start URL and parse response from it
Domain checker
A CLI tool to check the availability of web domains
