crawler
Blockchain crawler
A tool for retrieving on-chain data from Ethereum and other blockchain networks in a modular, efficient manner.
A tool chain to retrieve on-chain data from Ethereum.
6 stars
3 watching
1 forks
Language: JavaScript
last commit: about 1 month ago
Linked from 1 awesome list
ahmet/awesome-web3Related projects:
| Repository | Description | Stars |
|---|---|---|
 keep-reth-strange/reth-crawler | A software tool designed to crawl and analyze the Ethereum network | 60 |
 rndinfosecguy/scavenger | An OSINT bot that crawls pastebin sites to search for sensitive data leaks | 629 |
 puerkitobio/fetchbot | A flexible web crawler that follows robots.txt policies and crawl delays. | 786 |
 crypto-crawler/crypto-crawler-rs | A Rust-based library for building and managing cryptocurrency crawlers | 232 |
 archiveteam/grab-site | A web crawler designed to backup websites by recursively crawling and writing WARC files. | 1,398 |
 brendonboshell/supercrawler | A web crawler designed to crawl websites while obeying robots.txt rules, rate limits and concurrency limits, with customizable content handlers for parsing and processing crawled pages. | 378 |
 internetarchive/brozzler | A distributed web crawler that fetches and extracts links from websites using a real browser. | 671 |
 jmg/crawley | A Pythonic framework for building high-speed web crawlers with flexible data extraction and storage options. | 186 |
| puerkitobio/gocrawl | A concurrent web crawler written in Go that allows flexible and polite crawling of websites. | 2,038 |
 fredwu/crawler | A high-performance web crawling and scraping solution with customizable settings and worker pooling. | 945 |
 elixir-crawly/crawly | A framework for extracting structured data from websites | 987 |
 c-sto/recursebuster | A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content | 242 |
 stewartmckee/cobweb | A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner | 226 |
 elliotgao2/gain | A Python web crawling framework utilizing asyncio and aiohttp for efficient data extraction from websites. | 2,035 |
 webrecorder/browsertrix-crawler | A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner. | 652 |