grab-site
by ArchiveTeam
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
AI summary
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
- stars
- 1.4K
- forks
- 136
- watching
- 41
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Website scraper
Downloads and crawls web pages, allowing for the archiving of websites.
Crawler
A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.
Crawler
A Web crawler that creates custom archives in WARC/CDX format
WARC crawler
Converts HTTrack crawls to WARC files by reconstructing requests and responses from logs
Web archiver
A web archiving tool that archives websites with high-fidelity preservation capabilities.
Web archiver
An archival crawler built on top of Chrome or Chromium to preserve the web in high fidelity and user scriptable manner
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
Web archiver
A command-line tool for archiving and playing back websites in WARC format
WARC tools
Tools for working with archived web content
Web archiver
An HTTP proxy designed to capture and archive web traffic, including encrypted HTTPS connections.
Web crawler
A web crawler designed to efficiently collect and prioritize relevant content from the web
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
chfoo/warcat152
Archive tool
Tool for handling Web Archive files
crawler
Performs web page crawling at high performance.
spider-rs/spider1.2K
Web scraper
A tool for web data extraction and processing using Rust
Featured in 1 awesome list
Each link jumps to the spot where the list mentions grab-site.
