Web2Warc
by helgeho
An easy-to-use and highly customizable crawler that enables you to create your own little Web archives (WARC/CDX)
AI summary
Crawler
A Web crawler that creates custom archives in WARC/CDX format
- stars
- 25
- forks
- 4
- watching
- 3
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Crawler
A containerized browser-based crawler system for capturing web content in a high-fidelity and customizable manner.
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Web archiver
An archival crawler built on top of Chrome or Chromium to preserve the web in high fidelity and user scriptable manner
ARC file processor
Tool for partitioning and merging Web archive files by MIME type and year
WARC crawler
Converts HTTrack crawls to WARC files by reconstructing requests and responses from logs
Web Crawler
A distributed web crawler that fetches and extracts links from websites using a real browser.
Web crawler
A web crawler designed to efficiently collect and prioritize relevant content from the web
hominee/dyer135
Web crawler
A fast and flexible web crawling tool with features like asynchronous I/O and event-driven design.
Web Crawler
A versatile web crawler built with modern tools and concurrency to handle various crawl tasks
Web Crawler
A flexible web crawler that can be used to extract data from websites in a scalable and efficient manner
Crawler
A high-performance web crawling and scraping solution with customizable settings and worker pooling.
Web crawler
A tool for recursively querying web servers by sending HTTP requests and analyzing responses to discover hidden content
Converter
Converts HTTP Archive format to Web Archive format
Crawler
A modular, concurrent web crawler framework written in Go.
Crawler framework
A scalable and versatile web crawling framework based on Apache Storm
Featured in 1 awesome list
Each link jumps to the spot where the list mentions Web2Warc.
