warcrefs
by arcalex
Web archive deduplication tools
AI summary
Duplicate converter
Tools to identify and convert duplicate records in archived web content
- stars
- 6
- forks
- 1
- watching
- 5
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
ARC file processor
Tool for partitioning and merging Web archive files by MIME type and year
WARC crawler
Converts HTTrack crawls to WARC files by reconstructing requests and responses from logs
Web archive indexer
Tools for indexing and discovering archived web content
Web archive storage
A library for storing and querying web crawl data in a compact, easily sharable format.
WARC Indexer
Tools for bulk indexing of WARC/ARC files to create a shared url index
WARC tools
Tools for working with archived web content
Web archive parser
Provides tools for reading and parsing web archive formats used in digital preservation.
chfoo/warcat152
Archive tool
Tool for handling Web Archive files
Converter
Converts HTTP Archive format to Web Archive format
Web page archiver
A web-based archive service that allows users to store and manage web pages in various formats.
Data converter
Converts offline data into a standard archival format
WARC parser
A tool for parsing and generating Web Archive files in JavaScript using Node.js
Web archiver
A containerized web archive and search system using Elastic Search
Web archive server
A RocksDB-based server for managing and replicating capture indexes used in web archiving
WARC library
A fast streaming library for working with WARC format web archival data
Featured in 1 awesome list
Each link jumps to the spot where the list mentions warcrefs.
