WarcPartitioner
ARC file processor
Tool for partitioning and merging Web archive files by MIME type and year
Partition (W)ARC Files by MIME Type and Year
1 stars
2 watching
1 forks
Language: Java
last commit: about 8 years ago
Linked from 1 awesome list
hadoopwarcweb-archivingwebarchive
Related projects:
Repository | Description | Stars |
---|---|---|
| Tools for bulk indexing of WARC/ARC files to create a shared url index | 43 |
| A Web crawler that creates custom archives in WARC/CDX format | 25 |
| Tools to identify and convert duplicate records in archived web content | 6 |
| A fast streaming library for working with WARC format web archival data | 391 |
| Provides tools for reading and parsing web archive formats used in digital preservation. | 20 |
| A tool for parsing and generating Web Archive files in JavaScript using Node.js | 95 |
| Tool for handling Web Archive files | 152 |
| Tools for indexing and discovering archived web content | 117 |
| Tools for working with archived web content | 153 |
| Converts HTTP Archive format to Web Archive format | 48 |
| A framework for efficient data processing and extraction from archival collections, enabling the transformation of raw data into more accessible formats. | 145 |
| Converts HTTrack crawls to WARC files by reconstructing requests and responses from logs | 32 |
| A web archiving tool that archives websites with high-fidelity preservation capabilities. | 57 |
| An tool for easy compilation and testing of Rust applications on WebAssembly platforms. | 261 |
| A web crawler designed to backup websites by recursively crawling and writing WARC files. | 1,406 |