WarcDB
WarcDB: Web crawl data as SQLite databases.
AI summary
Web archive storage
A library for storing and querying web crawl data in a compact, easily sharable format.
- stars
- 397
- forks
- 11
- watching
- 10
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
chfoo/warcat152
Archive tool
Tool for handling Web Archive files
Web crawler
A web crawler designed to backup websites by recursively crawling and writing WARC files.
Web archiver
A command-line tool for archiving and playing back websites in WARC format
Web archiver
An HTTP proxy designed to capture and archive web traffic, including encrypted HTTPS connections.
WARC tools
Tools for working with archived web content
WARC Indexer
Tools for bulk indexing of WARC/ARC files to create a shared url index
WARC library
A fast streaming library for working with WARC format web archival data
Web archive server
A RocksDB-based server for managing and replicating capture indexes used in web archiving
Web archiver
A containerized web archive and search system using Elastic Search
Website scraper
Downloads and crawls web pages, allowing for the archiving of websites.
Web archive indexer
Tools for indexing and discovering archived web content
Web archive parser
Provides tools for reading and parsing web archive formats used in digital preservation.
ARC file processor
Tool for partitioning and merging Web archive files by MIME type and year
Duplicate converter
Tools to identify and convert duplicate records in archived web content
oduwsdl/ipwb617
Web archiver
A system for dispersing and replaying archived web content using peer-to-peer technology.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions WarcDB.
