aut
Archive analyzer
An open-source toolkit for analyzing web archives using Apache Spark.
The Archives Unleashed Toolkit is an open-source toolkit for analyzing web archives.
138 stars
15 watching
33 forks
Language: Scala
last commit: over 2 years agoLinked from 2 awesome lists
analysisapache-sparkbig-databig-data-analyticsdataframedigital-humanitieshadoopnetwork-graphingpysparkpython3scalasparktext-extractionwebarchives
iipc/awesome-web-archiving
awesome-spark/awesome-sparkRelated projects:
| Repository | Description | Stars |
|---|---|---|
| archivesunleashed/twut | Analyzes line-oriented JSON data from Twitter APIs using Apache Spark | 9 |
| archivesunleashed/notebooks | Provides tools and examples for working with web archives using the Archives Unleashed Toolkit | 23 |
 netarchivesuite/jwat | A toolkit for analyzing and extracting data from legacy web archives in a structured format suitable for further analysis or reuse | 3 |
 oduwsdl/archivenow | A tool to automate archiving of web resources into public archives. | 409 |
 archiveteam/wpull | Downloads and crawls web pages, allowing for the archiving of websites. | 556 |
 machawk1/wail | A graphical user interface layer for preserving and replaying web pages using multiple archiving tools. | 353 |
 jiiks/asar.net | A .NET implementation of the Atom Asar archive format, allowing extraction and manipulation of archived files. | 36 |
 internetarchive/arch | A distributed compute analysis system for web archive collections | 15 |
 ukwa/webarchive-discovery | Tools for indexing and discovering archived web content | 117 |
 chatnoir-eu/chatnoir-resiliparse | A toolkit for processing and analyzing web archive data | 89 |
 recrm/archivetools | A collection of tools for extracting and analyzing data from web archives | 71 |
 bellingcat/auto-archiver | Automates archiving of online content from various sources into local storage or cloud services | 585 |
 le0me55i/zsh-extract | A plugin that automates the extraction of archive files from various formats. | 19 |
 karust/gogetcrawl | A tool and package for extracting web archive data from popular sources like Wayback Machine and Common Crawl using the Go programming language. | 148 |
 helgeho/archivespark | A framework for efficient data processing and extraction from archival collections, enabling the transformation of raw data into more accessible formats. | 145 |