ArchiveSpark
by helgeho
An Apache Spark framework for easy data processing, extraction as well as derivation for web archives and archival collections, developed at Internet Archive.
AI summary
Data processor
A framework for efficient data processing and extraction from archival collections, enabling the transformation of raw data into more accessible formats.
- stars
- 145
- forks
- 19
- watching
- 15
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
Web data processor
A data processing library built on top of Apache Spark to handle temporal web data
apache/spark40.2K
Data processor
An analytics engine designed to handle large-scale data processing and analysis
ARC file processor
Tool for partitioning and merging Web archive files by MIME type and year
Archive processor
A distributed compute analysis system for web archive collections
Dataset processor
A lightweight Python implementation of Spark's RDD and DStream interfaces for improved performance on small datasets
Spark Data Processor
An introductory Scala app using Apache Spark Streaming to process data from Kafka and write summaries to Cassandra.
Data processor
Enables data processing and transformation in large files using a high-level language with compile-time optimizations for efficient execution on distributed computing frameworks.
GZIP handler
Provides a custom input format for handling concatenated GZIP files in distributed processing systems like Hadoop
Parallel data processor
A Haskell-based framework for processing and distributing large datasets across multiple nodes in parallel.
Data processor
A library for data processing and querying on large datasets without the need for Hadoop expertise
Data processor
A distributed stream processing framework for handling high-volume data streams with fault tolerance and durability guarantees
Archive analyzer
An open-source toolkit for analyzing web archives using Apache Spark.
Crawler
A Web crawler that creates custom archives in WARC/CDX format
Data repository
A web application for managing and sharing high-energy physics data from experiments
Data serializer
A Haskell library for serializing and deserializing data structures in a persistent manner.
Featured in 1 awesome list
Each link jumps to the spot where the list mentions ArchiveSpark.
