awesome-spark

Spark toolkit

A curated collection of packages and resources for working with Apache Spark, an open-source cluster-computing framework.

A curated list of awesome Apache Spark packages and resources.

GitHub

2k stars
85 watching
331 forks
Language: Shell
last commit: almost 2 years ago
Linked from 3 awesome lists

apache-sparkawesomepysparksparkr

Awesome Spark / Packages / Language Bindings

Kotlin for Apache Spark463about 2 years agoKotlin API bindings and extensions
.NET for Apache Spark2,032almost 2 years ago.NET bindings
sparklyr955almost 2 years agoAn alternative R backend, using
sparkle447over 3 years agoHaskell on Apache Spark
spark-connect-rs91almost 2 years agoRust bindings
spark-connect-go168almost 2 years agoGolang bindings
spark-connect-csharp1over 2 years agoC# bindings

Awesome Spark / Packages / Notebooks and IDEs

almondA scala kernel for
Apache ZeppelinWeb-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box
PolynotePolynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from
sparkmagic1,334almost 2 years agomagics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through , in Jupyter notebooks

Awesome Spark / Packages / General Purpose Libraries

itachi56about 3 years agoA library that brings useful functions from modern database management systems to Apache Spark
spark-daria754almost 2 years agoA Scala library with essential Spark functions and extensions to make you more productive
quinn651almost 2 years agoA native PySpark implementation of spark-daria
Apache DataFu119almost 2 years agoA library of general purpose functions and UDF's
Joblib Apache Spark Backend243about 2 years agobackend for running tasks on Spark clusters

Awesome Spark / Packages / SQL Data Sources

Spark XML504about 2 years agoXML parser and writer
Spark Cassandra Connector1,944about 2 years agoCassandra support including data source and API and support for arbitrary queries
Mongo-Spark713about 2 years agoOfficial MongoDB connector

Awesome Spark / Packages / Storage

Delta Lake7,677almost 2 years agoStorage layer with ACID transactions
Apache Hudi5,498almost 2 years agoUpserts, Deletes And Incremental Processing on Big Data
Apache Iceberg6,621almost 2 years agoUpserts, Deletes And Incremental Processing on Big Data
lakeFSIntegration with the lakeFS atomic versioned storage layer

Awesome Spark / Packages / Bioinformatics

ADAM1,005almost 2 years agoSet of tools designed to analyse genomics data
Hail984almost 2 years agoGenetic analysis framework

Awesome Spark / Packages / GIS

Apache Sedona1,974almost 2 years agoCluster computing system for processing large-scale spatial data

Awesome Spark / Packages / Graph Processing

GraphFrames1,007almost 2 years agoData frame based graph API
neo4j-spark-connector313almost 2 years agoBolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support

Awesome Spark / Packages / Machine Learning Extension

Apache SystemMLDeclarative machine learning framework on top of Spark
Mahout Spark Bindings[status unknown] - linear algebra DSL and optimizer with R-like syntax
KeystoneMLType safe machine learning pipelines with RDDs
JPMML-Spark94over 4 years agoPMML transformer library for Spark ML
ModelDBA system to manage machine learning models for and
Sparkling Water968almost 2 years agointeroperability layer
BigDL6,801almost 2 years agoDistributed Deep Learning library
MLeap1,506almost 2 years agoExecution engine and serialization format which supports deployment of models without dependency on
Microsoft ML for Apache Spark5,083almost 2 years agoA distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment
MLflowMachine learning orchestration platform

Awesome Spark / Packages / Middleware

Livy894almost 2 years agoREST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing
spark-jobserver2,839almost 2 years agoSimple Spark as a Service which supports objects sharing using so called named objects. JVM only
Apache Toree740almost 2 years agoIPython protocol based middleware for interactive applications
Apache Kyuubi2,116almost 2 years agoA distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark

Awesome Spark / Packages / Monitoring

Data Mechanics Delight344over 2 years agoCross-platform monitoring tool (Spark UI / Spark History Server replacement)

Awesome Spark / Packages / Utilities

sparkly61about 3 years agoHelpers & syntactic sugar for PySpark
Flintrock637almost 2 years agoA command-line tool for launching Spark clusters on EC2
Optimus1,486almost 2 years agoData Cleansing and Exploration utilities with the goal of simplifying data cleaning

Awesome Spark / Packages / Natural Language Processing

spark-nlp3,889almost 2 years agoNatural language processing library built on top of Apache Spark ML

Awesome Spark / Packages / Streaming

Apache BahirCollection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ)

Awesome Spark / Packages / Interfaces

Apache BeamUnified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments
Koalas3,343over 2 years agoPandas DataFrame API on top of Apache Spark

Awesome Spark / Packages / Data quality

deequ3,324almost 2 years agoDeequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets
python-deequ734almost 2 years agoPython API for Deequ

Awesome Spark / Packages / Testing

spark-testing-base1,525almost 2 years agoCollection of base test classes
spark-fast-tests437almost 2 years agoA lightweight and fast testing framework
chispa632almost 2 years agoPySpark test helpers with beautiful error messages

Awesome Spark / Packages / Web Archives

Archives Unleashed Toolkit138over 2 years agoOpen-source toolkit for analyzing web archives

Awesome Spark / Packages / Workflow Management

Cromwell1,004almost 2 years agoWorkflow management system with

Awesome Spark / Resources / Books

Learning Spark, 2nd EditionIntroduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts
Advanced Analytics with SparkUseful collection of Spark processing patterns. Accompanying GitHub repository:
Mastering Apache SparkInteresting compilation of notes by . Focused on different aspects of Spark internals
Spark in ActionNew book in the Manning's "in action" family with +400 pages. Starts gently, step-by-step and covers large number of topics. Free excerpt on how to and how to bootstrap a new application using the provided Maven Archetype. You can find the accompanying GitHub repo

Awesome Spark / Resources / Papers

Large-Scale Intelligent MicroservicesMicrosoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster ComputingPaper introducing a core distributed memory abstraction
Spark SQL: Relational Data Processing in SparkPaper introducing relational underpinnings, code generation and Catalyst optimizer
Structured Streaming: A Declarative API for Real-Time Applications in Apache SparkStructured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query

Awesome Spark / Resources / MOOCS

Data Science and Engineering with Apache Spark (edX XSeries)Series of five courses ( , , , , ) covering different aspects of software engineering and data science. Python oriented
Big Data Analysis with Scala and Spark (Coursera)Scala oriented introductory course. Part of

Awesome Spark / Resources / Workshops

AMP CampPeriodical training event organized by the . A source of useful exercise and recorded workshops covering different tools from the

Awesome Spark / Resources / Projects Using Spark

Oryx 21,787about 5 years agoplatform built on Apache Spark and with specialization for real-time large scale machine learning
Photon ML793about 5 years agoA machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model
PredictionIOMachine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time
Crossdata169almost 7 years agoData integration platform with extended DataSource API and multi-user environment

Awesome Spark / Resources / Docker Images

apache/sparkApache Spark Official Docker images
jupyter/docker-stacks/pyspark-notebook8,037almost 2 years agoPySpark with Jupyter Notebook and Mesos client
sequenceiq/docker-spark765over 5 years agoYarn images from
datamechanics/sparkAn easy to setup Docker image for Apache Spark from

Awesome Spark / Resources / Miscellaneous

Spark with Scala Gitter channel" " started by
Apache Spark User Listand - Mailing lists dedicated to usage questions and development topics respectively

Backlinks from these awesome lists:

More related projects: