awesome-spark

Kotlin for Apache Spark

Kotlin API bindings and extensions

.NET for Apache Spark

.NET bindings

sparklyr

An alternative R backend, using

sparkle

Haskell on Apache Spark

spark-connect-rs

Rust bindings

spark-connect-go

Golang bindings

spark-connect-csharp

C# bindings

A scala kernel for

Web-based notebook that enables interactive data analytics with plugable backends, integrated plotting, and extensive Spark support out-of-the-box

Polynote: an IDE-inspired polyglot notebook. It supports mixing multiple languages in one notebook, and sharing data between them seamlessly. It encourages reproducible notebooks with its immutable data model. Originating from

sparkmagic

magics and kernels for working with remote Spark clusters, for interactively working with remote Spark clusters through , in Jupyter notebooks

itachi

A library that brings useful functions from modern database management systems to Apache Spark

spark-daria

A Scala library with essential Spark functions and extensions to make you more productive

quinn

A native PySpark implementation of spark-daria

Apache DataFu

A library of general purpose functions and UDF's

Joblib Apache Spark Backend

backend for running tasks on Spark clusters

Spark XML

XML parser and writer

Spark Cassandra Connector

Cassandra support including data source and API and support for arbitrary queries

Mongo-Spark

Official MongoDB connector

Delta Lake

Storage layer with ACID transactions

Apache Hudi

Upserts, Deletes And Incremental Processing on Big Data

Apache Iceberg

Upserts, Deletes And Incremental Processing on Big Data

Integration with the lakeFS atomic versioned storage layer

ADAM

Set of tools designed to analyse genomics data

Hail

Genetic analysis framework

Apache Sedona

Cluster computing system for processing large-scale spatial data

GraphFrames

Data frame based graph API

neo4j-spark-connector

Bolt protocol based, Neo4j Connector with RDD, DataFrame and GraphX / GraphFrames support

Declarative machine learning framework on top of Spark

[status unknown] - linear algebra DSL and optimizer with R-like syntax

Type safe machine learning pipelines with RDDs

JPMML-Spark

PMML transformer library for Spark ML

A system to manage machine learning models for and

Sparkling Water

interoperability layer

BigDL

Distributed Deep Learning library

MLeap

Execution engine and serialization format which supports deployment of models without dependency on

Microsoft ML for Apache Spark

A distributed ml library with support for LightGBM, Vowpal Wabbit, OpenCV, Deep Learning, Cognitive Services, and Model Deployment

Machine learning orchestration platform

Livy

REST server with extensive language support (Python, R, Scala), ability to maintain interactive sessions and object sharing

spark-jobserver

Simple Spark as a Service which supports objects sharing using so called named objects. JVM only

Apache Toree

IPython protocol based middleware for interactive applications

Apache Kyuubi

A distributed multi-tenant JDBC server for large-scale data processing and analytics, built on top of Apache Spark

Data Mechanics Delight

Cross-platform monitoring tool (Spark UI / Spark History Server replacement)

sparkly

Helpers & syntactic sugar for PySpark

Flintrock

A command-line tool for launching Spark clusters on EC2

Optimus

Data Cleansing and Exploration utilities with the goal of simplifying data cleaning

spark-nlp

Natural language processing library built on top of Apache Spark ML

Collection of the streaming connectors excluded from Spark 2.0 (Akka, MQTT, Twitter. ZeroMQ)

Unified data processing engine supporting both batch and streaming applications. Apache Spark is one of the supported execution environments

Koalas

Pandas DataFrame API on top of Apache Spark

deequ

Deequ is a library built on top of Apache Spark for defining "unit tests for data", which measure data quality in large datasets

python-deequ

Python API for Deequ

spark-testing-base

Collection of base test classes

spark-fast-tests

A lightweight and fast testing framework

chispa

PySpark test helpers with beautiful error messages

Archives Unleashed Toolkit

Open-source toolkit for analyzing web archives

Cromwell

Workflow management system with

Introduction to Spark API with Spark 3.0 covered. Good source of knowledge about basic concepts

Useful collection of Spark processing patterns. Accompanying GitHub repository:

Interesting compilation of notes by . Focused on different aspects of Spark internals

New book in the Manning's "in action" family with +400 pages. Starts gently, step-by-step and covers large number of topics. Free excerpt on how to and how to bootstrap a new application using the provided Maven Archetype. You can find the accompanying GitHub repo

Microsoft paper that presents an Apache Spark-based micro-service orchestration framework that extends database operations to include web service primitives

Paper introducing a core distributed memory abstraction

Paper introducing relational underpinnings, code generation and Catalyst optimizer

Structured Streaming is a new high-level streaming API, it is a declarative API based on automatically incrementalizing a static relational query

Series of five courses ( , , , , ) covering different aspects of software engineering and data science. Python oriented

Scala oriented introductory course. Part of

Periodical training event organized by the . A source of useful exercise and recorded workshops covering different tools from the

Oryx 2

platform built on Apache Spark and with specialization for real-time large scale machine learning

Photon ML

A machine learning library supporting classical Generalized Mixed Model and Generalized Additive Mixed Effect Model

Machine Learning server for developers and data scientists to build and deploy predictive applications in a fraction of the time

Crossdata

Data integration platform with extended DataSource API and multi-user environment

Apache Spark Official Docker images

jupyter/docker-stacks/pyspark-notebook

PySpark with Jupyter Notebook and Mesos client

sequenceiq/docker-spark

Yarn images from

An easy to setup Docker image for Apache Spark from

" " started by

and - Mailing lists dedicated to usage questions and development topics respectively