awesome-data-engineering

RQLite

Replicated SQLite using the Raft consensus protocol

The world's most popular open source database

TiDB

TiDB is a distributed NewSQL database compatible with MySQL protocol

Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®

mysql_utils

Pinterest MySQL Management Tools

An enhanced, drop-in replacement for MySQL

The world's most advanced open source database

Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud

Scalable SQL database with the NOSQL goodies

An open source, BSD licensed, advanced key-value cache and store

A distributed database designed to deliver maximum data availability by distributing data across multiple servers

A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale

HyperDex

HyperDex is a scalable, searchable key-value store

A high performance NoSQL database supporting many data structures, an alternative to Redis

Kyoto Tycoon

Kyoto Tycoon is a lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency

IonDB

A key-value store for microcontroller and IoT applications

The right choice when you need scalability and high availability without compromising performance

This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application

CCM

A script to easily create and destroy an Apache Cassandra cluster on localhost

ScyllaDB

NoSQL data store using the seastar framework, compatible with Apache Cassandra

The Hadoop database, a distributed, scalable, big data store

Column oriented, open-source analytic database provides both speed and efficiency

A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools

https://github.com/tuplejump/FiloDB

FiloDB ( ) Distributed. Columnar. Versioned. Streaming. SQL

Distributed, MPP columnar database with extensive analytics SQL

An open-source, document database designed for ease of development and scaling

Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality

MemDB

Distributed Transactional In-Memory Database (based on MongoDB)

Search & Analyze Data in Real Time

The highest performing NoSQL distributed database

The open-source database for the realtime web

The world’s leading graph database

2nd Generation Distributed Graph Database with the flexibility of Documents in one product with an Open Source commercial friendly license

A distributed free and open-source database with a flexible data model for documents, graphs, and key-values

A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster

FlockDB

A distributed, fault-tolerant graph database by Twitter

The fully transactional, cloud-ready, distributed database

An open source, distributed, in-memory database for scale-out applications

Gaffer

A large-scale graph database

InfluxDB

Scalable datastore for metrics, events, and real-time analytics

OpenTSDB

A scalable, distributed Time Series Database

kairosdb

Fast scalable time series database

Heroic

A scalable time series database based on Cassandra and Elasticsearch, by Spotify

Druid

Column oriented distributed data store ideal for powering interactive applications

Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data

Akumuli

Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate"

A time-series object store for Cassandra that handles all the complexity of building wide row indexes

Dalmatiner DB

Fast distributed metrics database

Blueflood

A distributed system designed to ingest and process time series data

Timely

Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana

Tarantool

Tarantool is an in-memory database and application server

The Greenplum Database (GPDB) is an advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes

cayley

An open-source graph database. Google

Snappydata

SnappyData: OLTP + OLAP Database built on Apache Spark

Publish-subscribe messaging rethought as a distributed commit log

Camus

LinkedIn's Kafka to HDFS pipeline

BottledWater

Change data capture from PostgreSQL into Kafka

kafkat

Simplified command-line administration for Kafka brokers

kafkacat

Generic command line non-JVM Apache Kafka producer and consumer

pg-kafka

A PostgreSQL extension to produce messages to Apache Kafka

librdkafka

The Apache Kafka C/C++ library

kafka-docker

Kafka in Docker

kafka-manager

A tool for managing Apache Kafka

kafka-node

Node.js client for Apache Kafka 0.8

Secor

Pinterest's Kafka to S3 distributed consumer

Kafka-logger

Kafka-winston logger for nodejs from uber

Kafka Awesome List

A super list of resources about Apache Kafka

A fully managed, cloud-based service for real-time data processing over large, distributed data streams

Robust messaging for applications

An open source data collector for unified logging layer

An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services

A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases

Heka

Data Acquisition and Processing Made Easy

Gobblin

Universal data ingestion framework for Hadoop from Linkedin

Nakadi is an open source event messaging platform that provides a REST API on top of Kafka-like queues

Pravega provides a new storage abstraction - a stream - for continuous and unbounded data

Apache Pulsar is an open-source distributed pub-sub messaging system

Snakebite

A pure python HDFS client

smart_open

Utils for streaming large files (S3, HDFS, gzip, bz2)

Tachyon is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce

Ceph is a unified, distributed storage system designed for excellent performance, reliability and scalability

Orange File System is a branch of the Parallel Virtual File System

SnackFS

SnackFS is our bite-sized, lightweight HDFS compatible FileSystem built over Cassandra

Gluster Filesystem

fault-tolerant distributed file system for all storage needs

SeaweedFS

Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS"

S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack

LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system

Apache Avro™ is a data serialization system

Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language

Snappy

A fast compressor/decompressor. Used with Parquet

A parallel implementation of gzip for modern multi-processor, multi-core machines

The smallest, fastest columnar storage for Hadoop workloads

The Apache Thrift software framework, for scalable cross-language services development

ProtoBuf

Protocol Buffers - Google's data interchange format

SequenceFile is a flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats

Kryo

Kryo is a fast and efficient object graph serialization framework for Java

Apache Beam is a unified programming model that implements both batch and streaming data processing jobs that run on many execution engines

Spark Streaming makes it easy to build scalable fault-tolerant streaming applications

Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams

Apache Storm is a free and open source distributed realtime computation system

Apache Samza is a distributed stream processing framework

is an easy to use, powerful, and reliable system to process and distribute data

VoltDb is an ACID-compliant RDBMS which uses a

PipelineDB

The Streaming SQL Database

Streaming and tasks execution between Spring Boot apps

Bonobo is a data-processing toolkit for python 3.5+

Robinhood's Faust

Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing

Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) in-parallel on large clusters (thousands of nodes) of commodity hardware in a reliable, fault-tolerant manner

A community index of packages for Apache Spark

Deep Spark

Connecting Apache Spark with different data stores

by Zhen He

Livy

Livy, the REST Spark Server

A web service that makes it easy to quickly and cost-effectively process vast amounts of data

An application framework which allows for a complex directed-acyclic-graph of tasks for processing data

Bistro

is a light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via and processes data via as opposed to having only set operations in conventional approaches like MapReduce or SQL

Fast scalable machine learning API for smarter applications

An environment for quickly creating scalable performant machine learning applications

Spark’s scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives

A machine learning platform that enables data scientists and app developers to easily create intelligent apps at scale

An iterative graph processing system built for high scalability

Apache Spark's API for graphs and graph-parallel computation

A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources

Data warehouse software facilitates querying and managing large datasets residing in distributed storage

Hivemall

Scalable machine learning library for Hive/Hadoop

PyHive

Python interface to Hive and Presto

Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage

A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application

Fast JavaScript charts for any data set

D3-based reusable chart library

A JavaScript library for manipulating documents based on data

D3's simplier, easier to use cousin. Mostly predefined templates that you can just plug data in

A JavaScript Charting Library for Streaming Data

PyXley

Python helpers for building dashboards using Flask and React

Plotly

Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python

Apache Superset

Apache Superset (incubating) is a modern, enterprise-ready business intelligence web application

Make Your Company Data Driven. Connect to any data source, easily visualize and share your data

Metabase

Metabase is the easy, open source way for everyone in your company to ask questions and learn from data

PyQtGraph is a pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications

Luigi

Luigi is a Python module that helps you build complex pipelines of batch jobs

An application cron-like system. w/Luige

Java based application development platform

Airflow

Airflow is a system to programmaticaly author, schedule and monitor data pipelines

Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows

Oozie is a workflow scheduler system to manage Apache Hadoop jobs

Pinball

DAG based workflow manager. Job flows are defined programmaticaly in Python. Support output passing between jobs

docker-logstash

A highly configurable logstash (1.4.4) docker image running Elasticsearch (1.7.0) and Kibana (3.1.2)

elasticsearch-jdbc

JDBC importer for Elasticsearch

ZomboDB

Postgres Extension that allows creating an index backed by Elasticsearch

Gockerize

Package golang service into minimal docker containers

Flocker

Easily manage Docker containers & their data

RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers

Application Containers for Masses

Weave

Weaving Docker containers into applications

Zodiac

A lightweight tool for easy deployment and rollback of dockerized applications

cAdvisor

Analyzes resource usage and performance characteristics of running containers

Micro S3 persistence

Docker microservice for saving/restoring volume data to S3

Dockup

Docker image to backup/restore your Docker container volumes to AWS S3

Rocker-compose

Docker composition tool with idempotency features for deploying apps composed of multiple containers

Nomad

Nomad is a cluster manager, designed for both long lived services and short lived batch processing workloads

Vizualize docker images and the layers that compose them

The Streaming APIs give developers low latency access to Twitter’s global stream of Tweet data

Eventsim

Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic

Real-time data is available including comments, submissions and links posted to reddit

GitHub's public timeline since 2011, updated every hour

Open source repository of web crawl data

Wikipedia's complete copy of all wikis, in the form of wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available

Prometheus.io

An open-source service monitoring system and time series database

HAProxy Exporter

Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption

News, tips and background on Data Engineering

Subreddit focused on ETL

DataEngConf is the first technical conference that bridges the gap between data scientists, data engineers and data analysts

The show about modern data infrastructure