awesome-data-engineering

Data engineering toolkit

A curated list of tools and technologies for data engineering

A curated list of data engineering tools for software developers

GitHub
7k stars
260 watching
1k forks
last commit: 28 days ago
Linked from 5 awesome lists

awesomeawesome-list

Awesome Data Engineering / Databases / Relational
RQLite 15,790 6 days ago Replicated SQLite using the Raft consensus protocol
MySQL The world's most popular open source database

Awesome Data Engineering / Databases / Relational / MySQL
TiDB 37,275 6 days ago TiDB is a distributed NewSQL database compatible with MySQL protocol
Percona XtraBackup Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®
mysql_utils 884 over 5 years ago Pinterest MySQL Management Tools

Awesome Data Engineering / Databases / Relational
MariaDB An enhanced, drop-in replacement for MySQL
PostgreSQL The world's most advanced open source database
Amazon RDS Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud
Crate.IO Scalable SQL database with the NOSQL goodies

Awesome Data Engineering / Databases / Key-Value
Redis An open source, BSD licensed, advanced key-value cache and store
Riak A distributed database designed to deliver maximum data availability by distributing data across multiple servers
AWS DynamoDB A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale
HyperDex 1,394 6 months ago HyperDex is a scalable, searchable key-value store. Deprecated
SSDB A high performance NoSQL database supporting many data structures, an alternative to Redis
Kyoto Tycoon 276 12 months ago Kyoto Tycoon is a lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency
IonDB 586 6 months ago A key-value store for microcontroller and IoT applications

Awesome Data Engineering / Databases / Column
Cassandra The right choice when you need scalability and high availability without compromising performance

Awesome Data Engineering / Databases / Column / Cassandra
Cassandra Calculator This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application
CCM 1,216 about 2 months ago A script to easily create and destroy an Apache Cassandra cluster on localhost
ScyllaDB 13,581 6 days ago NoSQL data store using the seastar framework, compatible with Apache Cassandra

Awesome Data Engineering / Databases / Column
HBase The Hadoop database, a distributed, scalable, big data store
AWS Redshift A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools
FiloDB 1,430 7 days ago Distributed. Columnar. Versioned. Streaming. SQL
Vertica Distributed, MPP columnar database with extensive analytics SQL
ClickHouse Distributed columnar DBMS for OLAP. SQL

Awesome Data Engineering / Databases / Document
MongoDB An open-source, document database designed for ease of development and scaling

Awesome Data Engineering / Databases / Document / MongoDB
Percona Server for MongoDB Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality
MemDB 596 over 6 years ago Distributed Transactional In-Memory Database (based on MongoDB)

Awesome Data Engineering / Databases / Document
Elasticsearch Search & Analyze Data in Real Time
Couchbase The highest performing NoSQL distributed database
RethinkDB The open-source database for the realtime web
RavenDB Fully Transactional NoSQL Document Database

Awesome Data Engineering / Databases / Graph
Neo4j The world's leading graph database
OrientDB 2nd Generation Distributed Graph Database with the flexibility of Documents in one product with an Open Source commercial friendly license
ArangoDB A distributed free and open-source database with a flexible data model for documents, graphs, and key-values
Titan A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster
FlockDB 3,338 over 7 years ago A distributed, fault-tolerant graph database by Twitter. Deprecated

Awesome Data Engineering / Databases / Distributed
DAtomic The fully transactional, cloud-ready, distributed database
Apache Geode An open source, distributed, in-memory database for scale-out applications
Gaffer 1,769 8 days ago A large-scale graph database

Awesome Data Engineering / Databases / Timeseries
InfluxDB 28,968 6 days ago Scalable datastore for metrics, events, and real-time analytics
OpenTSDB 5,005 about 2 months ago A scalable, distributed Time Series Database
QuestDB A relational column-oriented database designed for real-time analytics on time series and event data
kairosdb 1,740 7 months ago Fast scalable time series database
Heroic 848 over 3 years ago A scalable time series database based on Cassandra and Elasticsearch, by Spotify
Druid 13,513 6 days ago Column oriented distributed data store ideal for powering interactive applications
Riak-TS Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data
Akumuli 836 over 2 years ago Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate"
Rhombus A time-series object store for Cassandra that handles all the complexity of building wide row indexes
Dalmatiner DB 695 almost 6 years ago Fast distributed metrics database
Blueflood 596 3 months ago A distributed system designed to ingest and process time series data
Timely 380 5 months ago Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana

Awesome Data Engineering / Databases / Other
Tarantool 3,418 8 days ago Tarantool is an in-memory database and application server
GreenPlum The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes
cayley 14,855 5 months ago An open-source graph database. Google
Snappydata 1,040 about 2 years ago SnappyData: OLTP + OLAP Database built on Apache Spark
TimescaleDB Built as an extension on top of PostgreSQL, TimescaleDB is a time-series SQL database providing fast analytics, scalability, with automated data management on a proven storage engine
DuckDB DuckDB is a fast in-process analytical database that has zero external dependencies, runs on Linux/macOS/Windows, offers a rich SQL dialect, and is free and extensible

Awesome Data Engineering / Data Comparison
datacompy 485 6 days ago DataComPy is a Python library that facilitates the comparison of two DataFrames in pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels

Awesome Data Engineering / Data Ingestion
Kafka Publish-subscribe messaging rethought as a distributed commit log

Awesome Data Engineering / Data Ingestion / Kafka
BottledWater 3 over 1 year ago Change data capture from PostgreSQL into Kafka. Deprecated
kafkat 504 over 5 years ago Simplified command-line administration for Kafka brokers
kafkacat 5,447 4 months ago Generic command line non-JVM Apache Kafka producer and consumer
pg-kafka 112 over 9 years ago A PostgreSQL extension to produce messages to Apache Kafka
librdkafka 284 4 days ago The Apache Kafka C/C++ library
kafka-docker 6,941 7 months ago Kafka in Docker
kafka-manager 11,836 over 1 year ago A tool for managing Apache Kafka
kafka-node 2,664 about 1 year ago Node.js client for Apache Kafka 0.8
Secor 1,846 about 1 month ago Pinterest's Kafka to S3 distributed consumer
Kafka-logger 45 about 6 years ago Kafka-winston logger for Node.js from Uber

Awesome Data Engineering / Data Ingestion
AWS Kinesis A fully managed, cloud-based service for real-time data processing over large, distributed data streams
RabbitMQ Robust messaging for applications
dlt A fast&simple pipeline building library for python data devs, runs in notebooks, cloud functions, airflow, etc
FluentD An open source data collector for unified logging layer
Embulk An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services
Apache Sqoop A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases
Heka 3,390 10 months ago Data Acquisition and Processing Made Easy. Deprecated
Gobblin 2,229 7 days ago Universal data ingestion framework for Hadoop from LinkedIn
Nakadi Nakadi is an open source event messaging platform that provides a REST API on top of Kafka-like queues
Pravega Pravega provides a new storage abstraction - a stream - for continuous and unbounded data
Apache Pulsar Apache Pulsar is an open-source distributed pub-sub messaging system
AWS Data Wrangler 3,932 6 days ago Utility belt to handle data on AWS
Airbyte Open-source data integration for modern data teams
Artie Real-time data ingestion tool leveraging change data capture
Sling Sling is CLI data integration tool specialized in moving data between databases, as well as storage systems
Meltano CLI & code-first ELT

Awesome Data Engineering / Data Ingestion / Meltano
Singer SDK The fastest way to build custom data extractors and loaders compliant with the Singer Spec

Awesome Data Engineering / Data Ingestion
Google Sheets ETL 18 about 2 months ago Live import all your Google Sheets to your data warehouse

Awesome Data Engineering / File System
HDFS A distributed file system designed to run on commodity hardware

Awesome Data Engineering / File System / HDFS
Snakebite 855 over 2 years ago A pure python HDFS client

Awesome Data Engineering / File System
AWS S3 Object storage built to retrieve any amount of data from anywhere

Awesome Data Engineering / File System / AWS S3
smart_open 3,215 22 days ago Utils for streaming large files (S3, HDFS, gzip, bz2)

Awesome Data Engineering / File System
Alluxio Alluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce
CEPH Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability
JuiceFS 10,904 6 days ago JuiceFS is a high-performance Cloud-Native file system driven by object storage for large-scale data storage
OrangeFS Orange File System is a branch of the Parallel Virtual File System
SnackFS 14 over 9 years ago SnackFS is our bite-sized, lightweight HDFS compatible file system built over Cassandra
GlusterFS Gluster Filesystem
XtreemFS Fault-tolerant distributed file system for all storage needs
SeaweedFS 22,853 6 days ago Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS"
S3QL 1,112 19 days ago S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack
LizardFS LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system

Awesome Data Engineering / Serialization format
Apache Avro Apache Avro™ is a data serialization system
Apache Parquet Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language

Awesome Data Engineering / Serialization format / Apache Parquet
Snappy 6,161 3 months ago A fast compressor/decompressor. Used with Parquet
PigZ A parallel implementation of gzip for modern multi-processor, multi-core machines

Awesome Data Engineering / Serialization format
Apache ORC The smallest, fastest columnar storage for Hadoop workloads
Apache Thrift The Apache Thrift software framework, for scalable cross-language services development
ProtoBuf 65,753 6 days ago Protocol Buffers - Google's data interchange format
SequenceFile SequenceFile is a flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats
Kryo 6,202 17 days ago Kryo is a fast and efficient object graph serialization framework for Java

Awesome Data Engineering / Stream Processing
Apache Beam Apache Beam is a unified programming model that implements both batch and streaming data processing jobs that run on many execution engines
Spark Streaming Spark Streaming makes it easy to build scalable fault-tolerant streaming applications
Apache Flink Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams
Apache Storm Apache Storm is a free and open source distributed realtime computation system
Apache Samza Apache Samza is a distributed stream processing framework
Apache NiFi An easy to use, powerful, and reliable system to process and distribute data
Apache Hudi An open source framework for managing storage for real time processing, one of the most interesting feature is the Upsert
VoltDB VoltDb is an ACID-compliant RDBMS which uses a
PipelineDB 2,633 almost 3 years ago The Streaming SQL Database
Spring Cloud Dataflow Streaming and tasks execution between Spring Boot apps
Bonobo Bonobo is a data-processing toolkit for python 3.5+
Robinhood's Faust 1,656 7 days ago Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing
HStreamDB 708 about 2 months ago The streaming database built for IoT data storage and real-time processing
Kuiper 1,480 5 days ago An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices
Zilla 543 8 days ago - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol
SwimOS 314 6 days ago A framework for building real-time streaming data processing applications that supports a wide range of ingestion sources

Awesome Data Engineering / Batch Processing
Hadoop MapReduce Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner
Spark A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters

Awesome Data Engineering / Batch Processing / Spark
Spark Packages A community index of packages for Apache Spark
Deep Spark 197 over 8 years ago Connecting Apache Spark with different data stores. Deprecated
Spark RDD API Examples Examples by Zhen He
Livy The REST Spark Server
Delight 342 6 months ago A free & cross platform monitoring tool (Spark UI / Spark History Server alternative)

Awesome Data Engineering / Batch Processing
AWS EMR A web service that makes it easy to quickly and cost-effectively process vast amounts of data
Data Mechanics A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective
Tez An application framework which allows for a complex directed-acyclic-graph of tasks for processing data
Bistro 7 about 6 years ago A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via and processes data via as opposed to having only set operations in conventional approaches like MapReduce or SQL

Awesome Data Engineering / Batch Processing / Batch ML
H2O Fast scalable machine learning API for smarter applications
Mahout An environment for quickly creating scalable performant machine learning applications
Spark MLlib Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives

Awesome Data Engineering / Batch Processing / Batch Graph
GraphLab Create A machine learning platform that enables data scientists and app developers to easily create intelligent apps at scale
Giraph An iterative graph processing system built for high scalability
Spark GraphX Apache Spark's API for graphs and graph-parallel computation

Awesome Data Engineering / Batch Processing / Batch SQL
Presto A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources
Hive Data warehouse software facilitates querying and managing large datasets residing in distributed storage

Awesome Data Engineering / Batch Processing / Batch SQL / Hive
Hivemall 310 about 2 years ago Scalable machine learning library for Hive/Hadoop
PyHive 1,671 4 months ago Python interface to Hive and Presto

Awesome Data Engineering / Batch Processing / Batch SQL
Drill Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage

Awesome Data Engineering / Charts and Dashboards
Highcharts A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application
ZingChart Fast JavaScript charts for any data set
C3.js D3-based reusable chart library
D3.js A JavaScript library for manipulating documents based on data

Awesome Data Engineering / Charts and Dashboards / D3.js
D3Plus D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in

Awesome Data Engineering / Charts and Dashboards
SmoothieCharts A JavaScript Charting Library for Streaming Data
PyXley 2,269 over 6 years ago Python helpers for building dashboards using Flask and React
Plotly 21,482 9 days ago Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python
Apache Superset 62,799 7 days ago Apache Superset (incubating) - A modern, enterprise-ready business intelligence web application
Redash Make Your Company Data Driven. Connect to any data source, easily visualize and share your data
Metabase 38,770 6 days ago Metabase is the easy, open source way for everyone in your company to ask questions and learn from data
PyQtGraph PyQtGraph is a pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications

Awesome Data Engineering / Workflow
Luigi 17,869 9 days ago Luigi is a Python module that helps you build complex pipelines of batch jobs
CronQ An application cron-like system. w/Luige. Deprecated
Cascading Java based application development platform
Airflow 37,120 6 days ago Airflow is a system to programmatically author, schedule, and monitor data pipelines
Azkaban Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy-to-use web user interface to maintain and track your workflows
Oozie Oozie is a workflow scheduler system to manage Apache Hadoop jobs
Pinball 1,047 almost 5 years ago DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs
Dagster 11,699 6 days ago Dagster is an open-source Python library for building data applications
Hamilton 1,861 7 days ago Hamilton is a lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL transforms, you will like Hamilton for Python processing
Kedro Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly
Dataform An open-source framework and web based IDE to manage datasets and their dependencies. SQLX extends your existing SQL warehouse dialect to add features that support dependency management, testing, documentation and more
Census A reverse-ETL tool that let you sync data from your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, Zendesk, etc. No engineering favors required—just SQL
dbt A command line tool that enables data analysts and engineers to transform data in their warehouses more effectively
Kestra Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code
RudderStack 4,089 6 days ago A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools
PACE 34 6 days ago An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.)
Prefect Prefect is an orchestration and observability platform. With it, developers can rapidly build and scale resilient code, and triage disruptions effortlessly
Multiwoven 1,550 4 days ago The open-source reverse ETL, data activation platform for modern data teams
SuprSend Create automated workflows and logic using API's for your notification service. Add templates, batching, preferences, inapp inbox with workflows to trigger notifications directly from your data warehouse
Kestra 12,971 4 days ago A versatile open source orchestrator and scheduler built on Java, designed to handle a broad range of workflows with a language-agnostic, API-first architecture
Mage Open-source data pipeline tool for transforming and integrating data

Awesome Data Engineering / Data Lake Management
lakeFS 4,458 4 days ago lakeFS is an open source platform that delivers resilience and manageability to object-storage based data lakes
Project Nessie 1,038 6 days ago Project Nessie is a Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables

Awesome Data Engineering / ELK Elastic Logstash Kibana
docker-logstash 236 almost 9 years ago A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2)
elasticsearch-jdbc 2,837 about 3 years ago JDBC importer for Elasticsearch
ZomboDB 4,684 19 days ago Postgres Extension that allows creating an index backed by Elasticsearch

Awesome Data Engineering / Docker
Gockerize 667 over 6 years ago Package golang service into minimal Docker containers
Flocker 3,391 over 7 years ago Easily manage Docker containers & their data
Rancher RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers
Kontena Application Containers for Masses
Weave 6,618 3 months ago Weaving Docker containers into applications
Zodiac 198 almost 5 years ago A lightweight tool for easy deployment and rollback of dockerized applications
cAdvisor 17,182 22 days ago Analyzes resource usage and performance characteristics of running containers
Micro S3 persistence 14 about 5 years ago Docker microservice for saving/restoring volume data to S3
Rocker-compose 407 over 1 year ago Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated
Nomad 14,943 4 days ago Nomad is a cluster manager, designed for both long-lived services and short-lived batch processing workloads
ImageLayers Visualize Docker images and the layers that compose them

Awesome Data Engineering / Datasets / Realtime
Twitter Realtime The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data
Eventsim 505 over 2 years ago Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic
Reddit Real-time data is available including comments, submissions and links posted to reddit

Awesome Data Engineering / Datasets / Data Dumps
GitHub Archive GitHub's public timeline since 2011, updated every hour
Common Crawl Open source repository of web crawl data
Wikipedia Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available

Awesome Data Engineering / Monitoring / Prometheus
Prometheus.io 55,742 6 days ago An open-source service monitoring system and time series database
HAProxy Exporter 618 over 1 year ago Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption

Awesome Data Engineering / Profiling / Data Profiler
Data Profiler 1,434 8 days ago The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy

Awesome Data Engineering / Testing
Grai 295 30 days ago A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production
DQOps 111 8 days ago An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring
DataKitchen Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and auto-created data quality validation tests

Awesome Data Engineering / Community / Forums
/r/dataengineering News, tips, and background on Data Engineering
/r/etl Subreddit focused on ETL

Awesome Data Engineering / Community / Conferences
Data Council Data Council is the first technical conference that bridges the gap between data scientists, data engineers and data analysts

Awesome Data Engineering / Community / Podcasts
Data Engineering Podcast The show about modern data infrastructure
The Data Stack Show A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data

Backlinks from these awesome lists:

More related projects: