awesome-data-engineering
Data engineering toolkit
A curated list of tools and technologies for data engineering
A curated list of data engineering tools for software developers
7k stars
261 watching
1k forks
last commit: almost 2 years agoLinked from 5 awesome lists
awesomeawesome-list
Awesome Data Engineering / Databases / Relational | |||
| RQLite | 15,906 | almost 2 years ago | Replicated SQLite using the Raft consensus protocol |
| MySQL | The world's most popular open source database | ||
Awesome Data Engineering / Databases / Relational / MySQL | |||
| TiDB | 37,447 | almost 2 years ago | TiDB is a distributed NewSQL database compatible with MySQL protocol |
| Percona XtraBackup | Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB® | ||
| mysql_utils | 883 | about 7 years ago | Pinterest MySQL Management Tools |
Awesome Data Engineering / Databases / Relational | |||
| MariaDB | An enhanced, drop-in replacement for MySQL | ||
| PostgreSQL | The world's most advanced open source database | ||
| Amazon RDS | Amazon RDS makes it easy to set up, operate, and scale a relational database in the cloud | ||
| Crate.IO | Scalable SQL database with the NOSQL goodies | ||
Awesome Data Engineering / Databases / Key-Value | |||
| Redis | An open source, BSD licensed, advanced key-value cache and store | ||
| Riak | A distributed database designed to deliver maximum data availability by distributing data across multiple servers | ||
| AWS DynamoDB | A fast and flexible NoSQL database service for all applications that need consistent, single-digit millisecond latency at any scale | ||
| HyperDex | 1,393 | over 2 years ago | HyperDex is a scalable, searchable key-value store. Deprecated |
| SSDB | A high performance NoSQL database supporting many data structures, an alternative to Redis | ||
| Kyoto Tycoon | 277 | almost 3 years ago | Kyoto Tycoon is a lightweight network server on top of the Kyoto Cabinet key-value database, built for high-performance and concurrency |
| IonDB | 592 | over 2 years ago | A key-value store for microcontroller and IoT applications |
Awesome Data Engineering / Databases / Column | |||
| Cassandra | The right choice when you need scalability and high availability without compromising performance | ||
Awesome Data Engineering / Databases / Column / Cassandra | |||
| Cassandra Calculator | This simple form allows you to try out different values for your Apache Cassandra cluster and see what the impact is for your application | ||
| CCM | 1,219 | almost 2 years ago | A script to easily create and destroy an Apache Cassandra cluster on localhost |
| ScyllaDB | 13,725 | almost 2 years ago | NoSQL data store using the seastar framework, compatible with Apache Cassandra |
Awesome Data Engineering / Databases / Column | |||
| HBase | The Hadoop database, a distributed, scalable, big data store | ||
| AWS Redshift | A fast, fully managed, petabyte-scale data warehouse that makes it simple and cost-effective to analyze all your data using your existing business intelligence tools | ||
| FiloDB | 1,429 | almost 2 years ago | Distributed. Columnar. Versioned. Streaming. SQL |
| Vertica | Distributed, MPP columnar database with extensive analytics SQL | ||
| ClickHouse | Distributed columnar DBMS for OLAP. SQL | ||
Awesome Data Engineering / Databases / Document | |||
| MongoDB | An open-source, document database designed for ease of development and scaling | ||
Awesome Data Engineering / Databases / Document / MongoDB | |||
| Percona Server for MongoDB | Percona Server for MongoDB® is a free, enhanced, fully compatible, open source, drop-in replacement for the MongoDB® Community Edition that includes enterprise-grade features and functionality | ||
| MemDB | 596 | over 8 years ago | Distributed Transactional In-Memory Database (based on MongoDB) |
Awesome Data Engineering / Databases / Document | |||
| Elasticsearch | Search & Analyze Data in Real Time | ||
| Couchbase | The highest performing NoSQL distributed database | ||
| RethinkDB | The open-source database for the realtime web | ||
| RavenDB | Fully Transactional NoSQL Document Database | ||
Awesome Data Engineering / Databases / Graph | |||
| Neo4j | The world's leading graph database | ||
| OrientDB | 2nd Generation Distributed Graph Database with the flexibility of Documents in one product with an Open Source commercial friendly license | ||
| ArangoDB | A distributed free and open-source database with a flexible data model for documents, graphs, and key-values | ||
| Titan | A scalable graph database optimized for storing and querying graphs containing hundreds of billions of vertices and edges distributed across a multi-machine cluster | ||
| FlockDB | 3,337 | over 9 years ago | A distributed, fault-tolerant graph database by Twitter. Deprecated |
Awesome Data Engineering / Databases / Distributed | |||
| DAtomic | The fully transactional, cloud-ready, distributed database | ||
| Apache Geode | An open source, distributed, in-memory database for scale-out applications | ||
| Gaffer | 1,774 | almost 2 years ago | A large-scale graph database |
Awesome Data Engineering / Databases / Timeseries | |||
| InfluxDB | 29,126 | almost 2 years ago | Scalable datastore for metrics, events, and real-time analytics |
| OpenTSDB | 5,009 | almost 2 years ago | A scalable, distributed Time Series Database |
| QuestDB | A relational column-oriented database designed for real-time analytics on time series and event data | ||
| kairosdb | 1,740 | almost 2 years ago | Fast scalable time series database |
| Heroic | 848 | over 5 years ago | A scalable time series database based on Cassandra and Elasticsearch, by Spotify |
| Druid | 13,548 | almost 2 years ago | Column oriented distributed data store ideal for powering interactive applications |
| Riak-TS | Riak TS is the only enterprise-grade NoSQL time series database optimized specifically for IoT and Time Series data | ||
| Akumuli | 835 | about 4 years ago | Akumuli is a numeric time-series database. It can be used to capture, store and process time-series data in real-time. The word "akumuli" can be translated from esperanto as "accumulate" |
| Rhombus | A time-series object store for Cassandra that handles all the complexity of building wide row indexes | ||
| Dalmatiner DB | 694 | over 7 years ago | Fast distributed metrics database |
| Blueflood | 595 | about 2 years ago | A distributed system designed to ingest and process time series data |
| Timely | 379 | about 2 years ago | Timely is a time series database application that provides secure access to time series data based on Accumulo and Grafana |
Awesome Data Engineering / Databases / Other | |||
| Tarantool | 3,437 | almost 2 years ago | Tarantool is an in-memory database and application server |
| GreenPlum | The Greenplum Database (GPDB) - An advanced, fully featured, open source data warehouse. It provides powerful and rapid analytics on petabyte scale data volumes | ||
| cayley | 14,868 | almost 2 years ago | An open-source graph database. Google |
| Snappydata | 1,041 | almost 4 years ago | SnappyData: OLTP + OLAP Database built on Apache Spark |
| TimescaleDB | Built as an extension on top of PostgreSQL, TimescaleDB is a time-series SQL database providing fast analytics, scalability, with automated data management on a proven storage engine | ||
| DuckDB | DuckDB is a fast in-process analytical database that has zero external dependencies, runs on Linux/macOS/Windows, offers a rich SQL dialect, and is free and extensible | ||
Awesome Data Engineering / Data Comparison | |||
| datacompy | 487 | almost 2 years ago | DataComPy is a Python library that facilitates the comparison of two DataFrames in pandas, Polars, Spark and more. The library goes beyond basic equality checks by providing detailed insights into discrepancies at both row and column levels |
Awesome Data Engineering / Data Ingestion | |||
| Kafka | Publish-subscribe messaging rethought as a distributed commit log | ||
Awesome Data Engineering / Data Ingestion / Kafka | |||
| BottledWater | 4 | over 3 years ago | Change data capture from PostgreSQL into Kafka. Deprecated |
| kafkat | 504 | over 7 years ago | Simplified command-line administration for Kafka brokers |
| kafkacat | 5,468 | about 2 years ago | Generic command line non-JVM Apache Kafka producer and consumer |
| pg-kafka | 111 | over 11 years ago | A PostgreSQL extension to produce messages to Apache Kafka |
| librdkafka | 332 | almost 2 years ago | The Apache Kafka C/C++ library |
| kafka-docker | 6,943 | over 2 years ago | Kafka in Docker |
| kafka-manager | 11,853 | about 3 years ago | A tool for managing Apache Kafka |
| kafka-node | 2,664 | about 3 years ago | Node.js client for Apache Kafka 0.8 |
| Secor | 1,846 | almost 2 years ago | Pinterest's Kafka to S3 distributed consumer |
| Kafka-logger | 45 | almost 8 years ago | Kafka-winston logger for Node.js from Uber |
Awesome Data Engineering / Data Ingestion | |||
| AWS Kinesis | A fully managed, cloud-based service for real-time data processing over large, distributed data streams | ||
| RabbitMQ | Robust messaging for applications | ||
| dlt | A fast&simple pipeline building library for python data devs, runs in notebooks, cloud functions, airflow, etc | ||
| FluentD | An open source data collector for unified logging layer | ||
| Embulk | An open source bulk data loader that helps data transfer between various databases, storages, file formats, and cloud services | ||
| Apache Sqoop | A tool designed for efficiently transferring bulk data between Apache Hadoop and structured datastores such as relational databases | ||
| Heka | 3,389 | over 2 years ago | Data Acquisition and Processing Made Easy. Deprecated |
| Gobblin | 2,232 | almost 2 years ago | Universal data ingestion framework for Hadoop from LinkedIn |
| Nakadi | Nakadi is an open source event messaging platform that provides a REST API on top of Kafka-like queues | ||
| Pravega | Pravega provides a new storage abstraction - a stream - for continuous and unbounded data | ||
| Apache Pulsar | Apache Pulsar is an open-source distributed pub-sub messaging system | ||
| AWS Data Wrangler | 3,951 | almost 2 years ago | Utility belt to handle data on AWS |
| Airbyte | Open-source data integration for modern data teams | ||
| Artie | Real-time data ingestion tool leveraging change data capture | ||
| Sling | Sling is CLI data integration tool specialized in moving data between databases, as well as storage systems | ||
| Meltano | CLI & code-first ELT | ||
Awesome Data Engineering / Data Ingestion / Meltano | |||
| Singer SDK | The fastest way to build custom data extractors and loaders compliant with the Singer Spec | ||
Awesome Data Engineering / Data Ingestion | |||
| Google Sheets ETL | 18 | almost 2 years ago | Live import all your Google Sheets to your data warehouse |
Awesome Data Engineering / File System | |||
| HDFS | A distributed file system designed to run on commodity hardware | ||
Awesome Data Engineering / File System / HDFS | |||
| Snakebite | 854 | over 4 years ago | A pure python HDFS client |
Awesome Data Engineering / File System | |||
| AWS S3 | Object storage built to retrieve any amount of data from anywhere | ||
Awesome Data Engineering / File System / AWS S3 | |||
| smart_open | 3,233 | almost 2 years ago | Utils for streaming large files (S3, HDFS, gzip, bz2) |
Awesome Data Engineering / File System | |||
| Alluxio | Alluxio is a memory-centric distributed storage system enabling reliable data sharing at memory-speed across cluster frameworks, such as Spark and MapReduce | ||
| CEPH | Ceph is a unified, distributed storage system designed for excellent performance, reliability, and scalability | ||
| JuiceFS | 11,030 | almost 2 years ago | JuiceFS is a high-performance Cloud-Native file system driven by object storage for large-scale data storage |
| OrangeFS | Orange File System is a branch of the Parallel Virtual File System | ||
| SnackFS | 14 | about 11 years ago | SnackFS is our bite-sized, lightweight HDFS compatible file system built over Cassandra |
| GlusterFS | Gluster Filesystem | ||
| XtreemFS | Fault-tolerant distributed file system for all storage needs | ||
| SeaweedFS | 23,207 | almost 2 years ago | Seaweed-FS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word "NoSQL", you can call it as "NoFS" |
| S3QL | 1,126 | almost 2 years ago | S3QL is a file system that stores all its data online using storage services like Google Storage, Amazon S3, or OpenStack |
| LizardFS | LizardFS Software Defined Storage is a distributed, parallel, scalable, fault-tolerant, Geo-Redundant and highly available file system | ||
Awesome Data Engineering / Serialization format | |||
| Apache Avro | Apache Avro™ is a data serialization system | ||
| Apache Parquet | Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem, regardless of the choice of data processing framework, data model or programming language | ||
Awesome Data Engineering / Serialization format / Apache Parquet | |||
| Snappy | 6,217 | about 2 years ago | A fast compressor/decompressor. Used with Parquet |
| PigZ | A parallel implementation of gzip for modern multi-processor, multi-core machines | ||
Awesome Data Engineering / Serialization format | |||
| Apache ORC | The smallest, fastest columnar storage for Hadoop workloads | ||
| Apache Thrift | The Apache Thrift software framework, for scalable cross-language services development | ||
| ProtoBuf | 65,999 | almost 2 years ago | Protocol Buffers - Google's data interchange format |
| SequenceFile | SequenceFile is a flat file consisting of binary key/value pairs. It is extensively used in MapReduce as input/output formats | ||
| Kryo | 6,217 | almost 2 years ago | Kryo is a fast and efficient object graph serialization framework for Java |
Awesome Data Engineering / Stream Processing | |||
| Apache Beam | Apache Beam is a unified programming model that implements both batch and streaming data processing jobs that run on many execution engines | ||
| Spark Streaming | Spark Streaming makes it easy to build scalable fault-tolerant streaming applications | ||
| Apache Flink | Apache Flink is a streaming dataflow engine that provides data distribution, communication, and fault tolerance for distributed computations over data streams | ||
| Apache Storm | Apache Storm is a free and open source distributed realtime computation system | ||
| Apache Samza | Apache Samza is a distributed stream processing framework | ||
| Apache NiFi | An easy to use, powerful, and reliable system to process and distribute data | ||
| Apache Hudi | An open source framework for managing storage for real time processing, one of the most interesting feature is the Upsert | ||
| VoltDB | VoltDb is an ACID-compliant RDBMS which uses a | ||
| PipelineDB | 2,639 | over 4 years ago | The Streaming SQL Database |
| Spring Cloud Dataflow | Streaming and tasks execution between Spring Boot apps | ||
| Bonobo | Bonobo is a data-processing toolkit for python 3.5+ | ||
| Robinhood's Faust | 1,675 | almost 2 years ago | Forever scalable event processing & in-memory durable K/V store as a library with asyncio & static typing |
| HStreamDB | 713 | almost 2 years ago | The streaming database built for IoT data storage and real-time processing |
| Kuiper | 1,505 | almost 2 years ago | An edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices |
| Zilla | 553 | almost 2 years ago | - An API gateway built for event-driven architectures and streaming that supports standard protocols such as HTTP, SSE, gRPC, MQTT, and the native Kafka protocol |
| SwimOS | 321 | almost 2 years ago | A framework for building real-time streaming data processing applications that supports a wide range of ingestion sources |
Awesome Data Engineering / Batch Processing | |||
| Hadoop MapReduce | Hadoop MapReduce is a software framework for easily writing applications which process vast amounts of data (multi-terabyte data-sets) - in-parallel on large clusters (thousands of nodes) - of commodity hardware in a reliable, fault-tolerant manner | ||
| Spark | A multi-language engine for executing data engineering, data science, and machine learning on single-node machines or clusters | ||
Awesome Data Engineering / Batch Processing / Spark | |||
| Spark Packages | A community index of packages for Apache Spark | ||
| Deep Spark | 197 | about 10 years ago | Connecting Apache Spark with different data stores. Deprecated |
| Spark RDD API Examples | Examples by Zhen He | ||
| Livy | The REST Spark Server | ||
| Delight | 344 | over 2 years ago | A free & cross platform monitoring tool (Spark UI / Spark History Server alternative) |
Awesome Data Engineering / Batch Processing | |||
| AWS EMR | A web service that makes it easy to quickly and cost-effectively process vast amounts of data | ||
| Data Mechanics | A cloud-based platform deployed on Kubernetes making Apache Spark more developer-friendly and cost-effective | ||
| Tez | An application framework which allows for a complex directed-acyclic-graph of tasks for processing data | ||
| Bistro | 7 | about 8 years ago | A light-weight engine for general-purpose data processing including both batch and stream analytics. It is based on a novel unique data model, which represents data via and processes data via as opposed to having only set operations in conventional approaches like MapReduce or SQL |
Awesome Data Engineering / Batch Processing / Batch ML | |||
| H2O | Fast scalable machine learning API for smarter applications | ||
| Mahout | An environment for quickly creating scalable performant machine learning applications | ||
| Spark MLlib | Spark's scalable machine learning library consisting of common learning algorithms and utilities, including classification, regression, clustering, collaborative filtering, dimensionality reduction, as well as underlying optimization primitives | ||
Awesome Data Engineering / Batch Processing / Batch Graph | |||
| GraphLab Create | A machine learning platform that enables data scientists and app developers to easily create intelligent apps at scale | ||
| Giraph | An iterative graph processing system built for high scalability | ||
| Spark GraphX | Apache Spark's API for graphs and graph-parallel computation | ||
Awesome Data Engineering / Batch Processing / Batch SQL | |||
| Presto | A distributed SQL query engine designed to query large data sets distributed over one or more heterogeneous data sources | ||
| Hive | Data warehouse software facilitates querying and managing large datasets residing in distributed storage | ||
Awesome Data Engineering / Batch Processing / Batch SQL / Hive | |||
| Hivemall | 311 | about 4 years ago | Scalable machine learning library for Hive/Hadoop |
| PyHive | 1,676 | about 2 years ago | Python interface to Hive and Presto |
Awesome Data Engineering / Batch Processing / Batch SQL | |||
| Drill | Schema-free SQL Query Engine for Hadoop, NoSQL and Cloud Storage | ||
Awesome Data Engineering / Charts and Dashboards | |||
| Highcharts | A charting library written in pure JavaScript, offering an easy way of adding interactive charts to your web site or web application | ||
| ZingChart | Fast JavaScript charts for any data set | ||
| C3.js | D3-based reusable chart library | ||
| D3.js | A JavaScript library for manipulating documents based on data | ||
Awesome Data Engineering / Charts and Dashboards / D3.js | |||
| D3Plus | D3's simpler, easier to use cousin. Mostly predefined templates that you can just plug data in | ||
Awesome Data Engineering / Charts and Dashboards | |||
| SmoothieCharts | A JavaScript Charting Library for Streaming Data | ||
| PyXley | 2,272 | over 8 years ago | Python helpers for building dashboards using Flask and React |
| Plotly | 21,641 | almost 2 years ago | Flask, JS, and CSS boilerplate for interactive, web-based visualization apps in Python |
| Apache Superset | 63,320 | almost 2 years ago | Apache Superset (incubating) - A modern, enterprise-ready business intelligence web application |
| Redash | Make Your Company Data Driven. Connect to any data source, easily visualize and share your data | ||
| Metabase | 39,103 | almost 2 years ago | Metabase is the easy, open source way for everyone in your company to ask questions and learn from data |
| PyQtGraph | PyQtGraph is a pure-python graphics and GUI library built on PyQt4 / PySide and numpy. It is intended for use in mathematics / scientific / engineering applications | ||
Awesome Data Engineering / Workflow | |||
| Luigi | 17,950 | almost 2 years ago | Luigi is a Python module that helps you build complex pipelines of batch jobs |
| CronQ | An application cron-like system. w/Luige. Deprecated | ||
| Cascading | Java based application development platform | ||
| Airflow | 37,580 | almost 2 years ago | Airflow is a system to programmatically author, schedule, and monitor data pipelines |
| Azkaban | Azkaban is a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy-to-use web user interface to maintain and track your workflows | ||
| Oozie | Oozie is a workflow scheduler system to manage Apache Hadoop jobs | ||
| Pinball | 1,046 | almost 7 years ago | DAG based workflow manager. Job flows are defined programmatically in Python. Support output passing between jobs |
| Dagster | 12,055 | almost 2 years ago | Dagster is an open-source Python library for building data applications |
| Hamilton | 1,900 | almost 2 years ago | Hamilton is a lightweight library to define data transformations as a directed-acyclic graph (DAG). If you like dbt for SQL transforms, you will like Hamilton for Python processing |
| Kedro | Kedro is a framework that makes it easy to build robust and scalable data pipelines by providing uniform project templates, data abstraction, configuration and pipeline assembly | ||
| Dataform | An open-source framework and web based IDE to manage datasets and their dependencies. SQLX extends your existing SQL warehouse dialect to add features that support dependency management, testing, documentation and more | ||
| Census | A reverse-ETL tool that let you sync data from your cloud data warehouse to SaaS applications like Salesforce, Marketo, HubSpot, Zendesk, etc. No engineering favors required—just SQL | ||
| dbt | A command line tool that enables data analysts and engineers to transform data in their warehouses more effectively | ||
| Kestra | Scalable, event-driven, language-agnostic orchestration and scheduling platform to manage millions of workflows declaratively in code | ||
| RudderStack | 4,109 | almost 2 years ago | A warehouse-first Customer Data Platform that enables you to collect data from every application, website and SaaS platform, and then activate it in your warehouse and business tools |
| PACE | 34 | almost 2 years ago | An open source framework that allows you to enforce agreements on how data should be accessed, used, and transformed, regardless of the data platform (Snowflake, BigQuery, DataBricks, etc.) |
| Prefect | Prefect is an orchestration and observability platform. With it, developers can rapidly build and scale resilient code, and triage disruptions effortlessly | ||
| Multiwoven | 1,556 | almost 2 years ago | The open-source reverse ETL, data activation platform for modern data teams |
| SuprSend | Create automated workflows and logic using API's for your notification service. Add templates, batching, preferences, inapp inbox with workflows to trigger notifications directly from your data warehouse | ||
| Kestra | 14,708 | almost 2 years ago | A versatile open source orchestrator and scheduler built on Java, designed to handle a broad range of workflows with a language-agnostic, API-first architecture |
| Mage | Open-source data pipeline tool for transforming and integrating data | ||
Awesome Data Engineering / Data Lake Management | |||
| lakeFS | 4,496 | almost 2 years ago | lakeFS is an open source platform that delivers resilience and manageability to object-storage based data lakes |
| Project Nessie | 1,064 | almost 2 years ago | Project Nessie is a Transactional Catalog for Data Lakes with Git-like semantics. Works with Apache Iceberg tables |
Awesome Data Engineering / ELK Elastic Logstash Kibana | |||
| docker-logstash | 236 | almost 11 years ago | A highly configurable Logstash (1.4.4) - Docker image running Elasticsearch (1.7.0) - and Kibana (3.1.2) |
| elasticsearch-jdbc | 2,838 | almost 5 years ago | JDBC importer for Elasticsearch |
| ZomboDB | 4,687 | almost 2 years ago | Postgres Extension that allows creating an index backed by Elasticsearch |
Awesome Data Engineering / Docker | |||
| Gockerize | 666 | over 8 years ago | Package golang service into minimal Docker containers |
| Flocker | 3,390 | over 9 years ago | Easily manage Docker containers & their data |
| Rancher | RancherOS is a 20mb Linux distro that runs the entire OS as Docker containers | ||
| Kontena | Application Containers for Masses | ||
| Weave | 6,621 | about 2 years ago | Weaving Docker containers into applications |
| Zodiac | 198 | over 6 years ago | A lightweight tool for easy deployment and rollback of dockerized applications |
| cAdvisor | 17,304 | almost 2 years ago | Analyzes resource usage and performance characteristics of running containers |
| Micro S3 persistence | 14 | almost 7 years ago | Docker microservice for saving/restoring volume data to S3 |
| Rocker-compose | 406 | over 3 years ago | Docker composition tool with idempotency features for deploying apps composed of multiple containers. Deprecated |
| Nomad | 15,029 | almost 2 years ago | Nomad is a cluster manager, designed for both long-lived services and short-lived batch processing workloads |
| ImageLayers | Visualize Docker images and the layers that compose them | ||
Awesome Data Engineering / Datasets / Realtime | |||
| Twitter Realtime | The Streaming APIs give developers low latency access to Twitter's global stream of Tweet data | ||
| Eventsim | 508 | over 4 years ago | Event data simulator. Generates a stream of pseudo-random events from a set of users, designed to simulate web traffic |
| Real-time data is available including comments, submissions and links posted to reddit | |||
Awesome Data Engineering / Datasets / Data Dumps | |||
| GitHub Archive | GitHub's public timeline since 2011, updated every hour | ||
| Common Crawl | Open source repository of web crawl data | ||
| Wikipedia | Wikipedia's complete copy of all wikis, in the form of Wikitext source and metadata embedded in XML. A number of raw database tables in SQL form are also available | ||
Awesome Data Engineering / Monitoring / Prometheus | |||
| Prometheus.io | 56,244 | almost 2 years ago | An open-source service monitoring system and time series database |
| HAProxy Exporter | 619 | over 3 years ago | Simple server that scrapes HAProxy stats and exports them via HTTP for Prometheus consumption |
Awesome Data Engineering / Profiling / Data Profiler | |||
| Data Profiler | 1,442 | almost 2 years ago | The DataProfiler is a Python library designed to make data analysis, monitoring, and sensitive data detection easy |
Awesome Data Engineering / Testing | |||
| Grai | 301 | almost 2 years ago | A data catalog tool that integrates into your CI system exposing downstream impact testing of data changes. These tests prevent data changes which might break data pipelines or BI dashboards from making it to production |
| DQOps | 118 | almost 2 years ago | An open-source data quality platform for the whole data platform lifecycle from profiling new data sources to applying full automation of data quality monitoring |
| DataKitchen | Open Source Data Observability for end-to-end Data Journey Observability, data profiling, anomaly detection, and auto-created data quality validation tests | ||
Awesome Data Engineering / Community / Forums | |||
| /r/dataengineering | News, tips, and background on Data Engineering | ||
| /r/etl | Subreddit focused on ETL | ||
Awesome Data Engineering / Community / Conferences | |||
| Data Council | Data Council is the first technical conference that bridges the gap between data scientists, data engineers and data analysts | ||
Awesome Data Engineering / Community / Podcasts | |||
| Data Engineering Podcast | The show about modern data infrastructure | ||
| The Data Stack Show | A show where they talk to data engineers, analysts, and data scientists about their experience around building and maintaining data infrastructure, delivering data and data products, and driving better outcomes across their businesses with data | ||
sindresorhus/awesome
jnv/lists
mark-rushakoff/awesome-influxdb
kelvins/awesome-dataops
0ex/more-awesome
morganstanley/modern-cpp-kafka
karafka/karafka
bookingcom/perl-net-kafka
didier-wenzek/ocaml-kafka
icemachined/kafka-kotlin-native
turbolytics/sql-flow
fede1024/rust-rdkafka
seaweedfs/seaweedfs-operator
tobymao/sqlglot