awesome-etl

Workflow management software

A curated collection of notable ETL frameworks and tools for managing workflows and data pipelines.

A curated list of awesome ETL frameworks, libraries, and software.

GitHub
3k stars
155 watching
344 forks
last commit: 4 months ago
Linked from 4 awesome lists


awesome-pipeline 6,204 about 1 month ago

awesome-etl / Workflow Management/Engines
Airflow 37,120 6 days ago "Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed."
Azkaban "a batch workflow job scheduler created at LinkedIn to run Hadoop jobs. Azkaban resolves the ordering through job dependencies and provides an easy to use web user interface to maintain and track your workflows."
Dray.it "Docker workflow engine. Allows users to separate a workflow into discrete steps each to be handled by a single container."
Luigi 17,869 9 days ago "a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in."
Mara Pipelines 2,081 11 months ago "A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow"
Pinball 1,047 almost 5 years ago "a scalable workflow management platform developed at Pinterest. It is built based on layered approach."
prefect 17,407 6 days ago "a new workflow management system, designed for modern infrastructure and powered by the open-source Prefect Core workflow engine. Users organize Tasks into Flows, and Prefect takes care of the rest."
TaskFlow "allows the creation of lightweight task objects and/or functions that are combined together into flows (aka: workflows) in a declarative manner. It includes engines for running these flows in a manner that can be stopped, resumed, and safely reverted."
Toil Similar to Luigi, jobs are classes with a run method. Supports executing jobs on other machines (workers) which can include AWS spot instances
Argo Container based workflow management system for Kubernetes. Workflows are specified as a directed acyclic graph (DAG), and each step is executed on a container, and the latter is run on a Kubernetes Pod. There is also support for Airflow DAGs
Dagster "Dagster is a data orchestrator for machine learning, analytics, and ETL. It lets you define pipelines in terms of the data flow between reusable, logical components, then test locally and run anywhere. With a unified view of pipelines and the assets they produce, Dagster can schedule and orchestrate Pandas, Spark, SQL, or anything else that Python can invoke."

awesome-etl / Job Scheduling
Chronos 4,390 over 2 years ago "a distributed and fault-tolerant scheduler that runs on top of Apache Mesos that can be used for job orchestration."
Dagobah 755 over 5 years ago "a simple dependency-based job scheduler written in Python. Dagobah allows you to schedule periodic jobs using Cron syntax. Each job then kicks off a series of tasks (subprocesses) in an order defined by a dependency graph you can easily draw with click-and-drag in the web interface."
Jenkins 23,258 4 days ago "the leading open-source automation server. Built with Java, it provides over 1000 plugins to support automating virtually anything, so that humans can actually spend their time doing things machines cannot."

awesome-etl / Java
GETL 56 11 months ago Groovy toolbox for ETL Tasks from practicing architectures
JSR 352 Java native API for batch processing
Scriptella Java-XML ETL toolbox for every day use
Spring Batch ETL on Spring ecosystem

awesome-etl / Python / Libraries
BeautifulSoup Popular library used to extract data from web pages
Blaze 3,187 about 1 year ago "translates a subset of modified NumPy and Pandas-like syntax to databases and other computing systems."
Bonobo Simple, modern and atomic data transformation graphs for Python 3.5+
Celery "an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well."
Dask 12,593 6 days ago Ever tried using Pandas to process data that won't fit into memory? Dask makes it easy. Dask also has functionality to make it easy to processing continuous streams of data
dataset A wrapper around SQLAlchemy that simplifies database operations (including upserting)
ijson 845 27 days ago Allows processing JSON iteratively (as a stream) without loading the whole file into memory at once
Joblib "a set of tools to provide lightweight pipelining in Python."
lxml 2,705 16 days ago Parses XML using C libraries libxml2 and libxslt, so it's very fast. Also supports a "recover" mode that will try its best to use invalid xml or discard it. Great for large XML files and advanced functionality (like using xpaths). IBM also has a great article on high-performance parsing with lxml here:
MrJob "lets you write MapReduce jobs in Python 2.6+ and run them on several platforms. The easiest route to writing Python programs that run on Hadoop."
Odo 1,004 over 2 years ago Moves data across containers (SQL, CSV, MongoDB, Pandas, etc). Claims to be the easiest and fastest way to load a CSV into your database
Pandas Implements dataframes in Python for easier data processing and includes a number of tools that make it easier to extract data from multiple file formats
parse 1,713 5 days ago The opposite of Python's format(). Easier to use than regex, but more limited
PETL 1,250 6 months ago "a general purpose Python package for extracting, transforming and loading tables of data." Slower than Pandas and not as good for larger amounts of data, but simpler
PyQuery Extracts data from web pages with a jquery-like syntax
polars 30,400 4 days ago Dataframes powered by a multithreaded, vectorized query engine, written in Rust
Retrying 1,917 over 3 years ago Allows you to add a decorator to any function/method to retry on an exception
Requests-HTML 303 5 months ago Combines PyQuery, Requests, parse, and other libraries for a pleasant and intuitive web scraping experience
riko 1,604 almost 3 years ago A python stream processing engine modeled after Yahoo! Pipes
Ruffus "The Ruffus module is a lightweight way to add support for running computational pipelines."
SQLAlchemy "the Python SQL toolkit and Object Relational Mapper that gives application developers the full power and flexibility of SQL."
Toolz "A functional standard library for python." Includes a function that allows you to pipe a value through a sequence of functions. There's also a cython implementation here:
xmltodict 5,508 about 1 month ago Makes working with XML as easy as working with JSON. Also allows streaming so you don't run out of memory on large XML files. Great for simple operations on small XML files

awesome-etl / Python / Talks/Articles
http://www.parsely.com/misc/slides/streamparse/notes/

awesome-etl / Ruby
Kiba 1,754 13 days ago "Data processing & ETL framework for Ruby"
nokogiri 6,153 7 days ago an excellent XML parser that "just works"
Sequel 4,994 7 days ago "The Database Toolkit for Ruby"
Embulk 1,755 about 2 months ago "a parallel bulk data loader that helps data transfer between various storages, databases, NoSQL and cloud services."

awesome-etl / Go
Benthos "The stream processor for mundane tasks."
Crunch 214 about 10 years ago "A fast to develop, fast to run, Go based toolkit for ETL and feature extraction on Hadoop."
Pachyderm 6,179 6 days ago A system for running processing pipeline jobs in containers and version controlling all data using a commit-based distributed filesystem
CloudQuery 5,877 6 days ago An open source high performance ELT Framework

awesome-etl / Javascript
Datapumps 291 over 6 years ago "Use pumps to import, export, transform or transfer data."
NoFlo "a JavaScript implementation of Flow-Based Programming"

awesome-etl / Talks/Articles
https://medium.com/@samson_hu/building-analytics-at-500px-92e9a7005c83
http://www.slideshare.net/g33ktalk/data-pipeline-acial-lyceum20140624
http://chairnerd.seatgeek.com/building-out-the-seatgeek-data-pipeline/
http://www.garynissen.com/etl-hand-code-or-tool/
http://www.slideshare.net/CasertaConcepts/big-data-warehousing-meetup-bigetl-trad-tool-vs-pig-vs-hive-vs-python-what-to-use-when-slide-set-2
https://deepfriedcode.com/books/darps/index.html
http://blog.cloudera.com/wp-content/uploads/2010/01/IntroToPig.pdf
http://tech.adroll.com/blog/data/2015/10/15/luigi.html?adrolldev

awesome-etl / Cloud Services
Alteryx Cloud ETL tool with an interface similar to GUI ETL tools
AWS Data Pipeline "a web service that helps you reliably process and move data between different AWS compute and storage services, as well as on-premise data sources, at specified intervals."
AWS Glue AWS Glue generates the code (using Python and Spark) to execute your data transformations and data loading processes
Amazon Simple Workflow Service (SWF) "helps developers build, run, and scale background jobs that have parallel or sequential steps. You can think of Amazon SWF as a fully-managed state tracker and task coordinator in the Cloud."
AWS Batch Allows executing jobs as containerized applications running on Amazon ECS. Also includes features for dynamically bidding for Spot Instances, integration with existing workflow engines, scheduling, monitoring, dependency modeling, and dynamic scaling/provisioning based on amount of work
Google Dataflow "Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines."
Cloud Data Fusion "Fully managed, cloud-native data integration platform."
Stitch "Stitch is a cloud-first, open source platform for rapidly moving data. A simple, powerful ETL service, Stitch connects to all your data sources – from databases like MySQL and MongoDB, to SaaS applications like Salesforce and Zendesk – and replicates that data to a destination of your choosing."
Hevo Hevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services

awesome-etl / Big Data (Hadoop Stack)
Pig "a platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs, coupled with infrastructure for evaluating these programs."
Spark "a fast and general-purpose cluster computing system. It provides high-level APIs in Scala, Java, and Python that make parallel jobs easy to write, and an optimized engine that supports general computation graphs. It also supports a rich set of higher-level tools including Shark (Hive on Spark), MLlib for machine learning, GraphX for graph processing, and Spark Streaming."

awesome-etl / ETL Tools (GUI)
Apache NiFi "a rich, web-based interface for designing, controlling, and monitoring a dataflow."
Informatica PowerCenter "a toolset for establishing and maintaining enterprise-wide data warehouses. It has a customer base of over 5,000 companies."
Microsoft SSIS "a component of the Microsoft SQL Server database software that can be used to perform a broad range of data migration tasks."
Pentaho Kettle The most popular open-source graphical ETL tool
Talend "an open source application for data integration job design with a graphical development environment"
N8n 48,956 4 days ago "Free and open fair-code licensed node based Workflow Automation Tool. Easily automate tasks across different services."
CDAP "Use Cask Data Application Platform to visually build and manage data applications in hybrid and multi-cloud environments."

Backlinks from these awesome lists:

More related projects: