awesome-pipeline

Workflow toolkit collection

A curated collection of workflow toolkits for managing complex processes and data pipelines.

A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin

GitHub

6k stars
231 watching
626 forks
last commit: almost 2 years ago
Linked from 10 awesome lists

awesome-listworkflow

Awesome Pipeline / Pipeline frameworks & libraries

ActionChainA workflow system for simple linear success/failure workflows
Adage56over 3 years agoSmall package to describe workflows that are not completely known at definition time
AiiDA440almost 2 years agoworkflow manager with a strong focus on provenance, performance and extensibility
Airflow37,580almost 2 years agoPython-based workflow system created by AirBnb
AndurilComponent-based workflow framework for scientific data analysis
AnthaHigh-level language for biology
Argo WorkflowsContainer-native workflow engine for orchestrating parallel data processing, ML, or CI jobs on Kubernetes
AutosubmitAn open source Python experiment and workflow manager used to manage complex workflows on Cloud and HPC platforms
AWE69almost 6 years agoWorkflow and resource management system with CWL support
Balsam77over 2 years agoPython-based high throughput task and workflow engine
BdsScripting language for data pipelines
BeamUnified programming model for batch and streaming data-parallel processing pipelines
BioMake103almost 3 years agoGNU-Make-like utility for managing builds and complex workflows
BioQueue29over 3 years agoExplicit framework with web monitoring and resource estimation
Bioshake55about 7 years agoHaskell DSL built on shake with strong typing and EDAM support
Bistro48over 2 years agoLibrary to build and execute typed scientific workflows
Bpipe233almost 2 years agoTool for running and managing bioinformatics pipelines
Briefly106about 8 years agoPython Meta-programming Library for Job Flow Control
Burr1,368almost 2 years agoPython based lightweight graph (i.e. can do loops and conditional branching, and not just DAGs) orchestrator
Cluster FlowCommand-line tool which uses common cluster managers to run bioinformatics pipelines
Clusterjob20over 2 years agoAutomated reproducibility, and hassle-free submission of computational jobs to clusters
CompiApplication framework for portable computational pipelines
CompssProgramming model for distributed infrastructures
Conan23about 12 years agoLight-weight workflow management application
Consecution169over 5 years agoA Python pipeline abstraction inspired by Apache Storm topologies
CosmosPython library for massively parallel workflows
Couler919almost 2 years agoUnified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow
Covalent785almost 2 years agoWorkflow orchestration toolkit for high-performance and quantum computing research and development
Cromwell1,004almost 2 years agoWorkflow Management System geared towards scientific workflows from the Broad Institute
Cuneiform232about 3 years agoAdvanced functional workflow language and framework, implemented in Erlang
CylcA workflow engine for cycling systems, originally developed for operational environmental forecasting
Dagobah757about 7 years agoSimple DAG-based job scheduler in Python
Dagr69over 4 years agoA scala based DSL and framework for writing and executing bioinformatics pipelines as Directed Acyclic Graphs
Dagster12,055almost 2 years agoPython-based API for defining DAGs that interfaces with popular workflow managers for building data applications
DataJointan open-source relational framework for scientific data pipelines
Dask12,691almost 2 years agoDask is a flexible parallel computing library for analytics
DbtFramework for writing analytics workflows entirely in SQL. The T part of ETL, focuses on analytics engineering
Dockerflow97almost 9 years agoWorkflow runner that uses Dataflow to run a series of tasks in Docker
Drake1,480over 4 years agoRobust DSL akin to Make, implemented in Clojure
Drake R package1,343almost 2 years agoReproducibility and high-performance computing with an easy R-focused interface. Unrelated to . Succeeded by
Dray383over 6 years agoAn engine for managing the execution of container-based workflows
ecFlow41almost 2 years agoWorkflow manager
eHive53almost 2 years agoSystem for creating and running pipelines on a distributed compute resource
Fission Workflows371over 3 years agoA fast, lightweight workflow engine for serverless/FaaS functions
Flex56over 5 years agoLanguage agnostic framework for building flexible data science pipelines (Python/Shell/Gnuplot)
Flowr84over 5 years agoRobust and efficient workflows using a simple language agnostic approach (R package)
Gc3pie44over 3 years agoPython libraries and tools for running applications on diverse Grids and clusters
Guix Workflow LanguageA workflow management language extension for GNU Guix
Gwf31almost 2 years agoMake-like utility for submitting workflows via qsub
Hamilton1,900almost 2 years agoA python micro-framework for describing dataflows; runs anywhere python runs
Hera621almost 2 years agoHera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo
HyperLoom16about 4 years agoPlatform for defining and executing workflow pipelines in large-scale distributed environments
HyperQueue292almost 2 years agoHPC-focused task scheduler that automatically assigns tasks to Slurm/PBS allocations and submits them for the user
JoblibSet of tools to provide lightweight pipelining in Python
JugA task Based parallelization framework for Python
Kedro10,050almost 2 years agoWorkflow development tool that helps you build data pipelines
Kestra14,708almost 2 years agoOpen source data orchestration and scheduling platform with declarative syntax
Ketrew77over 8 years agoEmbedded DSL in the OCAML language alongside a client-server management application
https://github.com/Nike-Inc/koheesio][ - Python framework for building efficient data pipelines
Kronos19almost 10 years agoWorkflow assembler for cancer genome analytics and informatics
Kubeflow PipelinesFramework for building and deploying portable, scalable machine learning workflows using Docker containers and Argo Workflows
Loom29almost 7 years agoTool for running bioinformatics workflows locally or in the cloud
LongbowJob proxying tool for biomolecular simulations
Luigi17,950almost 2 years agoPython module that helps you build complex pipelines of batch jobs
Maestro139almost 2 years agoYAML based HPC workflow execution tool
MakeflowWorkflow engine for executing large complex workflows on clusters
makepipe31almost 4 years agoAn R package which provides a set of simple tools for transforming an existing workflow into a self-documenting pipeline with very minimal upfront costs
Mara2,082almost 3 years agoA lightweight, opinionated ETL framework, halfway between plain scripts and Apache Airflow
Mario139over 8 years agoScala library for defining data pipelines
MartianA language and framework for developing and executing complex computational pipelines
MD Studio12almost 7 years agoMicroservice based workflow engine
MetaFlowOpen-sourced framework from Netflix, for DAG generation for data scientists. Python and R API's
Mistral291almost 2 years agoPython based workflow engine by the Open Stack project
Moa23almost 12 years agoLightweight workflows in bioinformatics
NextflowFlow-based computational toolkit for reproducible and scalable bioinformatics pipelines
nFlow206almost 2 years agoEmbeddable JVM-based workflow engine with high availability, fault tolerance, and support for multiple databases. Additional libraries are provided for visualization and REST API
NiPype750almost 2 years agoWorkflows and interfaces for neuroimaging packages
OpenGE26about 13 years agoAccelerated framework for manipulating and interpreting high-throughput sequencing data
PachydermDistributed and reproducible data pipelining and data management, built on the container ecosystem
ParslProductive parallel programming, for creating parallel programs composed of Python functions and external components
PipeFunc230almost 2 years agoLightweight function pipeline (DAG) creation in pure Python for scientific workflows
PipEngine20about 9 years agoRuby based launcher for complex biological pipelines
Pinball1,046almost 7 years agoPython based workflow engine by Pinterest
Popper304over 4 years agoYAML based container-native workflow engine supporting Docker, Singularity, Vagrant VMs with Docker daemon in VM, and local host
Porcupine89over 4 years agoHaskell workflow tool to express and compose tasks (optionally cached) whose datasources and sinks are known ahead of time and rebindable, and which can expose arbitrary sets of parameters to the outside world
PrefectPython based workflow engine powering Prefect
Pydra123almost 2 years agoLightweight, DAG-based Python dataflow engine for reproducible and scalable scientific pipelines
PyFlow146about 6 years agoLightweight parallel task engine
pyperator60over 9 years agoSimple push-based python workflow framework using asyncio, supporting recursive networks
pyppl105about 2 years agoA python lightweight pipeline framework
pypyrAutomation task-runner for sequential steps defined in a pipeline yaml, with AWS and Slack plug-ins
pytask115almost 2 years agoA workflow management system that facilitates reproducible data analyses
Pwrake57over 6 years agoParallel workflow extension for Rake
QdoLightweight high-throughput queuing system for workflows with many small tasks to perform
Qsubsec10over 3 years agoSimple tokenised template system for SGE
Rabix106over 7 years agoPython-based workflow toolkit based on the Common Workflow Language and Docker
Rain749over 3 years agoFramework for large distributed task-based pipelines, written in Rust with Python API
Ray34,412almost 2 years agoFlexible, high-performance distributed Python execution framework
Redun537about 2 years agoYet another redundant workflow engine
Reflow965almost 3 years agoLanguage and runtime for distributed, incremental data processing in the cloud
Remake340over 8 years agoMake-like declarative workflows in R
RmakeWrapper for the creation of Makefiles, enabling massive parallelization
Rubra38over 11 years agoPipeline system for bioinformatics workflows
RuffusComputation Pipeline library for Python
Ruigi42over 7 years agoPipeline tool for R, inspired by Luigi
SakeSelf-documenting build automation tool
SciLuigi335almost 2 years agoHelper library for writing flexible scientific workflows in Luigi
SciPipeLibrary for writing Scientific Workflows in Go
SignacLightweight, but scalable framework for file-driven workflows to be run locally and on HPC systems
Scoop642over 3 years agoScalable Concurrent Operations in Python
Seqtools49over 2 years agoPython library for lazy evaluation of pipelined transformations on indexable containers
SmartPipeline25over 2 years agoA framework for rapid development of robust data pipelines following a simple design pattern
SnakemakeTool for running and managing bioinformatics pipelines
Spiff1,713almost 2 years agoBased on the Workflow Patterns initiative and implemented in Python
Stolos130about 8 years agoDirected Acyclic Graph task dependency scheduler that simplify distributed pipelines
Steppy134almost 8 years agolightweight, open-source, Python 3 library for fast and reproducible experimentation. (This repository has been archived by the owner on Jun 22, 2022.)
StpipeFile processing pipelines as a Python library
StreamFlow54almost 2 years agoContainer native workflow management system focused on hybrid workflows
StreamPipesA self-service IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams
SundialJobsystem on AWS ECS or AWS Batch managing dependencies and scheduling
Suro794over 3 years agoJava-based distributed pipeline from Netflix
SwiftFast easy parallel scripting - on multicores, clusters, clouds and supercomputers
TAF3almost 2 years agoR package to organize reproducible scientific workflows
Targets949almost 2 years agoDynamic, function-oriented -like reproducible pipelines at scale in R
TaskGraph22over 2 years agoA library to help manage complicated computational software pipelines consisting of long running individual tasks
Tibanna70almost 2 years agoTool that helps you run genomic pipelines on Amazon cloud
Toil901almost 2 years agoDistributed pipeline workflow manager (mostly for genomics)
YapExtensible parallel framework, written in Python using OpenMPI libraries
Yapp62almost 4 years agoA C++ parallel pipeline library for stream processing
WallarooFramework for streaming data applications and algorithms that react to real-time events
WorldMakeEasy Collaborative Reproducible Computing
ZenatonWorkflow engine for orchestrating jobs, data and events across your applications and third party services
ZenMLExtensible open-source MLOps framework to create reproducible pipelines for data scientists

Awesome Pipeline / Workflow platforms

ActivePapersComputational science made reproducible and publishable
Active Workflow845over 3 years agoPolyglot workflows without leaving the comfort of your technology stack
Anvi’oA community and framework centered around metagenomics, designed to facilitate reproducible exploration and visualization of data
Apache IravataFramework for executing and managing computational workflows on distributed computing resources
ArteriaEvent-driven automation for sequencing centers. Initiates workflows based on events
ArvadosA container based workflow platform
inactive since 10/2019Biokepler - Bioinformatics Scientific Workflow for Distributed Analysis of Large-Scale Biological Data. ( )
ButlerFramework for running scientific workflows on public and academic clouds
ChipsterOpen source platform for data analysis
ClubberCluster Load Balancer for Bioinformatics e-Resources
DigdagWorkflow manager designed for simplicity, extensibility and collaboration
Domino155about 2 years agoUser friendly and open source visual workflow management platform
Fireworks367about 2 years agoCentralized workflow server for dynamic workflows of high-throughput computations
Flojoy210almost 2 years agoOpen source visual Python scripting for test, measurement, and robotics control
Flyte5,850almost 2 years agoContainer-native, type-safe workflow and pipelines platform for large scale processing and ML
GalaxyPowerful workflow system which can be used on the command line or with the GUI
Geoweaver82almost 2 years agoIn-browser tool for data processing workflows with high-performance server support, featuring code history and workflow orchestration
KeplerKepler scientific workflow application from University of California
KNIME Analytics PlatformGeneral-purpose platform with many specialized domain extensions
KubeflowToolkit for making deployments of machine learning workflows on Kubernetes simple, portable and scalable
NextflowWorkbenchIntegrated development environment for Nextflow, Docker and Reusable Workflows
omega|ml DataOps Platform96almost 2 years agoData & model pipeline deployment for humans - integrated, scalable, extensible
OpenMOLEWorkflow Management System for exploration of models and parameter optimization
OphidiaData-analytics platform with declarative workflows of distributed operations
Orchest4,091over 3 years agoAn IDE for Data Science
PegasusWorkflow Management System
Piper491about 3 years agoDistributed workflow engine designed to be dead simple
Polyaxon3,581almost 2 years agoA platform for machine learning experimentation workflow
Reana127almost 2 years agoPlatform for reusable research data analyses developed by CERN
Sushi25almost 2 years agoSupporting User for SHell script Integration
YabiOnline research environment for grid, HPC and cloud computing
TavernaDomain independent workflow system
TemporalHighly scalable developer oriented engine
Windmill11,216almost 2 years agoDeveloper platform and workflow engine to turn scripts into internal tools
VisTrailsScientific workflow and provenance management system
WingsSemantic workflow system utilizing Pegasus as execution system
Watchdog13almost 2 years agoWorkflow management system for the automated and distributed analysis of large-scale experimental data
FlowHubFlowHub is a new workflow cloud platform

Awesome Pipeline / Workflow languages

Common Workflow Language1,456almost 2 years ago
Cloudgene Workflow Language
OpenMOLE DSL
Workflow Description Language780almost 2 years ago
Yet Another Workflow Language
Pipelines375almost 7 years ago

Awesome Pipeline / Workflow standardization initiatives

Workflow 4 Ever Initiative
Workflow 4 Ever workflow research object model
Workflow Patterns Initiative
Workflow Patterns Library
ResearchObject.org

Awesome Pipeline / ETL & Data orchestration

DataLadgit and git-annex based data version control system with lightweight provenance capture/re-execution support
DVCData version control system for ML project with lightweight pipeline support
lakeFS4,496almost 2 years agoRepeatable, atomic and versioned data lake on top of object storage
Nessie1,064almost 2 years agoProvides Git-like capability & version control for Iceberg Tables, Delta Lake Tables & SQL Views

Awesome Pipeline / Literate programming (aka interactive notebooks)

BeakerNotebook-style development environment
BinderTurn a GitHub repo into a collection of interactive notebooks powered by Jupyter and Kubernetes
IPythonA rich architecture for interactive computing
JupyterLanguage-agnostic notebook literate programming environment
Org ModeGNU Emacs major mode for computational notebooks, literate programming, and much more
PathomxInteractive data workflows built on Python
Polynote4,542almost 2 years agoA better notebook for Scala (and more). Built by Netflix
Ploomber3,530about 2 years agoConsolidate your notebooks and scripts in a reproducible pipeline using a file
R NotebooksR Markdown notebook literate programming environment
RedPoint NotebooksWeb-native computational notebook for programmers supporting multiple languages, APIs and webooks
SoSReadable, interactive, cross-platform and cross-language data science workflow system
ZeppelinWeb-based notebook that enables interactive data analytics

Awesome Pipeline / Extract, transform, load (ETL)

Cadence8,375almost 2 years agoDistributed, scalable, durable, and highly available orchestration engine developed by Uber
Dataform860almost 2 years agoDataform is a framework for managing SQL based operations in your data warehouse
HevoHevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services
Kiba ETLA data processing & ETL framework for Ruby
LinkedPipes ETLLinked Data publishing and consumption ETL tool
Pentaho KettleA plataform that delivers poweful ETL capabilities, using a groundbreaking, metadata-driven approach
Substation332almost 2 years agoSubstation is a cloud native data pipeline and transformation toolkit written in Go

Awesome Pipeline / Continuous Delivery workflows

Argo15,155almost 2 years agoGet stuff done with container-native workflows for Kubernetes
CDS4,620almost 2 years agoA pipeline based Continuous Delivery Service written in Golang

Awesome Pipeline / Build automation tools

BazelBuild software just as engineers do at Google
doit1,893about 2 years agoHighly generalized task-management and automation in Python
GradleUnified cross platforms builds
Just22,560almost 2 years agoCommand and recipe runner similar to Make, built in Rust
MakeThe GNU Make build system
Prodmodel58over 4 years agoBuild system for data science pipelines
SconsPython library focused on C/C++ builds
Shake772over 2 years agoDefine robust build systems akin to GNU Make using Haskell

Awesome Pipeline / Automated workflow composition

APE18almost 2 years agoA tool for the automated exploration of possible computational workflows based on semantic annotations

Awesome Pipeline / Other projects

HPC Grid Runner
NiFiPowerful and scalable directed graphs of data routing, transformation, and system mediation logic
noWorkflow122almost 2 years agoSupporting infrastructure to run scientific experiments without a scientific workflow management system, and still get things like provenance
ReprozipSimplifies the process of creating reproducible experiments from command-line executions
Awesome streaming2,720almost 2 years agoCurated list of awesome streaming frameworks, applications
Awesome ETL3,304about 2 years agoCurated list of notable ETL (extract, transform, load) frameworks, libraries and software
Awesome workflow engines6,519almost 2 years agoCurated list of awesome open source workflow engines
Computational Data Analysis Workflow Systems1,456almost 2 years ago

Backlinks from these awesome lists:

More related projects: