awesome-pipeline
A curated list of awesome pipeline toolkits inspired by Awesome Sysadmin
6k stars
231 watching
627 forks
last commit: about 1 month ago
Linked from 10 awesome lists
awesome-listworkflow
Awesome Pipeline / Pipeline frameworks & libraries | |||
ActionChain | A workflow system for simple linear success/failure workflows | ||
Adage | 55 | over 1 year ago | Small package to describe workflows that are not completely known at definition time |
AiiDA | 431 | 12 days ago | workflow manager with a strong focus on provenance, performance and extensibility |
Airflow | 36,519 | 11 days ago | Python-based workflow system created by AirBnb |
Anduril | Component-based workflow framework for scientific data analysis | ||
Antha | High-level language for biology | ||
Autosubmit | An open source Python experiment and workflow manager used to manage complex workflows on Cloud and HPC platforms | ||
AWE | 69 | almost 4 years ago | Workflow and resource management system with CWL support |
Balsam | 76 | 7 months ago | Python-based high throughput task and workflow engine |
Bds | Scripting language for data pipelines | ||
Beam | Unified programming model for batch and streaming data-parallel processing pipelines | ||
BioMake | 102 | 11 months ago | GNU-Make-like utility for managing builds and complex workflows |
BioQueue | 29 | over 1 year ago | Explicit framework with web monitoring and resource estimation |
Bioshake | 55 | about 5 years ago | Haskell DSL built on shake with strong typing and EDAM support |
Bistro | 47 | 5 months ago | Library to build and execute typed scientific workflows |
Bpipe | 226 | about 1 month ago | Tool for running and managing bioinformatics pipelines |
Briefly | 105 | about 6 years ago | Python Meta-programming Library for Job Flow Control |
Cluster Flow | Command-line tool which uses common cluster managers to run bioinformatics pipelines | ||
Clusterjob | 19 | 7 months ago | Automated reproducibility, and hassle-free submission of computational jobs to clusters |
Compi | Application framework for portable computational pipelines | ||
Compss | Programming model for distributed infrastructures | ||
Conan2 | 3 | about 10 years ago | Light-weight workflow management application |
Consecution | 168 | over 3 years ago | A Python pipeline abstraction inspired by Apache Storm topologies |
Cosmos | Python library for massively parallel workflows | ||
Couler | 908 | 4 months ago | Unified interface for constructing and managing workflows on different workflow engines, such as Argo Workflows, Tekton Pipelines, and Apache Airflow |
Covalent | 761 | about 1 month ago | Workflow orchestration toolkit for high-performance and quantum computing research and development |
Cromwell | 990 | 10 days ago | Workflow Management System geared towards scientific workflows from the Broad Institute |
Cuneiform | 232 | over 1 year ago | Advanced functional workflow language and framework, implemented in Erlang |
Cylc | A workflow engine for cycling systems, originally developed for operational environmental forecasting | ||
Dagobah | 755 | about 5 years ago | Simple DAG-based job scheduler in Python |
Dagr | 69 | over 2 years ago | A scala based DSL and framework for writing and executing bioinformatics pipelines as Directed Acyclic Graphs |
Dagster | 11,237 | 12 days ago | Python-based API for defining DAGs that interfaces with popular workflow managers for building data applications |
DataJoint | an open-source relational framework for scientific data pipelines | ||
Dask | 12,468 | 5 days ago | Dask is a flexible parallel computing library for analytics |
Dbt | Framework for writing analytics workflows entirely in SQL. The T part of ETL, focuses on analytics engineering | ||
Dockerflow | 97 | almost 7 years ago | Workflow runner that uses Dataflow to run a series of tasks in Docker |
Drake | 1,481 | over 2 years ago | Robust DSL akin to Make, implemented in Clojure |
Drake R package | 1,341 | 3 months ago | Reproducibility and high-performance computing with an easy R-focused interface. Unrelated to . Succeeded by |
Dray | 383 | over 4 years ago | An engine for managing the execution of container-based workflows |
ecFlow | 40 | 4 days ago | Workflow manager |
eHive | 52 | 10 days ago | System for creating and running pipelines on a distributed compute resource |
Fission Workflows | 371 | over 1 year ago | A fast, lightweight workflow engine for serverless/FaaS functions |
Flex | 56 | over 3 years ago | Language agnostic framework for building flexible data science pipelines (Python/Shell/Gnuplot) |
Flowr | 83 | over 3 years ago | Robust and efficient workflows using a simple language agnostic approach (R package) |
Gc3pie | 44 | almost 2 years ago | Python libraries and tools for running applications on diverse Grids and clusters |
Guix Workflow Language | A workflow management language extension for GNU Guix | ||
Gwf | 31 | 4 months ago | Make-like utility for submitting workflows via qsub |
Hamilton | 1,785 | 4 days ago | A python micro-framework for describing dataflows; runs anywhere python runs |
Hera | 571 | 9 days ago | Hera is an Argo Python SDK. Hera aims to make construction and submission of various Argo Project resources easy and accessible to everyone! Hera abstracts away low-level setup details while still maintaining a consistent vocabulary with Argo |
HyperLoom | 16 | about 2 years ago | Platform for defining and executing workflow pipelines in large-scale distributed environments |
HyperQueue | 276 | 1 day ago | HPC-focused task scheduler that automatically assigns tasks to Slurm/PBS allocations and submits them for the user |
Joblib | Set of tools to provide lightweight pipelining in Python | ||
Jug | A task Based parallelization framework for Python | ||
Kedro | 9,876 | 12 days ago | Workflow development tool that helps you build data pipelines |
Kestra | 9,914 | 1 day ago | Open source data orchestration and scheduling platform with declarative syntax |
Ketrew | 77 | over 6 years ago | Embedded DSL in the OCAML language alongside a client-server management application |
https://github.com/Nike-Inc/koheesio] | [ - Python framework for building efficient data pipelines | ||
Kronos | 19 | almost 8 years ago | Workflow assembler for cancer genome analytics and informatics |
Kubeflow Pipelines | Framework for building and deploying portable, scalable machine learning workflows using Docker containers and Argo Workflows | ||
Loom | 29 | almost 5 years ago | Tool for running bioinformatics workflows locally or in the cloud |
Longbow | Job proxying tool for biomolecular simulations | ||
Luigi | 17,746 | 18 days ago | Python module that helps you build complex pipelines of batch jobs |
Maestro | 133 | 25 days ago | YAML based HPC workflow execution tool |
Makeflow | Workflow engine for executing large complex workflows on clusters | ||
makepipe | 30 | almost 2 years ago | An R package which provides a set of simple tools for transforming an existing workflow into a self-documenting pipeline with very minimal upfront costs |
Mara | 2,072 | 10 months ago | A lightweight, opinionated ETL framework, halfway between plain scripts and Apache Airflow |
Mario | 139 | over 6 years ago | Scala library for defining data pipelines |
Martian | A language and framework for developing and executing complex computational pipelines | ||
MD Studio | 12 | almost 5 years ago | Microservice based workflow engine |
MetaFlow | Open-sourced framework from Netflix, for DAG generation for data scientists. Python and R API's | ||
Mistral | 284 | 4 days ago | Python based workflow engine by the Open Stack project |
Moa | 23 | almost 10 years ago | Lightweight workflows in bioinformatics |
Nextflow | Flow-based computational toolkit for reproducible and scalable bioinformatics pipelines | ||
nFlow | 199 | 2 days ago | Embeddable JVM-based workflow engine with high availability, fault tolerance, and support for multiple databases. Additional libraries are provided for visualization and REST API |
NiPype | 746 | 5 days ago | Workflows and interfaces for neuroimaging packages |
OpenGE | 26 | over 11 years ago | Accelerated framework for manipulating and interpreting high-throughput sequencing data |
Pachyderm | Distributed and reproducible data pipelining and data management, built on the container ecosystem | ||
Parsl | 498 | about 21 hours ago | Parallel Scripting Library |
PipeFunc | 166 | 2 days ago | Lightweight function pipeline (DAG) creation in pure Python for scientific workflows |
PipEngine | 20 | about 7 years ago | Ruby based launcher for complex biological pipelines |
Pinball | 1,048 | almost 5 years ago | Python based workflow engine by Pinterest |
Popper | 303 | over 2 years ago | YAML based container-native workflow engine supporting Docker, Singularity, Vagrant VMs with Docker daemon in VM, and local host |
Porcupine | 89 | over 2 years ago | Haskell workflow tool to express and compose tasks (optionally cached) whose datasources and sinks are known ahead of time and rebindable, and which can expose arbitrary sets of parameters to the outside world |
Prefect | Python based workflow engine powering Prefect | ||
Pydra | 120 | 5 days ago | Lightweight, DAG-based Python dataflow engine for reproducible and scalable scientific pipelines |
PyFlow | 146 | over 4 years ago | Lightweight parallel task engine |
pyperator | 59 | over 7 years ago | Simple push-based python workflow framework using asyncio, supporting recursive networks |
pyppl | 101 | 2 months ago | A python lightweight pipeline framework |
pypyr | Automation task-runner for sequential steps defined in a pipeline yaml, with AWS and Slack plug-ins | ||
pytask | 112 | 12 days ago | A workflow management system that facilitates reproducible data analyses |
Pwrake | 57 | over 4 years ago | Parallel workflow extension for Rake |
Qdo | Lightweight high-throughput queuing system for workflows with many small tasks to perform | ||
Qsubsec | 10 | over 1 year ago | Simple tokenised template system for SGE |
Rabix | 106 | over 5 years ago | Python-based workflow toolkit based on the Common Workflow Language and Docker |
Rain | 747 | over 1 year ago | Framework for large distributed task-based pipelines, written in Rust with Python API |
Ray | 33,240 | 13 days ago | Flexible, high-performance distributed Python execution framework |
Redun | 517 | about 1 month ago | Yet another redundant workflow engine |
Reflow | 965 | 12 months ago | Language and runtime for distributed, incremental data processing in the cloud |
Remake | 340 | over 6 years ago | Make-like declarative workflows in R |
Rmake | Wrapper for the creation of Makefiles, enabling massive parallelization | ||
Rubra | 38 | over 9 years ago | Pipeline system for bioinformatics workflows |
Ruffus | Computation Pipeline library for Python | ||
Ruigi | 42 | over 5 years ago | Pipeline tool for R, inspired by Luigi |
Sake | Self-documenting build automation tool | ||
SciLuigi | 334 | almost 2 years ago | Helper library for writing flexible scientific workflows in Luigi |
SciPipe | Library for writing Scientific Workflows in Go | ||
Signac | Lightweight, but scalable framework for file-driven workflows to be run locally and on HPC systems | ||
Scoop | 636 | over 1 year ago | Scalable Concurrent Operations in Python |
Seqtools | 48 | 6 months ago | Python library for lazy evaluation of pipelined transformations on indexable containers |
SmartPipeline | 23 | 8 months ago | A framework for rapid development of robust data pipelines following a simple design pattern |
Snakemake | Tool for running and managing bioinformatics pipelines | ||
Spiff | 1,678 | about 2 months ago | Based on the Workflow Patterns initiative and implemented in Python |
Stolos | 130 | about 6 years ago | Directed Acyclic Graph task dependency scheduler that simplify distributed pipelines |
Steppy | 134 | almost 6 years ago | lightweight, open-source, Python 3 library for fast and reproducible experimentation. (This repository has been archived by the owner on Jun 22, 2022.) |
Stpipe | File processing pipelines as a Python library | ||
StreamFlow | 50 | 11 days ago | Container native workflow management system focused on hybrid workflows |
StreamPipes | A self-service IoT toolbox to enable non-technical users to connect, analyze and explore IoT data streams | ||
Sundial | Jobsystem on AWS ECS or AWS Batch managing dependencies and scheduling | ||
Suro | 794 | over 1 year ago | Java-based distributed pipeline from Netflix |
Swift | Fast easy parallel scripting - on multicores, clusters, clouds and supercomputers | ||
TAF | 3 | 11 days ago | R package to organize reproducible scientific workflows |
Targets | 926 | about 23 hours ago | Dynamic, function-oriented -like reproducible pipelines at scale in R |
TaskGraph | 21 | 4 months ago | A library to help manage complicated computational software pipelines consisting of long running individual tasks |
Tibanna | 70 | 2 months ago | Tool that helps you run genomic pipelines on Amazon cloud |
Toil | 894 | 8 days ago | Distributed pipeline workflow manager (mostly for genomics) |
Yap | Extensible parallel framework, written in Python using OpenMPI libraries | ||
Yapp | 59 | almost 2 years ago | A C++ parallel pipeline library for stream processing |
Wallaroo | Framework for streaming data applications and algorithms that react to real-time events | ||
WorldMake | Easy Collaborative Reproducible Computing | ||
Zenaton | Workflow engine for orchestrating jobs, data and events across your applications and third party services | ||
ZenML | Extensible open-source MLOps framework to create reproducible pipelines for data scientists | ||
Awesome Pipeline / Workflow platforms | |||
ActivePapers | Computational science made reproducible and publishable | ||
Active Workflow | 830 | over 1 year ago | Polyglot workflows without leaving the comfort of your technology stack |
Anvi’o | A community and framework centered around metagenomics, designed to facilitate reproducible exploration and visualization of data | ||
Apache Iravata | Framework for executing and managing computational workflows on distributed computing resources | ||
Arteria | Event-driven automation for sequencing centers. Initiates workflows based on events | ||
Arvados | A container based workflow platform | ||
inactive since 10/2019 | Biokepler - Bioinformatics Scientific Workflow for Distributed Analysis of Large-Scale Biological Data. ( ) | ||
Butler | Framework for running scientific workflows on public and academic clouds | ||
Chipster | Open source platform for data analysis | ||
Clubber | Cluster Load Balancer for Bioinformatics e-Resources | ||
Digdag | Workflow manager designed for simplicity, extensibility and collaboration | ||
Domino | 136 | 3 months ago | User friendly and open source visual workflow management platform |
Fireworks | 353 | 3 months ago | Centralized workflow server for dynamic workflows of high-throughput computations |
Flojoy | 198 | about 1 month ago | Open source visual Python scripting for test, measurement, and robotics control |
Flyte | 5,619 | about 23 hours ago | Container-native, type-safe workflow and pipelines platform for large scale processing and ML |
Galaxy | Powerful workflow system which can be used on the command line or with the GUI | ||
Geoweaver | 78 | 21 days ago | In-browser tool for data processing workflows with high-performance server support, featuring code history and workflow orchestration |
Kepler | Kepler scientific workflow application from University of California | ||
KNIME Analytics Platform | General-purpose platform with many specialized domain extensions | ||
Kubeflow | Toolkit for making deployments of machine learning workflows on Kubernetes simple, portable and scalable | ||
NextflowWorkbench | Integrated development environment for Nextflow, Docker and Reusable Workflows | ||
omega|ml DataOps Platform | 95 | 1 day ago | Data & model pipeline deployment for humans - integrated, scalable, extensible |
OpenMOLE | Workflow Management System for exploration of models and parameter optimization | ||
Ophidia | Data-analytics platform with declarative workflows of distributed operations | ||
Orchest | 4,054 | over 1 year ago | An IDE for Data Science |
Pegasus | Workflow Management System | ||
Piper | 487 | about 1 year ago | Distributed workflow engine designed to be dead simple |
Polyaxon | 3,555 | 14 days ago | A platform for machine learning experimentation workflow |
Reana | 123 | 23 days ago | Platform for reusable research data analyses developed by CERN |
Sushi | 24 | 1 day ago | Supporting User for SHell script Integration |
Yabi | Online research environment for grid, HPC and cloud computing | ||
Taverna | Domain independent workflow system | ||
Temporal | Highly scalable developer oriented engine | ||
Windmill | 9,882 | about 23 hours ago | Developer platform and workflow engine to turn scripts into internal tools |
VisTrails | Scientific workflow and provenance management system | ||
Wings | Semantic workflow system utilizing Pegasus as execution system | ||
Watchdog | 13 | 9 days ago | Workflow management system for the automated and distributed analysis of large-scale experimental data |
FlowHub | FlowHub is a new workflow cloud platform | ||
Awesome Pipeline / Workflow languages | |||
Common Workflow Language | 1,450 | 2 months ago | |
Cloudgene Workflow Language | |||
OpenMOLE DSL | |||
Workflow Description Language | 766 | 16 days ago | |
Yet Another Workflow Language | |||
Pipelines | 373 | almost 5 years ago | |
Awesome Pipeline / Workflow standardization initiatives | |||
Workflow 4 Ever Initiative | |||
Workflow 4 Ever workflow research object model | |||
Workflow Patterns Initiative | |||
Workflow Patterns Library | |||
ResearchObject.org | |||
Awesome Pipeline / ETL & Data orchestration | |||
DataLad | git and git-annex based data version control system with lightweight provenance capture/re-execution support | ||
DVC | Data version control system for ML project with lightweight pipeline support | ||
lakeFS | 4,389 | 3 days ago | Repeatable, atomic and versioned data lake on top of object storage |
Nessie | 997 | 1 day ago | Provides Git-like capability & version control for Iceberg Tables, Delta Lake Tables & SQL Views |
Awesome Pipeline / Literate programming (aka interactive notebooks) | |||
Beaker | Notebook-style development environment | ||
Binder | Turn a GitHub repo into a collection of interactive notebooks powered by Jupyter and Kubernetes | ||
IPython | A rich architecture for interactive computing | ||
Jupyter | Language-agnostic notebook literate programming environment | ||
Org Mode | GNU Emacs major mode for computational notebooks, literate programming, and much more | ||
Pathomx | Interactive data workflows built on Python | ||
Polynote | 4,517 | 2 months ago | A better notebook for Scala (and more). Built by Netflix |
Ploomber | 3,490 | 24 days ago | Consolidate your notebooks and scripts in a reproducible pipeline using a file |
R Notebooks | R Markdown notebook literate programming environment | ||
RedPoint Notebooks | Web-native computational notebook for programmers supporting multiple languages, APIs and webooks | ||
SoS | Readable, interactive, cross-platform and cross-language data science workflow system | ||
Zeppelin | Web-based notebook that enables interactive data analytics | ||
Awesome Pipeline / Extract, transform, load (ETL) | |||
Cadence | 8,225 | 11 days ago | Distributed, scalable, durable, and highly available orchestration engine developed by Uber |
Dataform | 837 | 5 days ago | Dataform is a framework for managing SQL based operations in your data warehouse |
Hevo | Hevo is a Fully Automated, No-code Data Pipeline Platform that supports 150+ ready-to-use integrations across Databases, SaaS Applications, Cloud Storage, SDKs, and Streaming Services | ||
Kiba ETL | A data processing & ETL framework for Ruby | ||
LinkedPipes ETL | Linked Data publishing and consumption ETL tool | ||
Pentaho Kettle | A plataform that delivers poweful ETL capabilities, using a groundbreaking, metadata-driven approach | ||
Substation | 321 | 15 days ago | Substation is a cloud native data pipeline and transformation toolkit written in Go |
Awesome Pipeline / Continuous Delivery workflows | |||
Argo | 14,933 | 5 days ago | Get stuff done with container-native workflows for Kubernetes |
CDS | 4,563 | 10 days ago | A pipeline based Continuous Delivery Service written in Golang |
Awesome Pipeline / Build automation tools | |||
Bazel | Build software just as engineers do at Google | ||
doit | 1,856 | 3 months ago | Highly generalized task-management and automation in Python |
Gradle | Unified cross platforms builds | ||
Just | 20,539 | 11 days ago | Command and recipe runner similar to Make, built in Rust |
Make | The GNU Make build system | ||
Prodmodel | 59 | over 2 years ago | Build system for data science pipelines |
Scons | Python library focused on C/C++ builds | ||
Shake | 772 | 5 months ago | Define robust build systems akin to GNU Make using Haskell |
Awesome Pipeline / Automated workflow composition | |||
APE | 16 | 10 days ago | A tool for the automated exploration of possible computational workflows based on semantic annotations |
Awesome Pipeline / Other projects | |||
HPC Grid Runner | |||
NiFi | Powerful and scalable directed graphs of data routing, transformation, and system mediation logic | ||
noWorkflow | 118 | 17 days ago | Supporting infrastructure to run scientific experiments without a scientific workflow management system, and still get things like provenance |
Reprozip | Simplifies the process of creating reproducible experiments from command-line executions | ||
Awesome Pipeline / Related lists | |||
Awesome streaming | 2,680 | about 2 months ago | Curated list of awesome streaming frameworks, applications |
Awesome ETL | 3,270 | 3 months ago | Curated list of notable ETL (extract, transform, load) frameworks, libraries and software |
Awesome workflow engines | 6,276 | 4 days ago | Curated list of awesome open source workflow engines |
Computational Data Analysis Workflow Systems | 1,450 | 2 months ago |