pydra
Dataflow engine
A lightweight Python dataflow engine for building and executing directed acyclic graphs (DAGs) in a scalable manner.
Pydra Dataflow Engine
123 stars
14 watching
59 forks
Language: Python
last commit: almost 2 years agoLinked from 1 awesome list
brainwebdataflow-enginepython3
pditommaso/awesome-pipelineRelated projects:
| Repository | Description | Stars |
|---|---|---|
 baffelli/pyperator | A Python library for building asynchronous workflows with a directed acyclic graph structure | 60 |
 nipy/nipype | A framework that simplifies workflow design and interaction between neuroimaging packages | 750 |
 danielgerlag/liteflow | A Python library for running workflows with pluggable persistence and concurrency providers. | 62 |
 wayfair-incubator/dagger | A distributed workflow engine for executing long-running business logic in a scalable and resilient way. | 55 |
 dagworks-inc/hamilton | Helps define and manage data transformations with a modular, self-documenting, and portable framework for directed acyclic graphs (DAGs) of data transformations. | 1,900 |
 man-group/mdf | A toolkit for expressing programs as directed acyclic graphs and wiring together computations over time-series data. | 169 |
 analysiscenter/batchflow | A framework for defining and executing data processing and machine learning workflows with support for batch processing, lazy execution, and model training. | 202 |
 pydap/pydap | A Python library for accessing and manipulating scientific data over the internet using the OPeNDAP protocol. | 139 |
 streamlet-dev/tributary | A Python library for constructing dataflow graphs with support for reactive and lazy evaluation. | 444 |
 johnsonc/lambdo | A workflow engine for unifying feature engineering and machine learning operations in data analysis pipelines | 1 |
 mehd-io/pypi-duck-flow | A data engineering project that extracts insights from Python projects using DuckDB and MotherDuck. | 173 |
 apache/datafusion-python | A Python library that provides a data processing and querying framework using the Apache Arrow in-memory query engine. | 385 |
 d6t/d6tflow | A Python library to build and manage complex data science workflows efficiently | 953 |
 deqitang/pymatflow | Automates preparation and submission of DFT calculations for materials science simulations | 6 |
 elyra-ai/elyra | An AI-centric extension to JupyterLab Notebooks | 1,861 |