datakit
Pipeline orchestrator
A tool to orchestrate applications using a version-controlled dataflow
Connect processes into powerful data pipelines with a simple git-like filesystem interface
1k stars
44 watching
152 forks
Language: OCaml
last commit: about 1 year ago data-flowdatabasedatakitdockerfilesystem-apipipeline
Related projects:
| Repository | Description | Stars |
|---|---|---|
 dagster-io/dagster | An orchestration platform for data pipelines and assets, providing a declarative programming model and integrated lineage and observability. | 11,699 |
 it4innovations/hyperloom | A platform for defining and executing scientific pipelines in distributed environments using C++ and Python. | 16 |
 databand-ai/dbnd | A framework for building and tracking data pipelines to simplify data engineering workflows | 251 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |
 orchestratora/orchestrator | A toolset for building and testing UI applications using Angular | 16 |
 apache/airflow | A platform to programmatically author, schedule and monitor complex workflows | 37,120 |
 streamsets/datacollector-oss | A continuous big data ingestion platform that enables easy creation of data pipelines for various data sources and destinations. | 90 |
 llnl/maestrowf | A tool to orchestrate computational workflows in high-performance computing environments. | 134 |
 huawei/containerops | An orchestration platform for automating DevOps workflows by combining tools and services into a single, GUI-based solution | 338 |
 pdpipe/pdpipe | A tool for creating and managing data pipelines with pandas DataFrames | 716 |
 kevin-hanselman/dud | A lightweight tool for managing and versioning large data alongside source code in data pipelines | 183 |
 knowsuchagency/orkestra | A Python-based workflow orchestration system built on top of AWS tools and services. | 50 |
 hyfather/pipeline | A package implementing pipelines using goroutines to manage concurrency in Go applications. | 58 |
 johnsonc/lambdo | A workflow engine for unifying feature engineering and machine learning operations in data analysis pipelines | 1 |
| moby/moby | Enables and accelerates software containerization by providing a modular framework for assembling custom systems | 68,758 |