datakit
Pipeline orchestrator
A tool to orchestrate applications using a version-controlled dataflow
Connect processes into powerful data pipelines with a simple git-like filesystem interface
1k stars
44 watching
153 forks
Language: OCaml
last commit: over 1 year ago data-flowdatabasedatakitdockerfilesystem-apipipeline
Related projects:
| Repository | Description | Stars |
|---|---|---|
 dagster-io/dagster | An orchestration platform for data pipelines and assets, providing a declarative programming model and integrated lineage and observability. | 12,055 |
 it4innovations/hyperloom | A platform for defining and executing scientific pipelines in distributed environments using C++ and Python. | 16 |
 databand-ai/dbnd | An agile pipeline framework for data engineering teams to track and orchestrate their data processes. | 260 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 230 |
 orchestratora/orchestrator | A toolset for building and testing UI applications using Angular | 16 |
 apache/airflow | A platform to programmatically author, schedule and monitor complex workflows | 37,580 |
 streamsets/datacollector-oss | A continuous big data ingestion platform that enables easy creation of data pipelines for various data sources and destinations. | 90 |
 llnl/maestrowf | A tool to orchestrate computational workflows in high-performance computing environments. | 139 |
 huawei/containerops | An orchestration platform for automating DevOps workflows by combining tools and services into a single, GUI-based solution | 339 |
 pdpipe/pdpipe | Provides a set of pre-defined data processing pipelines for pandas DataFrames. | 718 |
 kevin-hanselman/dud | A lightweight tool for managing and versioning large data alongside source code in data pipelines | 184 |
 knowsuchagency/orkestra | A Python-based workflow orchestration system built on top of AWS tools and services. | 50 |
 hyfather/pipeline | A package implementing pipelines using goroutines to manage concurrency in Go applications. | 56 |
 johnsonc/lambdo | A workflow engine for unifying feature engineering and machine learning operations in data analysis pipelines | 1 |
| moby/moby | Enables and accelerates software containerization by providing a modular framework for assembling custom systems | 68,896 |