dud
Data pipeline manager
A lightweight tool for managing and versioning large data alongside source code in data pipelines
A lightweight CLI tool for versioning data alongside source code and building data pipelines.
183 stars
8 watching
8 forks
Language: Go
last commit: 12 days ago
Linked from 1 awesome list
data-engineeringdata-pipelinesdata-sciencedatasetdvcsmachine-learningmlops
kelvins/awesome-mlopsRelated projects:
| Repository | Description | Stars |
|---|---|---|
 prodmodel/prodmodel | A tool for managing data science pipelines by automating build, testing, and deployment processes while ensuring correctness and performance. | 59 |
 linkedin/brooklin | A distributed system for streaming data between heterogeneous systems with high reliability and throughput at scale | 920 |
 johnsonc/lambdo | A workflow engine for unifying feature engineering and machine learning operations in data analysis pipelines | 1 |
 hyfather/pipeline | A package implementing pipelines using goroutines to manage concurrency in Go applications. | 57 |
 fluidattacks/makes | A framework for building and managing CI/CD pipelines and application environments with cryptographic signed dependencies. | 453 |
 galaxyproject/galaxy | An integrated framework for data-intensive scientific analysis and workflow management | 1,410 |
 ypares/porcupine | A tool that enables data manipulation and analysis pipelines to be flexible, reusable, and reproducible in different environments | 89 |
 samapriya/planet-gee-pipeline-cli | A command-line tool for automating data processing and uploads from Planet's API to Google Earth Engine. | 42 |
 pdpipe/pdpipe | A tool for creating and managing data pipelines with pandas DataFrames | 716 |
 apache/streampipes | A toolbox for industrial data analytics and stream processing | 607 |
 druths/xp | A tool for creating flexible and self-documenting data science pipelines | 56 |
 montilab/pipeliner | A framework for defining and automating bioinformatics pipelines using Nextflow. | 44 |
 calebwin/pipelines | A language and runtime for crafting massively parallel data pipelines | 374 |
 ssadedin/bpipe | A tool for running and managing bioinformatics pipelines by abstracting away low-level details and providing features such as dependency tracking, transactional management, and parallelism. | 230 |
 giacbrd/smartpipeline | A framework for designing and executing concurrent data pipelines with a focus on simplicity and efficiency | 23 |