prodmodel

Pipeline manager

A tool for managing data science pipelines by automating build, testing, and deployment processes while ensuring correctness and performance.

Build, test, deploy, iterate - Dev and prod tool for data science pipelines

GitHub

58 stars
3 watching
3 forks
Language: Python
last commit: over 2 years ago
Linked from 4 awesome lists

build-automationbuild-systembuild-tooldatadata-sciencedataengdataengineeringdatascienceproductionproductivity

Backlinks from these awesome lists:

Related projects:

Repository Description Stars
kevin-hanselman/dud A lightweight tool for managing and versioning large data alongside source code in data pipelines 184
ssadedin/bpipe A tool for running and managing bioinformatics pipelines by abstracting away low-level details and providing features such as dependency tracking, transactional management, and parallelism. 233
druths/xp A tool for creating flexible and self-documenting data science pipelines 56
bjpop/rubra A bioinformatics pipeline system that supports running workflow stages on a distributed compute cluster. 38
lightforever/mlcomp A distributed framework for building and managing complex machine learning pipelines with a user-friendly interface. 188
galaxyproject/galaxy A platform for data-intensive scientific analysis and workflow management 1,431
natcap/taskgraph A Python library for managing and optimizing computational workflows with parallel processing and data reuse. 22
samapriya/planet-gee-pipeline-cli A command-line tool for automating data processing and uploads from Planet's API to Google Earth Engine. 42
linkedin/brooklin A distributed system for streaming data between heterogeneous systems with high reliability and throughput at scale 931
fluidattacks/makes A framework for building and managing CI/CD pipelines and application environments with cryptographic signed dependencies. 461
hyfather/pipeline A package implementing pipelines using goroutines to manage concurrency in Go applications. 56
alexanderrichtertd/plex A comprehensive pipeline management system for VFX, animation, and game production workflows 249
ypares/porcupine A tool that enables data manipulation and analysis pipelines to be flexible, reusable, and reproducible in different environments 89
montilab/pipeliner A framework for defining and automating bioinformatics pipelines using Nextflow. 44
databiosphere/toil A workflow management system designed to efficiently run pipelines in various environments. 901