luigi
Batch pipeline builder
Helps build complex pipelines of batch jobs with dependency resolution and workflow management
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
18k stars
472 watching
2k forks
Language: Python
last commit: 11 days ago
Linked from 13 awesome lists
hadoopluigiorchestration-frameworkpythonscheduling
Backlinks from these awesome lists:
-
vinta/awesome-python
-
jobbole/awesome-python-cn
-
ethicalml/awesome-production-machine-learning
-
oxnr/awesome-bigdata
-
igorbarinov/awesome-data-engineering
-
meirwah/awesome-workflow-engines
-
pditommaso/awesome-pipeline
-
kelvins/awesome-mlops
-
pawl/awesome-etl
-
ly0n/awesome-robotic-tooling
-
cicdops/awesome-ciandcd
-
-
monksy/awesome-data-engineering
Related projects:
| Repository | Description | Stars |
|---|---|---|
 pharmbio/sciluigi | A lightweight wrapper around Spotify's Luigi workflow library to simplify writing scientific workflows | 334 |
 languagemachines/luiginlp | A workflow management system for Natural Language Processing tasks | 21 |
 python-mario/mario | Tools for executing Python code and building data pipelines in a Unix shell | 507 |
 quintoandar/butterfree | A Python library for building data pipelines to create and load features into a feature store using Apache Spark. | 283 |
 quickube/piper | Automates creation of Kubernetes workflows based on Git branch changes | 22 |
 minyus/pipelinex | A Python package to build and experiment with machine learning pipelines using Kedro, MLflow, and other tools | 224 |
 kubeflow-kale/kale | Simplifies the deployment of Kubeflow Pipelines workflows by providing a graphical interface for Data Scientists to define and deploy pipelines directly from JupyterLab. | 632 |
 fluidattacks/makes | A framework for building and managing CI/CD pipelines and application environments with cryptographic signed dependencies. | 453 |
 pwwang/pipen | A Python-based workflow automation framework that enables easy creation of data processing pipelines | 103 |
 kinto-b/makepipe | A tool for constructing simple pipelines in R with minimal overheads. | 30 |
 iommirocks/iommi | A toolkit to build web applications faster with Django. | 797 |
 hhio618/golem-ci | A decentralized task pipeline on Golem.network using Python. | 5 |
 history-frontend/bee-cli | A toolset for building small programs with WePy and UI components. | 3 |
 listyque/tactic-handler | A PySide-based client tool for managing pipelines, assets, and workflows in 3D animation software | 93 |
 kirillseva/ruigi | A tool for designing and managing data processing pipelines in R. | 42 |