luigi
by spotify
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
AI summary
Batch workflow manager
A tool for building and managing complex workflows of batch jobs
- stars
- 18K
- forks
- 2.4K
- watching
- 471
- awesome lists
- 13
Similar projects
Found by comparing what the projects do, not just their names.
Workflow wrapper
A lightweight wrapper around Spotify's Luigi workflow library to simplify writing scientific workflows
NLP workflow manager
A workflow management system for Natural Language Processing tasks
Pipeline tool
Tools for executing Python code and building data pipelines in a Unix shell
Feature pipeline builder
A Python library for building data pipelines to create and load features into a feature store using Apache Spark.
Pipeline manager
Automates creation of Kubernetes workflows based on Git branch changes
ML pipeline builder
A Python package to build and experiment with machine learning pipelines using Kedro, MLflow, and other tools
Pipeline builder
Simplifies the deployment of Kubeflow Pipelines workflows by providing a graphical interface for Data Scientists to define and deploy pipelines directly from JupyterLab.
Pipeline manager
A framework for building and managing CI/CD pipelines and application environments with cryptographic signed dependencies.
pwwang/pipen105
Pipeline tool
A Python-based workflow automation framework that enables easy creation of data processing pipelines
Pipeline builder
A tool for constructing simple pipelines in R with minimal overheads.
Web app builder
A toolkit to build web applications faster with Django.
Task pipeline
A decentralized task pipeline on Golem.network using Python.
Program builder
A toolset for building small programs with WePy and UI components.
Pipeline manager
A PySide-based client tool for managing pipelines, assets, and workflows in 3D animation software
Pipeline builder
A tool for designing and managing data processing pipelines in R.
Featured in 13 awesome lists
Each link jumps to the spot where the list mentions luigi.
vinta/awesome-python
jobbole/awesome-python-cn
ethicalml/awesome-production-machine-learning
oxnr/awesome-bigdata
igorbarinov/awesome-data-engineering
meirwah/awesome-workflow-engines
pditommaso/awesome-pipeline
kelvins/awesome-mlops
ly0n/awesome-robotic-tooling
pawl/awesome-etl
cicdops/awesome-ciandcd
monksy/awesome-data-engineering