airflow
Workflow orchestrator
A platform to programmatically author, schedule and monitor complex workflows
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
38k stars
759 watching
14k forks
Language: Python
last commit: almost 2 years agoLinked from 13 awesome lists
airflowapacheapache-airflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationpythonschedulerworkflowworkflow-engineworkflow-orchestration
Backlinks from these awesome lists:
ethicalml/awesome-production-machine-learning
oxnr/awesome-bigdata
humiaozuzu/awesome-flask
igorbarinov/awesome-data-engineering
meirwah/awesome-workflow-engines
pditommaso/awesome-pipeline
pawl/awesome-etl
mjhea0/awesome-flask
youngwookim/awesome-hadoop
stn1slv/awesome-integration
vihar/awesome-oss-saas
kelvins/awesome-dataops
monksy/awesome-data-engineering
Related projects:
| Repository | Description | Stars |
|---|---|---|
 couler-proj/couler | Provides a unified interface for constructing and managing workflows across different workflow engines. | 919 |
 knowsuchagency/orkestra | A Python-based workflow orchestration system built on top of AWS tools and services. | 50 |
 llnl/maestrowf | A tool to orchestrate computational workflows in high-performance computing environments. | 139 |
 caicloud/cyclone | A workflow engine that enables users to orchestrate complex business processes and pipelines in a scalable and extensible manner | 1,073 |
| apache/oozie | A system to define, manage, schedule, and execute complex data processing workflows across multiple systems using a declarative framework. | 717 |
 danielgerlag/conductor | A distributed workflow management system that coordinates services and scripts into complex workflows. | 538 |
 alpha-unito/streamflow | A container-native workflow management system for managing multi-container environments and hybrid workflows. | 54 |
 americanexpress/unify-flowret | An orchestration engine for building and executing workflows in Java | 108 |
 fieldryand/goflow | A lightweight, single-binary DAG scheduler and dashboard for orchestrating workflows with tasks and operators | 396 |
 analysiscenter/batchflow | A framework for defining and executing data processing and machine learning workflows with support for batch processing, lazy execution, and model training. | 202 |
 automaticmode/active_workflow | An orchestration platform that allows developers to create and manage workflows using agents written in any programming language, without vendor lock-in. | 845 |
 moby/datakit | A tool to orchestrate applications using a version-controlled dataflow | 1,083 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 230 |
 nvidia/earth2studio | A Python-based framework for building and deploying AI-driven weather and climate workflows | 116 |
 wayfair-incubator/dagger | A distributed workflow engine for executing long-running business logic in a scalable and resilient way. | 55 |