airflow
Workflow orchestrator
A platform to programmatically author, schedule and monitor complex workflows
Apache Airflow - A platform to programmatically author, schedule, and monitor workflows
37k stars
760 watching
14k forks
Language: Python
last commit: 6 days ago
Linked from 13 awesome lists
airflowapacheapache-airflowautomationdagdata-engineeringdata-integrationdata-orchestratordata-pipelinesdata-scienceeltetlmachine-learningmlopsorchestrationpythonschedulerworkflowworkflow-engineworkflow-orchestration
Backlinks from these awesome lists:
-
ethicalml/awesome-production-machine-learning
-
oxnr/awesome-bigdata
-
humiaozuzu/awesome-flask
-
igorbarinov/awesome-data-engineering
-
meirwah/awesome-workflow-engines
-
pditommaso/awesome-pipeline
-
pawl/awesome-etl
-
mjhea0/awesome-flask
-
youngwookim/awesome-hadoop
-
stn1slv/awesome-integration
-
vihar/awesome-oss-saas
-
kelvins/awesome-dataops
-
monksy/awesome-data-engineering
Related projects:
| Repository | Description | Stars |
|---|---|---|
 couler-proj/couler | Provides a unified interface for constructing and managing workflows across different workflow engines. | 915 |
 knowsuchagency/orkestra | A Python-based workflow orchestration system built on top of AWS tools and services. | 50 |
 llnl/maestrowf | A tool to orchestrate computational workflows in high-performance computing environments. | 134 |
 caicloud/cyclone | A workflow engine that enables users to orchestrate complex business processes and pipelines in a scalable and extensible manner | 1,072 |
| apache/oozie | A system to define, manage, schedule, and execute complex data processing workflows across multiple systems using a declarative framework. | 713 |
 danielgerlag/conductor | A distributed workflow management system that coordinates services and scripts into complex workflows. | 532 |
 alpha-unito/streamflow | A container-native workflow management system for managing multi-container environments and hybrid workflows. | 52 |
 americanexpress/unify-flowret | An orchestration engine for building and executing workflows in Java | 106 |
 fieldryand/goflow | A lightweight, single-binary DAG scheduler and dashboard for orchestrating workflows with tasks and operators | 386 |
 analysiscenter/batchflow | A framework for defining and executing data processing and machine learning workflows with support for batch processing, lazy execution, and model training. | 201 |
 automaticmode/active_workflow | An orchestration platform that allows developers to create and manage workflows using agents written in any programming language, without vendor lock-in. | 836 |
 moby/datakit | A tool to orchestrate applications using a version-controlled dataflow | 1,082 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |
 nvidia/earth2studio | A Python-based framework for building and deploying AI-driven weather and climate workflows | 98 |
 wayfair-incubator/dagger | A distributed workflow engine for executing long-running business logic in a scalable and resilient way. | 55 |