prefect
Workflow automator
A framework for building and managing workflows that automate data processing tasks in Python
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
18k stars
165 watching
2k forks
Language: Python
last commit: almost 2 years agoLinked from 9 awesome lists
automationdatadata-engineeringdata-opsdata-scienceinfrastructureml-opsobservabilityorchestrationpipelineprefectpythonworkflowworkflow-engine
vinta/awesome-python
jobbole/awesome-python-cn
ethicalml/awesome-production-machine-learning
runacapital/awesome-oss-alternatives
meirwah/awesome-workflow-engines
pawl/awesome-etl
gunnarmorling/awesome-opensource-data-engineering
vihar/awesome-oss-saas
simomay/find-ossRelated projects:
| Repository | Description | Stars |
|---|---|---|
 skydio/revup | Tools for managing pull requests without changing branches | 322 |
 shobrook/rebound | Instantly fetches Stack Overflow results when an exception is thrown in a running program | 4,109 |
 facebook/relay | A framework that simplifies data-driven React applications by abstracting away imperative API interactions and providing GraphQL-based query management. | 18,434 |
 microsoft/taskweaver | An agent framework for planning and executing data analytics tasks through code snippets. | 5,398 |
 bogdanp/dramatiq | A fast and reliable way to process tasks in the background. | 4,388 |
 agnostiqhq/covalent | A Python library for orchestrating machine-learning and high-performance computing workflows in heterogeneous environments | 785 |
 amitt001/delegator.py | A library for managing subprocesses in Python. | 1,702 |
 pathwaycom/pathway | A Python-based ETL framework for stream processing and real-time analytics with a scalable Rust engine. | 7,174 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 230 |
 coleifer/huey | A lightweight task queue written in Python with support for multiple execution models and scheduling. | 5,259 |
 natcap/taskgraph | A Python library for managing and optimizing computational workflows with parallel processing and data reuse. | 22 |
 stackstorm/st2 | An event-driven automation platform that ties together services and tools to automate common operational patterns | 6,116 |
 couler-proj/couler | Provides a unified interface for constructing and managing workflows across different workflow engines. | 919 |
 flyteorg/flyte | An orchestrator platform that enables the building and deployment of production-grade data pipelines. | 5,850 |
 taskflow/taskflow | A tool for parallel task programming with automatic work-stealing and heterogeneous CPU-GPU task management | 10,338 |