prefect
Workflow automator
A framework for building and managing workflows that automate data processing tasks in Python
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
17k stars
165 watching
2k forks
Language: Python
last commit: 6 days ago
Linked from 9 awesome lists
automationdatadata-engineeringdata-opsdata-scienceinfrastructureml-opsobservabilityorchestrationpipelineprefectpythonworkflowworkflow-engine
vinta/awesome-python
jobbole/awesome-python-cn
ethicalml/awesome-production-machine-learning
runacapital/awesome-oss-alternatives
meirwah/awesome-workflow-engines
pawl/awesome-etl
gunnarmorling/awesome-opensource-data-engineering
vihar/awesome-oss-saas
simomay/find-ossRelated projects:
| Repository | Description | Stars |
|---|---|---|
 skydio/revup | Tools to streamline code review and pull request management in Git-based development workflows. | 318 |
 shobrook/rebound | Instantly fetches Stack Overflow results when an exception is thrown from a Python file. | 4,106 |
 facebook/relay | A framework that simplifies data-driven React applications by abstracting away imperative API interactions and providing GraphQL-based query management. | 18,414 |
 microsoft/taskweaver | An agent framework for planning and executing data analytics tasks through code snippets. | 5,348 |
 bogdanp/dramatiq | A fast and reliable way to process tasks in the background. | 4,346 |
 agnostiqhq/covalent | A Python library for orchestrating machine-learning and high-performance computing workflows in heterogeneous environments | 775 |
 amitt001/delegator.py | A library for managing subprocesses in Python. | 1,703 |
 pathwaycom/pathway | An ETL framework that enables real-time data processing and analytics using Python, with support for streaming data, batch processing, machine learning, and integration with various external data sources. | 4,324 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |
 coleifer/huey | A lightweight task queue written in Python with support for multiple execution models and scheduling. | 5,220 |
 natcap/taskgraph | A Python library for managing and optimizing computational workflows with parallel processing and data reuse. | 21 |
 stackstorm/st2 | An event-driven automation platform that ties together services and tools to automate common operational patterns | 6,086 |
 couler-proj/couler | Provides a unified interface for constructing and managing workflows across different workflow engines. | 915 |
 flyteorg/flyte | An orchestrator platform that enables the building and deployment of production-grade data pipelines. | 5,785 |
 taskflow/taskflow | A tool for parallel task programming with automatic work-stealing and heterogeneous CPU-GPU task management | 10,235 |