prefect
Workflow automator
A framework for building and managing workflows that automate data processing tasks in Python
Prefect is a workflow orchestration framework for building resilient data pipelines in Python.
18k stars
165 watching
2k forks
Language: Python
last commit: about 1 month ago
Linked from 9 awesome lists
automationdatadata-engineeringdata-opsdata-scienceinfrastructureml-opsobservabilityorchestrationpipelineprefectpythonworkflowworkflow-engine
Related projects:
Repository | Description | Stars |
---|---|---|
skydio/revup | Tools for managing pull requests without changing branches | 322 |
shobrook/rebound | Instantly fetches Stack Overflow results when an exception is thrown in a running program | 4,109 |
facebook/relay | A framework that simplifies data-driven React applications by abstracting away imperative API interactions and providing GraphQL-based query management. | 18,434 |
microsoft/taskweaver | An agent framework for planning and executing data analytics tasks through code snippets. | 5,398 |
bogdanp/dramatiq | A fast and reliable way to process tasks in the background. | 4,388 |
agnostiqhq/covalent | A Python library for orchestrating machine-learning and high-performance computing workflows in heterogeneous environments | 785 |
amitt001/delegator.py | A library for managing subprocesses in Python. | 1,702 |
pathwaycom/pathway | A Python-based ETL framework for stream processing and real-time analytics with a scalable Rust engine. | 7,174 |
pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 230 |
coleifer/huey | A lightweight task queue written in Python with support for multiple execution models and scheduling. | 5,259 |
natcap/taskgraph | A Python library for managing and optimizing computational workflows with parallel processing and data reuse. | 22 |
stackstorm/st2 | An event-driven automation platform that ties together services and tools to automate common operational patterns | 6,116 |
couler-proj/couler | Provides a unified interface for constructing and managing workflows across different workflow engines. | 919 |
flyteorg/flyte | An orchestrator platform that enables the building and deployment of production-grade data pipelines. | 5,850 |
taskflow/taskflow | A tool for parallel task programming with automatic work-stealing and heterogeneous CPU-GPU task management | 10,338 |