batchflow
Workflow engine
A framework for defining and executing data processing and machine learning workflows with support for batch processing, lazy execution, and model training.
BatchFlow helps you conveniently work with random or sequential batches of your data and define data processing and machine learning workflows even for datasets that do not fit into memory.
202 stars
17 watching
46 forks
Language: Python
last commit: 16 days ago
Linked from 2 awesome lists
data-sciencemachine-learningpipelinepipeline-frameworkpythonpython3workflowworkflow-engine
ethicalml/awesome-production-machine-learning
krzjoa/awesome-python-data-scienceRelated projects:
| Repository | Description | Stars |
|---|---|---|
 asavinov/lambdo | A workflow engine that unifies feature engineering and machine learning operations for data analysis. | 23 |
 insitro/redun | A workflow engine that allows complex data flows to be defined and executed in a flexible and efficient manner | 537 |
 wayfair-incubator/dagger | A distributed workflow engine for executing long-running business logic in a scalable and resilient way. | 55 |
 baffelli/pyperator | A Python library for building asynchronous workflows with a directed acyclic graph structure | 60 |
 aelassas/wexflow | Automates recurring tasks using a cross-platform workflow engine and automation platform. | 600 |
 danielgerlag/liteflow | A Python library for running workflows with pluggable persistence and concurrency providers. | 62 |
 americanexpress/unify-flowret | An orchestration engine for building and executing workflows in Java | 108 |
 runabol/tork | A distributed workflow engine for automating complex tasks and workflows. | 616 |
 astrazeneca/runnable | A Python framework for defining and executing data science workflows with modular components | 40 |
 optimajet/workflowengine.net | A C# framework for building workflow engines that integrate with various databases and offer features like process design, execution, and version control. | 911 |
 nitorcreations/nflow | A Java-based workflow automation engine designed to handle business processes with high availability and fault tolerance. | 206 |
 aiidateam/aiida-core | A workflow manager for computational science with a focus on provenance and performance | 440 |
 floraison/flor | A Ruby-based workflow engine that executes process definitions and reuses taskers across multiple executions. | 265 |
 carlio/django-flows | A library that enables the creation of complex, stateful user workflows with optional branches and nested logic | 140 |
| danielgerlag/workflow-es | A workflow engine that supports long-running processes with pluggable persistence and concurrency providers. | 202 |