rain
Pipeline processor
A framework for processing large-scale task-based pipelines in a distributed manner
Framework for large distributed pipelines
748 stars
44 watching
54 forks
Language: Rust
last commit: over 1 year ago
Linked from 1 awesome list
distributed-computingpipelinespython-apirustworkflows
pditommaso/awesome-pipelineRelated projects:
| Repository | Description | Stars |
|---|---|---|
 raine/ramda-cli | A tool for composing functions into data-processing pipelines to produce desired output. | 573 |
 thephpleague/pipeline | Provides a flexible pipeline pattern implementation to compose sequential stages and process payloads in a composable manner. | 960 |
 wallaroolabs/wally | A distributed stream processing framework for real-time data reactions | 1,480 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 nazar256/parapipe | A non-blocking buffered pipeline library that allows concurrent processing of data while maintaining output order without locks or mutexes. | 31 |
 pystorm/streamparse | Enables real-time computation on live streams of data using Python | 1,495 |
 bytewax/bytewax | A Python framework for stateful stream and event processing with built-in connectors and flexible dataflow capabilities. | 1,558 |
 amitt001/delegator.py | A library for managing subprocesses in Python. | 1,701 |
 pwwang/pipen | A Python-based workflow automation framework that enables easy creation of data processing pipelines | 103 |
 olirice/flupy | A library that provides a fluent interface for processing data pipelines in Python without holding large amounts of memory | 193 |
 hhio618/golem-ci | A decentralized task pipeline on Golem.network using Python. | 5 |
 druths/xp | A tool for creating flexible and self-documenting data science pipelines | 56 |
 natcap/taskgraph | A Python library for managing and optimizing computational workflows with parallel processing and data reuse. | 21 |
 bjpop/rubra | A bioinformatics pipeline system that supports running workflow stages on a distributed compute cluster. | 38 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |