flowgraph
Data pipeline framework
A software framework for building scalable, asynchronous data pipelines with explicit back-pressure management and logging capabilities.
Flowgraph package for scalable asynchronous system development
60 stars
2 watching
8 forks
Language: Go
last commit: over 5 years agoLinked from 2 awesome lists
dataflowdistributedscalable
avelino/awesome-go
xwjdsh/awesome-go-extraRelated projects:
| Repository | Description | Stars |
|---|---|---|
 giacbrd/smartpipeline | A framework for designing and executing concurrent data pipelines with a focus on simplicity and efficiency | 25 |
 huggingface/datatrove | A platform-agnostic data processing framework for large-scale text data pipelines | 2,103 |
 dataform-co/dataform | A framework for managing data operations in BigQuery using SQL and software engineering best practices | 860 |
 jexia/semaphore | A tool to manage and expose complex data flows through multiple protocols. | 94 |
 fractalide/fractalide | A platform for composable software using dataflow graphs, focusing on reusable and reproducible nodes for efficient computation | 877 |
 raftlib/raftlib | A C++ library providing a framework for implementing parallel and concurrent data processing pipelines. | 955 |
 log2timeline/dftimewolf | A framework for orchestrating data collection, processing, and export | 299 |
 streamlet-dev/tributary | A Python library for constructing dataflow graphs with support for reactive and lazy evaluation. | 444 |
 acdemiralp/fg | An abstracted rendering pipeline framework describing frames as directed acyclic graphs of render tasks and resources. | 547 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 symphony09/ograph | A framework for building data pipelines with concurrent execution and dependency management | 33 |
 biocorecrg/bionextflow | A collection of reusable modules and sub-workflows for Nextflow pipelines in bioinformatics | 26 |
 pdpipe/pdpipe | Provides a set of pre-defined data processing pipelines for pandas DataFrames. | 718 |
 galaxyproject/galaxy | A platform for data-intensive scientific analysis and workflow management | 1,431 |
 sitecore/data-exchange-framework-docs | A documentation project for an ETL tool used in Sitecore to exchange and process data | 1 |