flowgraph
Data pipeline framework
A software framework for building scalable, asynchronous data pipelines with explicit back-pressure management and logging capabilities.
Flowgraph package for scalable asynchronous system development
60 stars
2 watching
8 forks
Language: Go
last commit: over 3 years ago
Linked from 2 awesome lists
dataflowdistributedscalable
avelino/awesome-go
xwjdsh/awesome-go-extraRelated projects:
| Repository | Description | Stars |
|---|---|---|
 giacbrd/smartpipeline | A framework for designing and executing concurrent data pipelines with a focus on simplicity and efficiency | 23 |
 huggingface/datatrove | A platform-agnostic data processing framework for large-scale text data pipelines | 2,043 |
 dataform-co/dataform | A framework for managing data operations in BigQuery using SQL and software engineering best practices | 850 |
 jexia/semaphore | A tool for building and managing complex data flows across multiple protocols | 94 |
 fractalide/fractalide | A platform for composable software using dataflow graphs, focusing on reusable and reproducible nodes for efficient computation | 873 |
 raftlib/raftlib | A C++ library providing a framework for implementing parallel and concurrent data processing pipelines. | 954 |
 log2timeline/dftimewolf | A framework for orchestrating data collection, processing, and export | 296 |
 streamlet-dev/tributary | A Python library for constructing dataflow graphs with support for reactive and lazy evaluation. | 442 |
 acdemiralp/fg | An abstracted rendering pipeline framework describing frames as directed acyclic graphs of render tasks and resources. | 544 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 symphony09/ograph | A framework for building data pipelines with concurrent execution and dependency management | 32 |
 biocorecrg/bionextflow | A collection of reusable modules and sub-workflows for Nextflow pipelines in bioinformatics | 26 |
 pdpipe/pdpipe | A tool for creating and managing data pipelines with pandas DataFrames | 716 |
 galaxyproject/galaxy | An integrated framework for data-intensive scientific analysis and workflow management | 1,410 |
 sitecore/data-exchange-framework-docs | A documentation project for an ETL tool used in Sitecore to exchange and process data | 1 |