hamilton
Dataflow framework
Helps define and manage data transformations with a modular, self-documenting, and portable framework for directed acyclic graphs (DAGs) of data transformations.
Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
2k stars
17 watching
124 forks
Language: Jupyter Notebook
last commit: 7 days ago
Linked from 4 awesome lists
dagdata-analysisdata-engineeringdata-sciencedataframeetletl-frameworketl-pipelinefeature-engineeringhacktoberfestlineagellmopsmachine-learningmlopsorchestrationpandaspythonragsoftware-engineering
ethicalml/awesome-production-machine-learning
pditommaso/awesome-pipeline
kelvins/awesome-mlops
krzjoa/awesome-python-data-scienceRelated projects:
| Repository | Description | Stars |
|---|---|---|
 nipype/pydra | A lightweight Python dataflow engine for building and executing directed acyclic graphs (DAGs) in a scalable manner. | 120 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |
| dagworks-inc/burr | A framework for building applications that make decisions using LLMs and state machines. | 1,266 |
 rhosocial/go-dag | A framework for managing and executing workflows described by directed acyclic graphs. | 21 |
 chunelfeng/cgraph | A C++ framework for building directed acyclic graph-based parallel computations with dependency and scheduling management | 1,779 |
 man-group/mdf | A toolkit for expressing programs as directed acyclic graphs and wiring together computations over time-series data. | 169 |
 symphony09/ograph | A framework for building data pipelines with concurrent execution and dependency management | 32 |
 graphprotocol/graph-client | A library and toolset for building fast, performant GraphQL-based decentralized applications | 174 |
 eclipse-zenoh-flow/zenoh-flow | A framework for declarative data flow programming and edge computing | 91 |
 daostack/arc | A platform providing a modular, upgradeable infrastructure for decentralized autonomous organizations (DAOs) on the Ethereum blockchain. | 47 |
 yadage/adage | A package to dynamically build and manage directed acyclic graphs (DAGs) of tasks that can be executed in parallel or sequentially. | 55 |
 erikbrinkman/d3-dag | A library that provides a data structure and algorithms for visualizing directed acyclic graphs | 1,454 |
 dagster-io/dagster | An orchestration platform for data pipelines and assets, providing a declarative programming model and integrated lineage and observability. | 11,699 |
 jexia/semaphore | A tool for building and managing complex data flows across multiple protocols | 94 |
 google/digitalbuildings | Provides tools and an ontology for representing and managing structured information about buildings and equipment. | 371 |