mara-pipelines
Pipeline framework
A lightweight ETL framework providing a simple way to define and execute data transformation pipelines using declarative Python code.
A lightweight opinionated ETL framework, halfway between plain scripts and Apache Airflow
2k stars
56 watching
103 forks
Language: Python
last commit: 11 months ago
Linked from 3 awesome lists
datadata-integrationetlpipelinepostgresqlpython
oxnr/awesome-bigdata
pditommaso/awesome-pipeline
pawl/awesome-etlRelated projects:
| Repository | Description | Stars |
|---|---|---|
 m3dev/gokart | A framework that solves common problems in machine learning pipeline development and provides an environment for reproducibility and team collaboration. | 318 |
 giacbrd/smartpipeline | A framework for designing and executing concurrent data pipelines with a focus on simplicity and efficiency | 23 |
 paysure/orinoco | A functional composable pipeline framework for Python that separates business logic from implementation. | 11 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 neuraxio/neuraxle | A machine learning pipeline library that enables the creation of modular and reusable data processing workflows | 608 |
 huggingface/datatrove | A platform-agnostic data processing framework for large-scale text data pipelines | 2,043 |
 calebwin/pipelines | A language and runtime for crafting massively parallel data pipelines | 374 |
 pwwang/pipen | A Python-based workflow automation framework that enables easy creation of data processing pipelines | 103 |
 bjpop/rubra | A bioinformatics pipeline system that supports running workflow stages on a distributed compute cluster. | 38 |
 thbar/kiba | A Ruby-based framework for defining and running reliable, concise, and maintainable ETL jobs | 1,754 |
 sitecore/data-exchange-framework-docs | A documentation project for an ETL tool used in Sitecore to exchange and process data | 1 |
 amphi-ai/amphi-etl | A Python-based ETL tool for data transformation and pipeline development with low-code interface and native code generation. | 904 |
 renkun-ken/piper | Provides functions and methods to chain operations in R, enhancing readability and maintainability of data pipelines. | 169 |
 acdemiralp/fg | An abstracted rendering pipeline framework describing frames as directed acyclic graphs of render tasks and resources. | 544 |
 ypares/porcupine | A tool that enables data manipulation and analysis pipelines to be flexible, reusable, and reproducible in different environments | 89 |