pathway
ETL framework
An ETL framework that enables real-time data processing and analytics using Python, with support for streaming data, batch processing, machine learning, and integration with various external data sources.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
4k stars
29 watching
139 forks
Language: Python
last commit: 3 days ago
Linked from 1 awesome list
batch-processingdata-analyticsdata-pipelinesdata-processingdataflowetletl-frameworkiot-analyticskafkamachine-learning-algorithmspathwaypythonreal-timeruststream-processingstreamingtime-series-analysis
manuzhang/awesome-streamingRelated projects:
| Repository | Description | Stars |
|---|---|---|
| pathwaycom/llm-app | Pre-built templates for integrating large language models into enterprise applications with real-time data APIs and various data sources. | 4,642 |
 robinhood/faust | Builds high-performance distributed systems and real-time data pipelines using asynchronous event processing and in-memory durable key-value stores. | 6,746 |
 amphi-ai/amphi-etl | A Python-based ETL tool for data transformation and pipeline development with low-code interface and native code generation. | 904 |
 tschellenbach/stream-framework | A Python library for building activity streams and newsfeeds using distributed data stores | 4,733 |
 entilzha/pyfunctional | A Python library for creating data pipelines using functional programming principles | 2,403 |
 agermanidis/livepython | A desktop application that visually traces the execution of Python code in real-time. | 2,553 |
 ml-tooling/opyrator | Automates conversion of machine learning code into production-ready microservices with web API and GUI. | 3,102 |
 pipedreamhq/pipedream | An integration platform for automating data flows between applications and services. | 8,981 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 flyteorg/flyte | An orchestrator platform that enables the building and deployment of production-grade data pipelines. | 5,785 |
 matz/streem | A programming language and runtime for building data-flow programs with concurrent execution. | 4,603 |
 mara/mara-pipelines | A lightweight ETL framework providing a simple way to define and execute data transformation pipelines using declarative Python code. | 2,081 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 215 |
 pysimplegui/pysimplegui | A Python GUI library that simplifies the development of desktop applications with a simple and intuitive interface. | 13,441 |
 orchest/orchest | Builds data pipelines by allowing direct coding in Python, R, or Julia without frameworks or YAML files. | 4,079 |