pathway
ETL framework
A Python-based ETL framework for stream processing and real-time analytics with a scalable Rust engine.
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
7k stars
29 watching
155 forks
Language: Python
last commit: 3 months ago
Linked from 1 awesome list
batch-processingdata-analyticsdata-pipelinesdata-processingdataflowetletl-frameworkiot-analyticskafkamachine-learning-algorithmspathwaypythonreal-timeruststream-processingstreamingtime-series-analysis
manuzhang/awesome-streamingRelated projects:
| Repository | Description | Stars |
|---|---|---|
| pathwaycom/llm-app | Provides pre-built AI application templates to integrate Large Language Models (LLMs) with various data sources for scalable RAG and enterprise search. | 7,426 |
 robinhood/faust | Builds high-performance distributed systems and real-time data pipelines using asynchronous event processing and in-memory durable key-value stores. | 6,751 |
 amphi-ai/amphi-etl | A tool that enables data analysts to create and manage data pipelines with an intuitive interface, generating Python code for deployment anywhere. | 933 |
 tschellenbach/stream-framework | A Python library for building activity streams and newsfeeds using distributed data stores | 4,733 |
 entilzha/pyfunctional | A Python library for creating data pipelines using functional programming principles | 2,407 |
 agermanidis/livepython | A desktop application that visually traces the execution of Python code in real-time. | 2,556 |
 ml-tooling/opyrator | Automates conversion of machine learning code into production-ready microservices with web API and GUI. | 3,116 |
 pipedreamhq/pipedream | An integration platform that enables developers to automate workflows across multiple applications and services using pre-built components and custom code | 9,075 |
 databiosphere/toil | A workflow management system designed to efficiently run pipelines in various environments. | 901 |
 flyteorg/flyte | An orchestrator platform that enables the building and deployment of production-grade data pipelines. | 5,850 |
 matz/streem | A programming language and runtime for building data-flow programs with concurrent execution. | 4,605 |
 mara/mara-pipelines | A lightweight ETL framework providing a simple way to define and execute data transformation pipelines using declarative Python code. | 2,082 |
 pipefunc/pipefunc | Automates and simplifies the creation of function pipelines for efficient execution of scientific workflows. | 230 |
 pysimplegui/pysimplegui | A Python GUI library that simplifies the development of desktop applications with a simple and intuitive interface. | 13,480 |
 orchest/orchest | Builds data pipelines by allowing direct coding in Python, R, or Julia without frameworks or YAML files. | 4,091 |