PowerInfer
by SJTU-IPADS
High-speed Large Language Model Serving on PCs with Consumer-grade GPUs
AI summary
LLM inference engine
An efficient Large Language Model inference engine leveraging consumer-grade GPUs on PCs
- stars
- 8K
- forks
- 415
- watching
- 78
- awesome lists
- 2
Similar projects
Found by comparing what the projects do, not just their names.
Batch processor
Generates large language model outputs in high-throughput mode on single GPUs
LLM toolkit
A toolkit for optimizing and serving large language models
microsoft/deepspeed35.9K
Deep Learning Optimizer
A deep learning optimization library that simplifies distributed training and inference on modern computing hardware.
lyogavin/airllm5.4K
Inference optimizer
Optimizes large language model inference on limited GPU resources
thudm/glm-130b7.7K
Bilingual Language Model
An open-source implementation of a large bilingual language model pre-trained on vast amounts of text data.
Server framework
A fast serving framework for large language models and vision language models.
LLM server
An inference and serving engine for large language models
hpcaitech/colossalai38.9K
AI parallelism toolkit
A toolkit for training and deploying large AI models in parallel on distributed computing infrastructure
xiaomi/mace4.9K
Mobile AI Compute Engine
A framework for deep learning inference on mobile devices
rapidsai/cuml4.3K
GPU ML library
A suite of libraries implementing machine learning algorithms and mathematical primitives on NVIDIA GPUs
LLM Quantizer
An open-source software project that enables efficient and accurate low-bit weight quantization for large language models.
Text Generation Toolkit
A toolkit for deploying and serving Large Language Models (LLMs) for high-performance text generation
autumnai/leaf5.6K
Machine learning framework
An open machine learning framework for building classical, deep, or hybrid models on various hardware platforms.
Transformer model
A PyTorch model definition and inference/sampling code repository for a powerful diffusion transformer with fine-grained Chinese understanding
GPU orchestration framework
A framework for efficient and fault-tolerant distributed training of large neural networks on multiple GPUs.
Featured in 2 awesome lists
Each link jumps to the spot where the list mentions PowerInfer.
