ipex-llm

lyogavin/airllm5.4K

Inference optimizer

Optimizes large language model inference on limited GPU resources

sjtu-ipads/powerinfer8K

LLM inference engine

An efficient Large Language Model inference engine leveraging consumer-grade GPUs on PCs

vllm-project/vllm32K

LLM server

An inference and serving engine for large language models

internlm/lmdeploy4.9K

LLM toolkit

A toolkit for optimizing and serving large language models

fminference/flexllmgen9.2K

Batch processor

Generates large language model outputs in high-throughput mode on single GPUs

nomic-ai/gpt4all71.2K

LLM client

An open-source Python client for running Large Language Models (LLMs) locally on any device.

haotian-liu/llava20.7K

Visual Instruction System

A system that uses large language and vision models to generate and process visual instructions

autogptq/autogptq4.6K

Model optimizer

A package for optimizing large language models for efficient inference on GPUs and other hardware platforms.

modeltc/lightllm2.7K

LLM server

A Python-based framework for serving large language models with low latency and high scalability.

ggerganov/llama.cpp69.2K

LLM inference software

Enables LLM inference with minimal setup and high performance on various hardware platforms

microsoft/deepspeed35.9K

Deep Learning Optimizer

A deep learning optimization library that simplifies distributed training and inference on modern computing hardware.

mit-han-lab/llm-awq2.6K

LLM Quantizer

An open-source software project that enables efficient and accurate low-bit weight quantization for large language models.

linkedin/liger-kernel3.8K

LLM optimization kernels

A collection of optimized kernels and post-training loss functions for large language models

facebookincubator/aitemplate4.6K

Neural network optimizer

A framework that transforms deep neural networks into high-performance GPU-optimized C++ code for efficient inference serving.

sgl-project/sglang6.6K

Server framework

A fast serving framework for large language models and vision language models.