vllm
by vllm-project
A high-throughput and memory-efficient inference and serving engine for LLMs
AI summary
LLM server
An inference and serving engine for large language models
- stars
- 32K
- forks
- 4.9K
- watching
- 258
- awesome lists
- 3
Similar projects
Found by comparing what the projects do, not just their names.
LLM toolkit
An open-source toolkit for pretraining and fine-tuning large language models
LLM toolkit
A toolkit for optimizing and serving large language models
Instruction-following model tuner
An implementation of a method for fine-tuning language models to follow instructions with high efficiency and accuracy
modeltc/lightllm2.7K
LLM server
A Python-based framework for serving large language models with low latency and high scalability.
LLM Quantizer
An open-source software project that enables efficient and accurate low-bit weight quantization for large language models.
lyogavin/airllm5.4K
Inference optimizer
Optimizes large language model inference on limited GPU resources
nomic-ai/gpt4all71.2K
LLM client
An open-source Python client for running Large Language Models (LLMs) locally on any device.
LLM toolkit
A toolkit for fine-tuning and inferring large machine learning models
lm-sys/fastchat37.3K
Chatbot Engine
An open platform for training, serving, and evaluating large language models used in chatbots.
AI application suite
An all-in-one Desktop & Docker AI application with built-in RAG and support for multiple LLMs and vector databases.
ggerganov/llama.cpp69.2K
LLM inference software
Enables LLM inference with minimal setup and high performance on various hardware platforms
Language Model Runtime
An efficient C#/.NET library for running Large Language Models (LLMs) on local devices
Server framework
A fast serving framework for large language models and vision language models.
LLM inference engine
An efficient Large Language Model inference engine leveraging consumer-grade GPUs on PCs
Evaluation framework
Provides a unified framework to test generative language models on various evaluation tasks.
Featured in 3 awesome lists
Each link jumps to the spot where the list mentions vllm.
