TensorRT-LLM
by NVIDIA
TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
AI summary
Inference optimizer
A software framework providing an easy-to-use Python API to optimize Large Language Models on NVIDIA GPUs for efficient inference.
- stars
- 8.9K
- forks
- 1K
- watching
- 95
- awesome list
- 1
Similar projects
Found by comparing what the projects do, not just their names.
LLM optimizer
A tool for optimizing large language models by collecting feedback and metrics to improve their performance over time
Deep learning accelerator
Provides a set of tools and libraries for optimizing deep learning inference on NVIDIA GPUs.
mlc-ai/mlc-llm19.4K
LLM Deployment Engine
A machine learning compiler and deployment engine for large language models
Transformer component
A high-performance transformer-based NLP component optimized for GPU acceleration and integration into various frameworks.
microsoft/deepspeed35.9K
Deep Learning Optimizer
A deep learning optimization library that simplifies distributed training and inference on modern computing hardware.
LLM optimization kernels
A collection of optimized kernels and post-training loss functions for large language models
LLM toolkit
A toolkit for optimizing and serving large language models
LLM inference engine
An efficient Large Language Model inference engine leveraging consumer-grade GPUs on PCs
Tensor regressor
A toolbox implementing a sparse and low-rank tensor regression algorithm with boosting
LLM toolkit
An integrated development platform for large language models (LLMs) that provides observability, analytics, and management tools.
Server framework
A fast serving framework for large language models and vision language models.
AI framework
A deep learning and reinforcement learning library that provides an extensive collection of customizable neural layers to build advanced AI models quickly.
Model optimizer
Tools and techniques for optimizing large language models on various frameworks and hardware platforms.
modeltc/lightllm2.7K
LLM server
A Python-based framework for serving large language models with low latency and high scalability.
LLM builder
Builds composable LLM applications with Java
Featured in 1 awesome list
Each link jumps to the spot where the list mentions TensorRT-LLM.
