Awesome-LLM

Deep-Live-Cam

real time face swap and one-click video deepfake with only a single image (uncensored)

MiniCPM-V 2.6

A GPT-4V Level MLLM for Single Image, Multi Image and Video on Your Phone

GPT-SoVITS

1 min voice data can also be used to train a good TTS model! (few shot voice cloning)

a benchmark platform for large language models (LLMs) that features anonymous, randomized battles in a crowdsourced manner

aims to track, rank, and evaluate LLMs and chatbots as they are released

ACLUE

an evaluation benchmark focused on ancient Chinese language comprehension

An Automatic Evaluator for Instruction-following Language Models using Nous benchmark suite

A pioneering benchmark specifically designed to assess honesty in LLMs comprehensively

evaluates LLM's ability to call external functions/tools

Chinese Large Model Leaderboard

an expert-driven benchmark for Chineses LLMs

CompassRank is dedicated to exploring the most advanced language and visual models, offering a comprehensive, objective, and neutral evaluation reference for the industry and research

a benchmark evaluating QA methods that operate over a mixture of heterogeneous input sources (KB, text, tables, infoboxes)

a benchmark for evaluating the performance of large language models (LLMs) in various tasks related to both textual and visual imagination

a meta-benchmark that evaluates how well factuality evaluators assess the outputs of large language models (LLMs)

a benchmark designed to evaluate large language models (LLMs) specifically in their ability to answer real-world coding-related questions

a benchmark designed to evaluate large language models in the legal domain

focuses on understanding how these models perform in various scenarios and analyzing results from an interpretability perspective

a benchmark that evaluates large language models on a variety of multimodal reasoning tasks, including language, natural and social sciences, physical and social commonsense, temporal reasoning, algebra, and geometry

a comprehensive benchmarking platform designed to evaluate large models' mathematical abilities across 20 fields and nearly 30,000 math problems

a ground-truth-based dynamic benchmark derived from off-the-shelf benchmark mixtures, which evaluates LLMs with a highly capable model ranking (i.e., 0.96 correlation with Chatbot Arena) while running locally and quickly (6% the time and cost of running MMLU)

a benchmark that evaluates large language models' ability to answer medical questions across multiple languages

a multimodal question-answering benchmark designed to evaluate AI models' cognitive ability to understand human beliefs and goals

a benchmark for evaluating AI models across multiple academic disciplines like math, physics, chemistry, biology, and more

a biomedical question-answering benchmark designed for answering research-related questions using PubMed abstracts

benchmark designed to evaluate large language models (LLMs) on solving complex, college-level scientific problems from domains like chemistry, physics, and mathematics

a benchmark platform designed for evaluating large language models (LLMs) on a range of tasks, particularly focusing on their performance in different aspects such as natural language understanding, reasoning, and generalization

a Swedish language understanding benchmark that evaluates natural language processing (NLP) models on various tasks such as argumentation analysis, semantic similarity, and textual entailment

a large-scale Document Visual Question Answering (VQA) dataset designed for complex document understanding, particularly in financial reports

a large-scale question-answering benchmark focused on real-world financial data, integrating both tabular and textual information

a benchmark designed to assess the performance of multimodal web agents on realistic visually grounded tasks

a benchmark that evaluates large multimodal models (LMMs) on their ability to perform human-like mathematical reasoning

a benchmark dataset testing AI's ability to reason about visual commonsense through images that defy normal expectations

IBM data-prep-kit

Open-Source Toolkit for Efficient Unstructured Data Processing with Pre-built Modules and Local to Cluster Scalability

lm-evaluation-harness

A framework for few-shot evaluation of language models

MixEval

A reliable click-and-go evaluation suite compatible with both open-source and proprietary models, supporting MixEval and other benchmarks

lighteval

a lightweight LLM evaluation suite that Hugging Face has been using internally

OLMO-eval

a repository for evaluating open language models

instruct-eval

This repository contains code to quantitatively evaluate instruction-tuned models such as Alpaca and Flan-T5 on held-out tasks

simple-evals

Eval tools by OpenAI

Giskard

Testing & evaluation library for LLM applications, in particular RAGs

a unified platform from LangChain framework for: evaluation, collaboration HITL (Human In The Loop), logging and monitoring LLM applications

Ragas

a framework that helps you evaluate your Retrieval Augmented Generation (RAG) pipelines

DeepSpeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective

Megatron-DeepSpeed

DeepSpeed version of NVIDIA's Megatron-LM that adds additional support for several features such as MoE model training, Curriculum Learning, 3D Parallelism, and others

torchtune

A Native-PyTorch Library for LLM Fine-tuning

torchtitan

A native PyTorch Library for large model training

NeMo Framework

Generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains

Megatron-LM

Ongoing research training transformer models at scale

Colossal-AI

Making large AI models cheaper, faster, and more accessible

BMTrain

Efficient Training for Big Models

Mesh Tensorflow

Mesh TensorFlow: Model Parallelism Made Easier

maxtext

A simple, performant and scalable Jax LLM!

GPT-NeoX

An implementation of model parallel autoregressive transformers on GPUs, based on the DeepSpeed library

SGLang

SGLang is a fast serving framework for large language models and vision language models

vLLM

A high-throughput and memory-efficient inference and serving engine for LLMs

a toolkit for deploying and serving Large Language Models (LLMs)

exllama

A more memory-efficient rewrite of the HF transformers implementation of Llama for use with quantized weights

llama.cpp

LLM inference in C/C++

ollama

Get up and running with Llama 3, Mistral, Gemma, and other large language models

Langfuse

Open Source LLM Engineering Platform 🪢 Tracing, Evaluations, Prompt Management, Evaluations and Playground

FastChat

A distributed multi-model LLM serving system with web UI and OpenAI-compatible RESTful APIs

mistral.rs

Blazingly fast LLM inference

MindSQL

A python package for Txt-to-SQL with self hosting functionalities and RESTful APIs compatible with proprietary as well as open source LLM

SkyPilot

Run LLMs and batch jobs on any cloud. Get maximum cost savings, highest GPU availability, and managed execution -- all with a simple interface

an open-source NLP framework that allows you to use LLMs and transformer-based models from Hugging Face, OpenAI and Cohere to interact with your own data

Sidekick

Data integration platform for LLMs

QA-Pilot

An interactive chat project that leverages Ollama/OpenAI/MistralAI LLMs for rapid understanding and navigation of GitHub code repository or compressed file resources

Shell-Pilot

Interact with LLM using Ollama models(or openAI, mistralAI)via pure shell scripts on your Linux(or MacOS) system, enhancing intelligent system management without any dependencies

LangChain

Building applications with LLMs through composability

Floom

AI gateway and marketplace for developers, enables streamlined integration of AI features into products

Swiss Army Llama

Comprehensive set of tools for working with local LLMs for various tasks

LiteChain

Lightweight alternative to LangChain for composing LLMs

magentic

Seamlessly integrate LLMs as Python functions

wechat-chatgpt

Use ChatGPT On Wechat via wechaty

promptfoo

Test your prompts. Evaluate and compare LLM outputs, catch regressions, and improve prompt quality

Agenta

Easily build, version, evaluate and deploy your LLM-powered apps

Serge

a chat interface crafted with llama.cpp for running Alpaca models. No API keys, entirely self-hosted!

Langroid

Harness LLMs with Multi-Agent Programming

Embedchain

Framework to create ChatGPT like bots over your dataset

Opik

Confidently evaluate, test, and ship LLM applications with a suite of observability tools to calibrate language model outputs across your dev and production lifecycle

IntelliServer

simplifies the evaluation of LLMs by providing a unified microservice to access and test multiple AI models

OpenLLM

Fine-tune, serve, deploy, and monitor any open-source LLMs in production. Used in production at for LLMs-based applications

DeepSpeed-Mii

MII makes low-latency and high-throughput inference, similar to vLLM powered by DeepSpeed

Text-Embeddings-Inference

Inference for text-embeddings in Rust, HFOIL Licence

Infinity

Inference for text-embeddings in Python

TensorRT-LLM

Nvidia Framework for LLM Inference

FasterTransformer

NVIDIA Framework for LLM Inference(Transitioned to TensorRT-LLM)

Flash-Attention

A method designed to enhance the efficiency of Transformer models

Langchain-Chatchat

Formerly langchain-ChatGLM, local knowledge based LLM (like ChatGLM) QA app with langchain

Search with Lepton

Build your own conversational search engine using less than 500 lines of code by

Robocorp

Create, deploy and operate Actions using Python anywhere to enhance your AI agents and assistants. Batteries included with an extensive set of libraries, helpers and logging

LMDeploy

A high-throughput and low-latency inference and serving framework for LLMs and VLs

Playground for devs to finetune & deploy LLMs

LLocalSearch

Locally running websearch using LLM chains

AI Gateway

— Gateway streamlines requests to 100+ open & closed source models with a unified API. It is also production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency

talkd.ai dialog

Simple API for deploying any RAG or LLM that you want adding plugins

Wllama

WebAssembly binding for llama.cpp - Enabling in-browser LLM inference

GPUStack

An open-source GPU cluster manager for running LLMs

MNN-LLM

-- A Device-Inference framework, including LLM Inference on device(Mobile Phone/PC/IOT)

First LLM Multi-agent framework

AdalFlow

AdalFlow: The library to build&auto-optimize LLM applications

dspy

DSPy: The framework for programming—not prompting—foundation models

MLflow: An open-source framework for the end-to-end machine learning lifecycle, helping developers track experiments, evaluate models/prompts, deploy models, and add observability with tracing

YiVal

— Evaluate and Evolve: YiVal is an open-source GenAI-Ops tool for tuning and evaluating prompts, configurations, and model parameters using customizable datasets, evaluation methods, and improvement strategies

Guidance

— A handy looking Python library from Microsoft that uses Handlebars templating to interleave generation, prompting, and logical control

LangChain

— A popular Python/JavaScript library for chaining sequences of language model prompts

Evidently

— An open-source framework to evaluate, test and monitor ML and LLM-powered systems

: A Python library for automating selection of models, hyperparameters, and other tunable choices

— A Python library for making chatbot interfaces

— A Python library for validating outputs and retrying failures. Still in alpha, so expect sharp edges and bugs

Semantic Kernel

— A Python/C#/Java library from Microsoft that supports prompt templating, function chaining, vectorized memory, and intelligent planning

Prompttools

— Open-source Python tools for testing and evaluating models, vector DBs, and prompts

Outlines

— A Python library that provides a domain-specific language to simplify prompting and constrain generation

Promptify

— A small Python library for using language models to perform NLP tasks

— A paid product for building, comparing, and shipping language model apps

— A paid product for testing and improving prompts

— A paid product for tracking model training and prompt engineering experiments

OpenAI Evals

— An open-source library for evaluating task performance of language models and prompts

LlamaIndex

— A Python library for augmenting LLM apps with data

— A paid product for detecting toxicity, hallucination, prompt injection, etc

— A programming language for LLM interaction with support for typed prompting, control flow, constraints, and tools

ModelFusion

A TypeScript library for building apps with LLMs and other ML models (speech-to-text, text-to-speech, image generation)

Flappy

— Production-Ready LLM Agent SDK for Every Developer

GPTRouter is an open source LLM API Gateway that offers a universal API for 30+ LLMs, vision, and image models, with smart fallbacks based on uptime and latency, automatic retries, and streaming. Stay operational even when OpenAI is down

QAnything

A local knowledge base question-answering system designed to support a wide range of file formats and databases

— A bilingual Chinese-English knowledge extraction model with knowledge graphs and natural language processing technologies

llm-ui

A React library for building LLM UIs

A web-hosted IDE where non-technical domain experts work with AI Engineers to build task-specific AI agents. We approach prompting as a new programming language rather than low/no-code blocks

Deploy, manage, optimize any model at scale across any environment from cloud to edge. Let's you go from python notebook to inferencing in minutes

Dify

An open-source LLM app development platform with an intuitive interface that streamlines AI workflows, model management, and production deployment

LazyLLM

An open-source LLM app for building multi-agent LLMs applications in an easy and lazy way, supports model deployment and fine-tuning

MemFree

Open Source Hybrid AI Search Engine, Instantly Get Accurate Answers from the Internet, Bookmarks, Notes, and Docs. Support One-Click Deployment

unslothai

A framework that specializes in efficient fine-tuning. On its GitHub page, you can find ready-to-use fine-tuning templates for various LLMs, allowing you to easily train your own data for free on the Google Colab cloud

AutoRAG

Open source AutoML tool for RAG. Optimize the RAG answer quality automatically. From generation evaluation datset to deploying optimized RAG pipeline

llm-course

Course to get into Large Language Models (LLMs) with roadmaps and Colab notebooks

Recent Advances on Foundation Models

Minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization

femtoGPT

Pure Rust implementation of a minimal Generative Pretrained Transformer

it comes with a that showcases a lot of the functionality

A guide to building your own working LLM

explains how to code a Generative Pre-trained Transformer, or GPT, from scratch

Explore the world of Large Language Models with over 275 custom made figures in this illustrated guide!

Open-source tool for ML observability that runs in your notebook environment. Monitor and fine tune LLM, CV and Tabular Models

The latest AI news, curated & explained by GPT-4

Share your wildest ChatGPT conversations with one click

Introducing Cohere Summarize Beta: A New Endpoint for Text Summarization

chatgpt-wrapper

ChatGPT Wrapper is an open-source unofficial Python API and CLI that lets you interact with ChatGPT

Open-evals

A framework extend openai's for different language model

Write, edit, and chat about your code with a powerful AI

AutoGPT

an experimental open-source application showcasing the capabilities of the GPT-4 language model

OpenAGI

When LLM Meets Domain Experts

EasyEdit

An easy-to-use framework to edit large language models

chatgpt-shroud

A Chrome extension for OpenAI's ChatGPT, enhancing user privacy by enabling easy hiding and unhiding of chat history. Ideal for privacy during screen shares