lighteval
LLM evaluator
A toolkit for evaluating Large Language Models across multiple backends
Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends
804 stars
30 watching
95 forks
Language: Python
last commit: 6 days ago
Linked from 2 awesome lists
evaluationevaluation-frameworkevaluation-metricshuggingface
hannibal046/awesome-llm
ethicalml/awesome-production-machine-learningRelated projects:
| Repository | Description | Stars |
|---|---|---|
| huggingface/evaluate | An evaluation framework for machine learning models and datasets, providing standardized metrics and tools for comparing model performance. | 2,034 |
 mlabonne/llm-autoeval | A tool to automate the evaluation of large language models in Google Colab using various benchmarks and custom parameters. | 558 |
 psycoy/mixeval | An evaluation suite and dynamic data release platform for large language models | 224 |
 evolvinglmms-lab/lmms-eval | Tools and evaluation suite for large multimodal models | 2,058 |
 chenllliang/mmevalpro | A benchmarking framework for evaluating Large Multimodal Models by providing rigorous metrics and an efficient evaluation pipeline. | 22 |
 allenai/olmo-eval | An evaluation framework for large language models. | 310 |
 mlgroupjlu/llm-eval-survey | A repository of papers and resources for evaluating large language models. | 1,433 |
 flageval-baai/flageval | An evaluation toolkit and platform for assessing large models in various domains | 300 |
 prometheus-eval/prometheus-eval | An open-source framework that enables language model evaluation using Prometheus and GPT4 | 796 |
 open-compass/vlmevalkit | A toolkit for evaluating large vision-language models on various benchmarks and datasets. | 1,343 |
| huggingface/torchmoji | An open-source implementation of a deep learning model that analyzes sentiment and emotion in text using emojis | 918 |
 maluuba/nlg-eval | A toolset for evaluating and comparing natural language generation models | 1,347 |
| huggingface/pytorch-openai-transformer-lm | Implementing OpenAI's transformer language model in PyTorch with pre-trained weights and fine-tuning capabilities | 1,511 |
 aiverify-foundation/llm-evals-catalogue | A collaborative catalogue of Large Language Model evaluation frameworks and papers. | 14 |
 h2oai/h2o-llm-eval | An evaluation framework for large language models with Elo rating system and A/B testing capabilities | 50 |