VLMEvalKit
Evaluation framework
A toolkit for evaluating large vision-language models on various benchmarks and datasets.
Open-source evaluation toolkit of large vision-language models (LVLMs), support 160+ VLMs, 50+ benchmarks
1k stars
10 watching
188 forks
Language: Python
last commit: 4 days ago
Linked from 1 awesome list
chatgptclaudeclipcomputer-visionevaluationgeminigptgpt-4vgpt4large-language-modelsllavallmmulti-modalopenaiopenai-apipytorchqwenvitvqa
ethicalml/awesome-production-machine-learningRelated projects:
| Repository | Description | Stars |
|---|---|---|
| open-compass/lawbench | Evaluates the legal knowledge of large language models using a custom benchmarking framework. | 267 |
 evolvinglmms-lab/lmms-eval | Tools and evaluation suite for large multimodal models | 2,058 |
 freedomintelligence/mllm-bench | Evaluates and compares the performance of multimodal large language models on various tasks | 55 |
| open-compass/mmbench | A collection of benchmarks to evaluate the multi-modal understanding capability of large vision language models. | 163 |
 chenllliang/mmevalpro | A benchmarking framework for evaluating Large Multimodal Models by providing rigorous metrics and an efficient evaluation pipeline. | 22 |
 openai/simple-evals | A library for evaluating language models using standardized prompts and benchmarking tests. | 1,939 |
 mshukor/evalign-icl | Evaluating and improving large multimodal models through in-context learning | 20 |
 prometheus-eval/prometheus-eval | An open-source framework that enables language model evaluation using Prometheus and GPT4 | 796 |
 huggingface/lighteval | A toolkit for evaluating Large Language Models across multiple backends | 804 |
 pkunlp-icler/pca-eval | An open-source benchmark and evaluation tool for assessing multimodal large language models' performance in embodied decision-making tasks | 100 |
 mlgroupjlu/llm-eval-survey | A repository of papers and resources for evaluating large language models. | 1,433 |
 opengvlab/visionllm | A large language model designed to process and generate visual information | 915 |
 edublancas/sklearn-evaluation | A tool for evaluating and visualizing machine learning model performance | 3 |
 maluuba/nlg-eval | A toolset for evaluating and comparing natural language generation models | 1,347 |
 esmvalgroup/esmvaltool | A community-developed tool for evaluating climate models and providing diagnostic metrics. | 223 |