helm
ModelEvaluator
A framework to evaluate and compare language models by analyzing their performance on various tasks
Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110). This framework is also used to evaluate text-to-image models in HEIM (https://arxiv.org/abs/2311.04287) and vision-language models in VHELM (https://arxiv.org/abs/2410.07112).
2k stars
36 watching
250 forks
Language: Python
last commit: 7 days ago
Linked from 1 awesome list
ethicalml/awesome-production-machine-learningRelated projects:
| Repository | Description | Stars |
|---|---|---|
 huggingface/evaluate | An evaluation framework for machine learning models and datasets, providing standardized metrics and tools for comparing model performance. | 2,034 |
 chenllliang/mmevalpro | A benchmarking framework for evaluating Large Multimodal Models by providing rigorous metrics and an efficient evaluation pipeline. | 22 |
| stanford-crfm/levanter | A framework for building and training large language models with focus on reproducibility, scalability, and performance. | 516 |
 flageval-baai/flageval | An evaluation toolkit and platform for assessing large models in various domains | 300 |
 modelscope/evalscope | A framework for efficient large model evaluation and performance benchmarking. | 248 |
 declare-lab/instruct-eval | An evaluation framework for large language models trained with instruction tuning methods | 528 |
 allenai/olmo-eval | An evaluation framework for large language models. | 311 |
 mshukor/evalign-icl | Evaluating and improving large multimodal models through in-context learning | 20 |
 openai/simple-evals | A library for evaluating language models using standardized prompts and benchmarking tests. | 1,939 |
 esmvalgroup/esmvaltool | A community-developed tool for evaluating climate models and providing diagnostic metrics. | 223 |
 yuweihao/mm-vet | Evaluates the capabilities of large multimodal models using a set of diverse tasks and metrics | 267 |
 modeloriented/dalex | A tool to help understand and explain the behavior of complex machine learning models | 1,375 |
 marvinteichmann/convcrf | An implementation of a convolutional Conditional Random Field model for semantic segmentation tasks. | 564 |
| huggingface/lighteval | A toolkit for evaluating Large Language Models across multiple backends | 804 |
 edublancas/sklearn-evaluation | A tool for evaluating and visualizing machine learning model performance | 3 |