Skip to content
/ helm Public
forked from stanford-crfm/helm

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110).

License

Notifications You must be signed in to change notification settings

vijilAI/helm

This branch is 1192 commits behind stanford-crfm/helm:main.

Folders and files

NameName
Last commit message
Last commit date
Aug 22, 2023
Aug 22, 2023
Sep 14, 2023
Sep 15, 2023
Jun 22, 2023
Dec 29, 2021
Nov 22, 2022
Jul 25, 2023
Oct 12, 2022
Jun 22, 2023
Mar 22, 2023
Nov 16, 2022
Aug 24, 2023
Sep 6, 2023
Sep 6, 2023
May 9, 2023
Aug 24, 2023
May 13, 2023
Mar 9, 2023
Aug 24, 2023
Aug 24, 2023

Repository files navigation

Holistic Evaluation of Language Models

Welcome! The crfm-helm Python package contains code used in the Holistic Evaluation of Language Models project (paper, website) by Stanford CRFM. This package includes the following features:

  • Collection of datasets in a standard format (e.g., NaturalQuestions)
  • Collection of models accessible via a unified API (e.g., GPT-3, MT-NLG, OPT, BLOOM)
  • Collection of metrics beyond accuracy (efficiency, bias, toxicity, etc.)
  • Collection of perturbations for evaluating robustness and fairness (e.g., typos, dialect)
  • Modular framework for constructing prompts from datasets
  • Proxy server for managing accounts and providing unified interface to access models

To get started, refer to the documentation on Read the Docs for how to install and run the package.

About

Holistic Evaluation of Language Models (HELM), a framework to increase the transparency of language models (https://arxiv.org/abs/2211.09110).

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 94.9%
  • JavaScript 3.7%
  • Jupyter Notebook 0.6%
  • HTML 0.5%
  • Shell 0.2%
  • CSS 0.1%