Skip to content

Issues: stanford-crfm/helm

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

108 Open 987 Closed
108 Open 987 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

Switch AnthropicTokenizer to use client.beta.messages.count_tokens()
#3212 opened Dec 17, 2024 by yifanmai
Issue with Model Execution on the HumanEval Benchmark user question
#3201 opened Dec 6, 2024 by lucas-s-p
3
Add new model NECTEC (nectec/OpenThaiLLM-Prebuilt-7B, nectec/Pathumma-llm-text-1.0.0)
#3188 opened Nov 28, 2024 by JackJessada
3
At least one of model_deployment and model must be specified user question
#3184 opened Nov 25, 2024 by thallysonjsa
5
VHELM setup user question
#3183 opened Nov 24, 2024 by smsarov
1
Reproducing WMT14 user question
#3175 opened Nov 19, 2024 by MaxHahnbueck
1
Installation Guide: Required Dependencies for Supported Operating Systems and Distributions user question
#3160 opened Nov 14, 2024 by shakatoday
5
Is there any way to load GGUF models? user question
#3141 opened Nov 7, 2024 by InAnYan
3
Support few-shot chain-of-thought in GPQA / MMLU
#3088 opened Oct 22, 2024 by yifanmai
Issue with running HEIM documentation Improvements or additions to documentation HEIM (Text2Image) user question
#3080 opened Oct 22, 2024 by sudhir-mcw
3
flake8 in Python 3.12 produces a large number of errors
#3072 opened Oct 18, 2024 by yifanmai
How to use this package when I have the pompts and images across models HEIM (Text2Image) user question
#3062 opened Oct 14, 2024 by snehith3195
1
Not able to install install-heim-extras.sh for heim leaderboard HEIM (Text2Image) user question
#3060 opened Oct 11, 2024 by snehith3195
1
Proposal: Multi-turn support
#3059 opened Oct 10, 2024 by yifanmai
Proposal: User-provided custom adapters
#3058 opened Oct 10, 2024 by yifanmai
HatefulMemesScenario get_instances returning error bug Something isn't working HEIM (Text2Image) methodology Evaluation methodology user question
#3056 opened Oct 10, 2024 by dxwu2
3
Add MMLU-Pro additions New models or scenarios scenarios
#3018 opened Sep 24, 2024 by yifanmai
@siyagoel
Add amazon bedrock 3p models to the model configuration. user question
#2967 opened Sep 3, 2024 by subhaviv
2
Incorrect scoring due to answer format mismatch in MMLU evaluation user question
#2939 opened Aug 16, 2024 by DerryChan
1
Add Windows support
#2907 opened Aug 8, 2024 by yifanmai
Add sorting/filtering functionality to predictions pages frontend Frontend issues
#2887 opened Aug 1, 2024 by farzaank
HEIMHumanEvalScenario requires permissions to download data from codalab HEIM (Text2Image) user question
#2865 opened Jul 30, 2024 by slymane
1
pip needs to be temporarily downgraded to 24.1.2
#2855 opened Jul 30, 2024 by yifanmai
1
Sort by correctness functionality on Predictions page in front end
#2783 opened Jun 29, 2024 by farzaank
@farzaank
Annotator issues
#2697 opened May 31, 2024 by yifanmai
4 tasks
@farzaank
ProTip! Exclude everything labeled bug with -label:bug.