Skip to content

Fix(requirements): set evaluate>=0.4.5 to resolve GLUE crash with datasets>=4.0.0#2291

Merged
regisss merged 1 commit into
huggingface:mainfrom
HabanaAI:main-text_classification_datasets_4_0_0_fix
Oct 1, 2025
Merged

Fix(requirements): set evaluate>=0.4.5 to resolve GLUE crash with datasets>=4.0.0#2291
regisss merged 1 commit into
huggingface:mainfrom
HabanaAI:main-text_classification_datasets_4_0_0_fix

Conversation

@gplutop7
Copy link
Copy Markdown
Collaborator

@gplutop7 gplutop7 commented Oct 1, 2025

This PR updates the requirements to enforce evaluate>=0.4.5, as older versions of the library crash during metric computation when used together with datasets>=4.0.0.

After upgrading datasets from 3.6.0 to 4.0.0, evaluation on GLUE tasks started failing with evaluate==0.4.3 (and reproducible with 0.4.4). The issue occurs inside the simple_accuracy function of the GLUE metric:
AttributeError: 'bool' object has no attribute 'mean'

The problem was observed when running MRPC evaluation with Habana trainer:
"path": "/root/optimum-habana-fork/examples/text-classification", "command": "PT_HPU_LAZY_MODE=1 python3 run_glue.py --task_name mrpc --learning_rate 3e-5 --max_seq_length 128 --output_dir ./output/mrpc/ --use_hpu_graphs_for_inference --model_name_or_path bert-large-uncased-whole-word-masking --gaudi_config_name Habana/bert-large-uncased-whole-word-masking --do_eval --per_device_eval_batch_size 8 --use_habana --use_lazy_mode --throughput_warmup_steps 3 --sdp_on_bf16 "

Upgrading to evaluate>=0.4.5 resolves the crash and restores correct metric computation.

@gplutop7 gplutop7 requested a review from regisss as a code owner October 1, 2025 11:25
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Copy Markdown
Collaborator

@regisss regisss left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

No need to do additional PRs for other branches like #2292, you can just let me know to cherry-pick the change from main into the target branch 🙂

@regisss regisss merged commit 86f91c0 into huggingface:main Oct 1, 2025
2 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants