Skip to content

fix(flaky): test_generate_with_and_without_position_ids in GLM ORC#44173

Merged
tarekziade merged 8 commits intomainfrom
tarekziade-fix-glm-ocr-flakiness
Feb 20, 2026
Merged

fix(flaky): test_generate_with_and_without_position_ids in GLM ORC#44173
tarekziade merged 8 commits intomainfrom
tarekziade-fix-glm-ocr-flakiness

Conversation

@tarekziade
Copy link
Copy Markdown
Collaborator

@tarekziade tarekziade commented Feb 20, 2026 •
edited
Loading

Fixes flaky GLM OCR generation behavior when 2D position_ids are passed explicitly.

Reproducible locally with:

pytest tests/models/glm_ocr/test_modeling_glm_ocr.py::GlmOcrModelTest::test_generate_with_and_without_position_ids --flake-finder --flake-runs=500

Fix

We skip 2D from GenerationTesterMixin.test_generate_with_and_without_position_ids when model uses get_rope_index

15 models impacted

  1. glm_ocr
  2. glm4v
  3. glm46v,
  4. glm_image
  5. qwen2_vl
  6. qwen2_5_vl
  7. qwen2_5_omni
  8. qwen3_vl
  9. qwen3_vl_moe
  10. qwen3_5
  11. qwen3_5_moe
  12. qwen3_omni_moe
  13. paddleocr_vl
  14. ernie4_5_vl_moe
  15. glm4v_moe

@tarekziade tarekziade self-assigned this Feb 20, 2026
@tarekziade tarekziade marked this pull request as draft February 20, 2026 09:31
@HuggingFaceDocBuilderDev
Copy link
Copy Markdown

HuggingFaceDocBuilderDev commented Feb 20, 2026

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@tarekziade tarekziade marked this pull request as ready for review February 20, 2026 10:00
@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

run-slow: glm_ocr

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/glm_ocr"]
quantizations: []

@tarekziade tarekziade mentioned this pull request Feb 20, 2026
Merged
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 969af025 workflow commit (merge commit)
PR 7d35dffb branch commit (from PR)
main 00cc937c base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

zucchini-nlp
zucchini-nlp reviewed Feb 20, 2026
View reviewed changes
Comment thread src/transformers/models/glm_ocr/modeling_glm_ocr.py Outdated
@tarekziade
fix(flaky): test_generate_with_and_without_position_ids in GLM ORC
df10d4c 
Changes:

1. src/transformers/models/glm_ocr/modeling_glm_ocr.py:1431

- In prepare_inputs_for_generation, normalize user-provided 2D position_ids
  into packed multimodal format.
- On cache continuation, only reuse rope_deltas when cache length is non-zero
  (fixes stale-delta reuse across sequential generate() calls).

2. tests/models/glm_ocr/test_modeling_glm_ocr.py:166

- Avoid random accidental pad tokens in synthetic input IDs by remapping
  pad_token_id in random portions (0 -> 1), reducing test instability from
  pad-derived masks.
@tarekziade
moved the fix to modular
af2ea60
@tarekziade
simplify code
875c8a0
@tarekziade
dont tweak position ids here
cfc1321
@tarekziade tarekziade marked this pull request as draft February 20, 2026 12:57
@tarekziade tarekziade marked this pull request as draft February 20, 2026 12:57
@tarekziade tarekziade force-pushed the tarekziade-fix-glm-ocr-flakiness branch from 7d35dff to 893ab53 Compare February 20, 2026 13:15
@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

[For maintainers] Suggested jobs to run (before merge)

run-slow: glm_ocr

@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

run-slow: glm_ocr

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/glm_ocr"]
quantizations: []

zucchini-nlp
zucchini-nlp reviewed Feb 20, 2026
View reviewed changes
Comment thread src/transformers/models/glm_ocr/modeling_glm_ocr.py Outdated
@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

run-slow: glm_ocr now, glm4v, glm46v, glm_image, qwen2_vl, qwen2_5_vl, qwen2_5_omni, qwen3_vl, qwen3_vl_moe, qwen3_5, qwen3_5_moe, qwen3_omni_moe, paddleocr_vl, ernie4_5_vl_moe, glm4v_moe

@tarekziade tarekziade marked this pull request as ready for review February 20, 2026 13:55
@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

run-slow: glm_ocr, glm4v, glm46v, glm_image, qwen2_vl, qwen2_5_vl, qwen2_5_omni, qwen3_vl, qwen3_vl_moe, qwen3_5, qwen3_5_moe, qwen3_omni_moe, paddleocr_vl, ernie4_5_vl_moe, glm4v_moe

@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

note that the idefics failure is yet another different flake I will fix in a separate branch

@tarekziade
Copy link
Copy Markdown
Collaborator Author

tarekziade commented Feb 20, 2026

run-slow: glm_ocr, glm4v, glm46v

zucchini-nlp
zucchini-nlp approved these changes Feb 20, 2026
View reviewed changes
Copy link
Copy Markdown
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread tests/generation/test_utils.py
Comment on lines +2279 to +2280
if has_3d_rope_positions:
continue
Copy link
Copy Markdown
Member

@zucchini-nlp zucchini-nlp Feb 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: we can skip right away because usually all models in the set will be the same arch

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 32d35e03 workflow commit (merge commit)
PR 893ab53a branch commit (from PR)
main 8151000f base commit (on main)

⚠️ No test being reported (jobs are skipped or cancelled)!

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

Workflow Run ⚙️

This comment contains run-slow, running the specified jobs:

models: ["models/glm46v", "models/glm4v", "models/glm_ocr"]
quantizations: []

@github-actions
Copy link
Copy Markdown
Contributor

github-actions bot commented Feb 20, 2026

CI Results

Workflow Run ⚙️

Commit Info

Context Commit Description
RUN 25f42d53 workflow commit (merge commit)
PR 2a4e58e4 branch commit (from PR)
main 32deb4c3 base commit (on main)

✅ No failing test specific to this PR 🎉 👏 !

@tarekziade tarekziade merged commit a3bfc8b into main Feb 20, 2026
26 of 27 checks passed
@tarekziade tarekziade deleted the tarekziade-fix-glm-ocr-flakiness branch February 20, 2026 19:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zucchini-nlp zucchini-nlp zucchini-nlp approved these changes

@Rocketknight1 Rocketknight1 Awaiting requested review from Rocketknight1

Assignees

@tarekziade tarekziade

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tarekziade @HuggingFaceDocBuilderDev @zucchini-nlp