Skip to content

Conversation

@ydshieh
Copy link
Collaborator

@ydshieh ydshieh commented Jan 10, 2025

What does this PR do?

test_beam_search_low_memory is very flaky for Emu3Vision2TextModelTest.

The 3 fixes in this PR are all necessary to have 0 failure in a suite of 100 runs.

  • call set_xxx_less_flaky methods
  • avoid generate image tokens
  • compare with _check_similar_generate_outputs

The final version run 500 times without failure.

@ydshieh ydshieh marked this pull request as draft January 10, 2025 15:27
@ydshieh ydshieh marked this pull request as ready for review January 10, 2025 15:49
@ydshieh ydshieh requested a review from zucchini-nlp January 10, 2025 15:50
)
self.assertListEqual(low_output.tolist(), high_output.tolist())
# The two outputs must match and their shape must be as expected
self._check_similar_generate_outputs(low_output, high_output)
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is still required, otherwise 1% of chance to fail for Emu3Vision2TextModelTest

@HuggingFaceDocBuilderDev

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Copy link
Member

@zucchini-nlp zucchini-nlp left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

@ydshieh
Copy link
Collaborator Author

ydshieh commented Jan 10, 2025

tests/models/mobilenet_v1/test_modeling_mobilenet_v1.py::MobileNetV1ModelTest::test_batching_equivalence - AssertionError: tensor(False) is not true : Batched and Single row outputs are not equal in MobileNetV1ForImageClassification for key=logits. Difference=0.48479270935058594.

This is a bit nasty but irrelevant to this PR. Will merge and try to fix the above one next week.

@ydshieh ydshieh merged commit 04eae98 into main Jan 10, 2025
24 of 26 checks passed
@ydshieh ydshieh deleted the fix_flaky_test_beam_search_low_memory branch January 10, 2025 16:31
@zucchini-nlp
Copy link
Member

@ydshieh Cool, feel free to close #29516 after fixing that. We had a small tracker for flaky vision model with that test

@ydshieh
Copy link
Collaborator Author

ydshieh commented Jan 10, 2025

Thanks. The tracker is more for batching tests, but I will check with #35564 next week

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants