Fix slow tests for important models to be compatible with A10 runners #29905

ydshieh · 2024-03-27T13:35:49Z

What does this PR do?

This PR is the same as #29805 with some final changes. #29805 is closed, and we will merge this one without waiting @younesbelkada being back to the office.

I rebased the PR on a more recent main.

No need to go over again I think. Just check this comment (if you think necessary)

ydshieh · 2024-03-27T14:31:19Z

tests/models/gemma/test_modeling_gemma.py

    @require_flash_attn
    @require_torch_gpu
    @pytest.mark.flash_attn_test
+    @is_flaky


For the comment here, my response is

I will open an issue but leave @younesbelkada to fill more details on the issue page.

ydshieh · 2024-03-27T14:32:59Z

tests/models/gemma/test_modeling_gemma.py

+            # 8 is for A100 / A10 and 7 for T4
+            cls.cuda_compute_capability_major_version = torch.cuda.get_device_capability()[0]
+
+    @require_read_token


This should be a better name, cc @younesbelkada

See #29805 (comment)

ydshieh · 2024-03-27T14:36:21Z

Although it is newly opened (despite with tiny changes), formally let me get a ✅ approval before merge.

here are the changes: mostly, they are just some name changes

1
2
3

HuggingFaceDocBuilderDev · 2024-03-27T14:41:12Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ArthurZucker

Thanks both 🚀

ArthurZucker · 2024-03-28T13:26:40Z

tests/models/mistral/test_modeling_mistral.py

-            "mistralai/Mistral-7B-v0.1",
-            device_map="auto",
-            attn_implementation="sdpa",
+            "mistralai/Mistral-7B-v0.1", device_map="auto", attn_implementation="sdpa", torch_dtype=torch.float16


no torch_device here?

I think it is because we set "auto" and in this case, no need to set torch_device.

torch_device for the recent updates done by a contributor to support multiple devices when testing!

tests/models/mistral/test_modeling_mistral.py

ArthurZucker

Looks alright but let's make sure CIs are green please

tests/models/mistral/test_modeling_mistral.py

ydshieh · 2024-04-03T14:25:13Z

The only failing test is

FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_generate_with_prompt_ids_max_length - IndexError: index -1 is out of bounds for dimension 1 with size 0

which is irrelevant to this PR, and #30018 is on it.

ydshieh · 2024-04-03T16:58:48Z

@younesbelkada There are still some test failures

https://github.com/huggingface/transformers/actions/runs/8542383405/job/23404051565

ArthurZucker

Good for me as long as we fix the remaining tests

younesbelkada · 2024-04-09T09:45:58Z

tests are all green: https://github.com/huggingface/transformers/actions/runs/8613471616/job/23604832895

Co-authored-by: Arthur <[email protected]>

ydshieh · 2024-04-09T11:28:44Z

Thank you @younesbelkada again!

fxmarty · 2024-04-17T10:07:54Z

tests/models/llama/test_modeling_llama.py

    def test_compile_static_cache(self):
        NUM_TOKENS_TO_GENERATE = 40
-        EXPECTED_TEXT_COMPLETION = [
-            "Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.",
-            "My favorite all time favorite condiment is ketchup. I love it on everything. I love it on my eggs, my fries, my chicken, my burgers, my hot dogs, my sandwiches, my salads, my p",
-        ]
+        EXPECTED_TEXT_COMPLETION = {
+            7: [
+                "Simply put, the theory of relativity states that 1) the speed of light is constant, 2) the speed of light is the same for all observers, and 3) the laws of physics are the same for all observers.",


This fails on A100

on the A10 runners with the transformers-all-latest-gpu docker image it passed, might be an env issue?

yes the ref is probably good for A10G, but not A100

ydshieh force-pushed the fix-slow-tests-shieh branch from d8348e6 to 54c6918 Compare March 27, 2024 14:16

ydshieh changed the title ~~Fix slow tests shieh~~ Fix slow tests for important models to be compatible with A10 runners Mar 27, 2024

ydshieh commented Mar 27, 2024

View reviewed changes

ydshieh requested a review from amyeroberts March 27, 2024 14:36

ydshieh requested review from ArthurZucker and removed request for amyeroberts March 28, 2024 09:23

ArthurZucker reviewed Mar 28, 2024

View reviewed changes

ydshieh requested a review from ArthurZucker March 29, 2024 09:46

ArthurZucker reviewed Mar 30, 2024

View reviewed changes

tests/models/mistral/test_modeling_mistral.py Show resolved Hide resolved

ydshieh force-pushed the fix-slow-tests-shieh branch from c684692 to effc252 Compare April 3, 2024 13:20

ydshieh requested a review from ArthurZucker April 3, 2024 14:25

ArthurZucker approved these changes Apr 3, 2024

View reviewed changes

younesbelkada added 12 commits April 9, 2024 12:30

fix mistral and mixtral

e6a2b81

add pdb

3a628af

fix mixtral tesst

c7e970f

fix

66c2e5d

fix mistral ?

361e7cc

add fix gemma

55ccbca

fix mistral

3c0dc50

fix

f9ba37c

test

73dafde

anoter test

22585dd

fix

d60ca74

fix

6983ba9

younesbelkada and others added 22 commits April 9, 2024 12:30

fix them again

22d1b77

final fixes for mistral

4a4a9e7

fix padding right

04213a5

fix whipser fa2

374d23c

fix

79545e8

fix

9c5f47a

fix gemma

58cd487

test

0fd9b02

fix llama

518b6a2

fix

2bd652f

fix

25a299d

fix llama gemma

282a6d3

add class attribute

21a0ddb

fix CI

e2c035e

clarify whisper

b1e3a35

compute_capability

86e962e

rename names in some comments

e208db1

Add # fmt: skip

ce2cfa4

make style

d0c9f7c

Update tests/models/mistral/test_modeling_mistral.py

bd41a3a

Co-authored-by: Arthur <[email protected]>

update

751a62c

update

71debdb

ydshieh force-pushed the fix-slow-tests-shieh branch from 19a6a07 to 71debdb Compare April 9, 2024 10:30

ydshieh merged commit 08a194f into main Apr 9, 2024

ydshieh deleted the fix-slow-tests-shieh branch April 9, 2024 11:28

fxmarty reviewed Apr 17, 2024

View reviewed changes

ydshieh mentioned this pull request Jun 4, 2024

Fix MistralIntegrationTest #31231

Merged

Fix slow tests for important models to be compatible with A10 runners #29905

Fix slow tests for important models to be compatible with A10 runners #29905

Uh oh!

Conversation

ydshieh commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ydshieh commented Mar 27, 2024

Uh oh!

HuggingFaceDocBuilderDev commented Mar 27, 2024

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ydshieh commented Apr 3, 2024

Uh oh!

ydshieh commented Apr 3, 2024

Uh oh!

ArthurZucker left a comment

Choose a reason for hiding this comment

Uh oh!

younesbelkada commented Apr 9, 2024

Uh oh!

ydshieh commented Apr 9, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

ydshieh commented Mar 27, 2024 •

edited

Loading