fix condition where torch_dtype auto collides with model_kwargs.

[Vaibhavs10]

What does this PR do?

Fixes broken tests on nightly CI

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ydshieh]

we have some progress

FAILED tests/pipelines/test_pipelines_text_generation.py::TextGenerationPipelineTests::test_small_model_pt_bloom_accelerate - AssertionError: torch.bfloat16 != torch.float32

and for this

FAILED tests/pipelines/test_pipelines_image_text_to_text.py::ImageTextToTextPipelineTests::test_small_model_pt_token - AssertionError: Lists differ: [{'in[168 chars] are facing the camera, and they'}, {'input_te[192 chars]ey'}] != [{'in[168 chars] are sleeping and appear to be comfortable'}, [212 chars]le'}]

does the dtype used changed in your previous PR? maybe I can simply update the expected output. This is probably minor issue, don't stree

[ydshieh]

if you need a more detailed log, let me know

[Vaibhavs10]

yes this is expected:

we have some progress

FAILED tests/pipelines/test_pipelines_text_generation.py::TextGenerationPipelineTests::test_small_model_pt_bloom_accelerate - AssertionError: torch.bfloat16 != torch.float32

since we default to torch.bfloat16 and not float32 in the pipeline.

for the second one I think we can just update the output

[ydshieh]

great and thanks a lot!