BLIP2 models would stop working with next release of tranformers (4.27.0)

[AstraliteHeart]

I've been trying to use transformers to convert BLIP2 models into compatible format (https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/convert_blip_2_original_to_pytorch.py) and ran into some issues, specifically this commit huggingface/transformers#21405 breaks compatibility between lavis and transformers.

So while loading models via transformers (i.e. https://huggingface.co/docs/transformers/main/model_doc/blip-2#transformers.Blip2ForConditionalGeneration.forward.example) will continue to work, lavis will need to be updated for compatibility with next release.

Reference issue: huggingface/transformers#21713

[LiJunnan1992]

Thanks for the heads up! @dxli94 BLIP-2 OPT models will indeed not work with the next transformer release. Shall we constrain the transformer version until I can fix this issue?

[dxli94]

@AstraliteHeart Thanks for the information.

@LiJunnan1992 , I've now pinned the transformers version, also publish a pre-release on pypi. After we fix, we'll publish a new release, with the changes made recently according to the community feedback.

[LiJunnan1992]

We have made an update to BLIP-2 OPT models so that they can work with the latest transformers with version>=4.27.