new model: IDEFICS via HuggingFaceM4

[stas00]

**important: The following notes are for my team mates and they won't work for anybody else as the data isn't ready for the public yet. should be made public next week **

Meanwhile to try it out:

$ git clone https://github.com/huggingface/transformers -b add-model-idefics
$ cd transformers

$ cat generate.py
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/idefics-9b"
#checkpoint = "HuggingFaceM4/tiny-random-idefics"

model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint)

prompts = [
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image.",
        
        "Assistant: An image of two kittens in grass.",
        
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image.",
        
        "Assistant:",
    ],
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image.",
        
        "Assistant: An image of a dog wearing funny glasses.",

        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image.",

        "Assistant:",
    ],
]

# batched mode
inputs = processor(prompts, return_tensors="pt").to(device)
# single sample mode
#inputs = processor(prompts[0], return_tensors="pt").to(device)

generated_ids = model.generate(**inputs, max_length=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
for i,t in enumerate(generated_text):
    print(f"{i}:\n{t}\n")

and then run:

CUDA_VISIBLE_DEVICES=0 PYTHONPATH=src python generate.py

Demos

A PR with examples/demos, including finetuning, is here: huggingface/notebooks#418

TODOs before merging

• make the models public - which coincides with the announcement/release

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[flozi00]

Is it possible to be a private repo ? ;-) The m4 repo from huggingface organisation does not exist

[stas00]

Thank you for your interest, @flozi00 - please give us some time. It says WIP because it's not ready for a public consumption. I edited the OP to clarify that.

[stas00]

Thank you, @sgugger, @HugoLaurencon and @leot13 for your reviews - I have addressed everything you have raised.

[sgugger]

Thanks a lot for all your work on this @stas00 !

[amyeroberts]

Because of the ambiguity of how size is handled in torchvision transforms (and reflected in our feature extractors), the image size parameter for image processors is a dictionary size, which contains one of:

• {"height": h, "width": w}
• {"shortest_edge": x}
• {"shortest_edge": x, "longest_edge": y}

e.g. like here for PVT or here for CLIP.

[VictorSanh]

prompts = [
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image."
        
        "Assistant: An image of two kittens in grass.",
        
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image".
        
        "Assistant:",
    ],
    [
        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/dog-puns-1581708208.jpg",
        "Describe this image."
        
        "Assistant: An image of a dog wearing funny glasses.",

        "User:",
        "https://hips.hearstapps.com/hmg-prod/images/cute-photos-of-cats-in-grass-1593184777.jpg",
        "Describe this image".

        "Assistant:",
    ],
]

For posterity, that part of the OP (i can't edit unfortunately) is missing some "," (commas) at some end of string (for instance "Describe this image". -> "Describe this image",). this is important for the tokenization in particular when we call processor with add_end_of_utterance_token=True.

[sgugger]

I can edit if need be. You should also be able to push commits to this branch, since it's in the main fork and you have write permissions @VictorSanh :-)

[gante]

Nit: we have been avoiding using the variable name past in generate-related code, preferring the clearer past_key_values instead.

I've added suggested changes in the related lines :)

[stas00]

Thanks a lot, @gante, for the suggestions - merged

[sgugger]

Thanks again for all the work on this!