Support VLM in chat completion (+some specs updates)

[Wauplin]

Follow-up PR after huggingface/huggingface.js#915.

Updates chat completion input parameters (tools and stream_options) + add supports for VLMs => sending image as text message.

Also adds TextToSpeech spec definition but not used at the moment (same specs as TextToAudio).

Also added an example in docs:

from huggingface_hub import InferenceClient

# works with remote url or base64 encoded url
image_url ="https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"

client = InferenceClient("mistralai/Mistral-Nemo-Instruct-2407")
output = client.chat.completions.create(
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {"url": image_url},
                },
                {
                    "type": "text",
                    "text": "Describe this image in one sentence.",
                },
            ],
        },
    ],
)

print(output.choices[0].message.content)
#A determine figure of Lady Liberty stands tall, holding a torch aloft, atop a pedestal on an island.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[MoritzLaurer]

Cool! One great addition would be to also explicitly document how people can use a local, non-public image in PIL and convert it to a base64 string which the API also understands. People often get confused about this. I use code similar to this:

import base64
from io import BytesIO
from PIL import Image

# Open an image using PIL
image = Image.open('path_to_your_image.png')

def image_to_data_url(image):
    # Use the image's format or default to 'PNG' if format is None
    format = image.format or 'PNG'
    with BytesIO() as buffer:
        image.save(buffer, format=format)
        img_str = base64.b64encode(buffer.getvalue()).decode('utf-8')
    return f"data:image/{format.lower()};base64,{img_str}"

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "some prompt"
            },
            {
                "type": "image_url",
                "image_url": {
                    "url": image_to_data_url(image)
                },
            }
        ]
    }
]

[Wauplin]

I'll see how I can integrate that in the docs 👍 I don't want the snippet to be bloated with 3 different use cases. For now I have:

>>> image_path = "/path/to/image.jpeg"
>>> with open(image_path, "rb") as f:
...     base64_image = base64.b64encode(f.read()).decode("utf-8")
>>> image_url = f"data:image/jpeg;base64,{base64_image}"

I'm thinking that in addition to the method docstring we should maybe add a section in the Inference guide typically to explain better how to deal with system prompt, tools, images, etc.

[hanouticelina]

All good for me! thanks @Wauplin

[LysandreJik]

Thank you @Wauplin

[osanseviero]

Nice!

[osanseviero]

should this be pixtral? nemo instruct is text-only

[Wauplin]

Definitely an oversight from me. It seems that when the model is an LLM (not VLM), the data is still transferred to as text. In my case the URL contains "statue of liberty" so the LLM has been able to describe it (and tricked me into thinking the image has been used).

[osanseviero]

LLM: 1
Humans: 0

[hanouticelina]

I just realized that client.audio_classification() current implementation can take also a local audio file, or a URL pointing to an audio file as input (thanks to the _open_as_binary() context manager). We also handle base64 encoding in the code. This is actually the case for other tasks that take image or audio as input.

While it's quite convenient for users to not worry about file handling or encoding, it makes the documentation of the library not aligned with the current API specs for the inputs field.

[hanouticelina]

@Wauplin as discussed, this will be addressed in #2561
LGTM :)

[Wauplin]

Thanks for the reviews! I've updated the example to use "HuggingFaceM4/idefics2-8b-chatty" for now.