Add CLIP score

[chinoll]

🚀 Feature

Calculate the correlation between image and text

Motivation

Evaluating the performance of the text2image model

Pitch

pytorch-like Pseudocode

def clip_score(img_inputs, txt_inputs):
    img_features = clip.get_image_features(img_inputs)
    txt_features = clip.get_text_features(txt_inputs)
    img_features, txt_features = [
        x / torch.linalg.norm(x, axis=-1, keepdims=True)
        for x in [img_features, txt_features]
    ]
    return (img_features * txt_features).sum(axis=-1)

Alternatives

Additional context

clip score

[github-actions]

Hi! thanks for your contribution!, great first issue!

[stancld]

Hi @chinoll, this sounds like a nice addition to add the first multi-model metric. Would you have please any reference implementation?

[SkafteNicki]

Here is at least one reference implementation:
https://github.com/mehdidc/DALLE_clip_score

[chinoll]

Hi @chinoll, this sounds like a nice addition to add the first multi-model metric. Would you have please any reference implementation?

simple pytorch implementation,Reference CLIP-score-vs-FID-pareto-curves

from transformers import CLIPModel,CLIPTokenizer,CLIPFeatureExtractor
import torch
import PIL
version = "openai/clip-vit-large-patch14"
tokenizer = CLIPTokenizer.from_pretrained(version)
model = CLIPModel.from_pretrained(version)
feature_extractor = CLIPFeatureExtractor.from_pretrained(version)

def clip_score(text:str, image:PIL.Image):
    txt_features = model.get_text_features(tokenizer(text,return_tensors="pt")["input_ids"])
    img_features = model.get_image_features(torch.tensor(feature_extractor(image)['pixel_values'][0][None]))
    img_features, txt_features = [
        x / torch.linalg.norm(x, axis=-1, keepdims=True)
        for x in [img_features, txt_features]
    ]
    return (img_features * txt_features).sum(axis=-1)

[SkafteNicki]

I started the work of adding the metric in #1314