adding st-lpips

[abhijay9]

Shift-tolerant Perceptual Similarity Metric

We investigated a broad range of neural network elements and developed a robust perceptual similarity metric. Our shift-tolerant perceptual similarity metric (ST-LPIPS) aligns well with human perception and is less susceptible to subtle misalignments between two images compared to the metrics currently available.

If you find this metric useful for your research, please use the following to cite our work.

@inproceedings{ghildyal2022stlpips,
  title={Shift-tolerant Perceptual Similarity Metric},
  author={Abhijay Ghildyal and Feng Liu},
  booktitle={European Conference on Computer Vision},
  year={2022}
}

Test

You can use the following to test the alex_shift_tolerant and vgg_shift_tolerants models.

import numpy as np
from PIL import Image
import pyiqa
import torch

device = torch.device("cuda") if torch.cuda.is_available() else torch.device("cpu")

def load_image(path):
    if(path[-3:] == 'dng'):
        import rawpy
        with rawpy.imread(path) as raw:
            img = raw.postprocess()
    elif(path[-3:]=='bmp' or path[-3:]=='jpg' or path[-3:]=='png'):
        import cv2
        return cv2.imread(path)[:,:,::-1]
    else:
        img = (255*plt.imread(path)[:,:,:3]).astype('uint8')
    return img

def tensor2im(image_tensor, imtype=np.uint8, cent=1., factor=255./2.):
    image_numpy = image_tensor[0].cpu().float().numpy()
    image_numpy = (np.transpose(image_numpy, (1, 2, 0)) + cent) * factor
    return image_numpy.astype(imtype)

def im2tensor(image, imtype=np.uint8, cent=1., factor=255./2.):
# def im2tensor(image, imtype=np.uint8, cent=1., factor=1.):
    return torch.Tensor((image / factor - cent)
                        [:, :, :, np.newaxis].transpose((3, 2, 0, 1)))

# The images are available here: 
# https://github.com/abhijay9/ShiftTolerant-LPIPS/tree/main/imgs

path0 = "../clones/ShiftTolerant-LPIPS/imgs/ex_p0.png"
img0 = im2tensor(load_image(path0))

path1 = "../clones/ShiftTolerant-LPIPS/imgs/ex_ref.png"
img1 = im2tensor(load_image(path1))

# create metric with default setting
iqa_metric = pyiqa.create_metric('stlpips', device=device)
print(iqa_metric(img0,img1).item())
# 0.7777529954910278

iqa_metric = pyiqa.create_metric('stlpips-vgg', device=device)
print(iqa_metric(img0,img1).item())
# 0.651697039604187

Contact

If you come across any issues with the code, please don't hesitate to contact me via email. I'm also interested in opportunities for collaboration. Here is a link to my GitHub profile: Abhijay Ghildyal

[chaofengc]

Thanks for your contribution! The codes are well written with great quality.
It would be better if you could make the corresponding improvements for easy usage.

[abhijay9]

Thank you for your quick response. The models will now be automatically downloaded from my repo and work just as any other metric. Additionally, to make it compatible with IQA-PyTorch, I used the same environment as yours for testing.

Please let me know if there are any further modifications you would like me to make.

[chaofengc]

Thanks for your quick modifications. The codes are already quite good.

From your examples, I notice that the input range is [-1, 1]. To make a user friendly api, it might be better to use a unified [0, 1], RGB input tensor. This can be easily achieved by chaning the normalize option to True by default.

[abhijay9]

Done. Set normalize option to true.

[chaofengc]

Thank you so much!!!