Get the confidence of the segmented class during test tme

[poornimajd]

Hello,
I am trying to figure out if there is a way to get the shelfnet network also give the confidence of the segmented class during test time.Basically give how accurate the network thinks the segmentation output is.
Any suggestion is greatly appreciated.
Thank you

[poornimajd]

@juntang-zhuang here is a small update on the above issue.
I figured out that in the evaluate.py script,the following line gives the probability of each pixel belonging to a particular class.
https://github.com/juntang-zhuang/ShelfNet/blob/citys_lw/ShelfNet18_realtime/evaluate.py#L63
But the probability which is returned in the following function
https://github.com/juntang-zhuang/ShelfNet/blob/citys_lw/ShelfNet18_realtime/evaluate.py#L99
is a scaled probability,having values greater then 1,even though I passed the scaling factor as 1 in this function.
I had the following questions-

1. What is the difference between scaled probability and the probability returned by the first case.
2. To find the confidence of the segmented class,I am just taking the mean of the probability of the pixels belonging to a particular class ,returned by the first case(i.e no scaling).Is this a correct way of finding the confidence of the segmented class during test time(i.e no ground truth is present).
3. Since the scaled probability is greater than 1 for some pixels,how can we use this to find the confidence of the segmented region?
   Any suggestion would be really helpful.
   Thankyou

[juntang-zhuang]

@poornimajd Hi, sorry for the late reply, quite busy recently. I suspect the scaled prob occurs when reshape the predicted logits map to the same size as the actual image size, so it depends on the interpolation method it might output probability larger than 1. A walk-around might be to interpolate the logits rather than the probability, then perform softmax on resized feature map to get probability.
I don't think it's suitable to use the result here as a confidence. I came across several papers claiming that neural networks are over-confident, therefore the confidence score here is not really meaningful. Some works related to "bayesian neural networks" and "uncertainty estimation" might be more appropriate when you want to get a statistically meaningful confidence score.

[poornimajd]

Thank you for the reply.

A walk-around might be to interpolate the logits rather than the probability, then perform softmax on resized feature map to get probability.

I tried to interpolate the 'out' from this line to [720,1280] as this is my test image shape:

ShelfNet/ShelfNet18_realtime/evaluate.py

Line 55 in ea7b9f4

out = self.net(crop)[0]

This is then given to the softmax function.But in the crop_eval function,I get an error in this line:

ShelfNet/ShelfNet18_realtime/evaluate.py

Line 93 in ea7b9f4

prob[:, :, hst:hed, wst:wed] += prob_chip

The reason being prob and prob_chip do not have same shape.
Is this something to change with the dataloader? But I have already trained the model with the given dataloader.

I am just thinking if the following works-
just take mean of the probability of the pixels belonging to a particular class without any interpolation i.e from here:
https://github.com/juntang-zhuang/ShelfNet/blob/citys_lw/ShelfNet18_realtime/evaluate.py#L63
which is for the height=width=1024,and use it as a approximate confidence score for the input test image - height=720,width-1280.Since it is a mean,the interpolation will not affect the score drastically.
Will this be a fair assumption?

I came across several papers claiming that neural networks are over-confident, therefore the confidence score here is not really meaningful. Some works related to "bayesian neural networks" and "uncertainty estimation" might be more appropriate when you want to get a statistically meaningful confidence score.

Yes,I agree,but given the time constraints,I wanted to know if I can do something with the existing sources itself.
Any suggestion is appreciated.
Thank you

[juntang-zhuang]

Hi, sorry for the late reply. I'm not so sure about the shape mismatch, it has been too long since this work and I don't remember the details, I remember the network is predicting patch by patch (perhaps for each patch it predicts at different scales, then scale it back to the original size, need to check the code), and there are even overlaps between patches to get better smoothness. This might be the cause of error.

Taking the mean of probability might be fine, again I'm not sure, I would use Bayesian methods for uncertainty, rather than on the output probability. Also note that if we care about accuracy, not uncertainty, it doesn't matter whether it's normalized probability, or logits of probability; all we need to do is to find the max index (corresponding to a class) for each pixel, it may not even be a meaningful probability if we only care about the correct max index.