About extracting annotated and candidates fc7-layer features

[z-kun]

Hi jpeyre,

According to your paper, intuitively, we can extract fc7 features of all regions of interest (ROIs) in fast rcnn and regard them as the candidates features.
But, if we want to extract features of annotated bounding boxes, we can not use the NET with the same parameters as which we use to extract the candidates feature ,because it will generate ROIs in pool5 layer before fc7 layer if annotated bounding boxes are inputted to fast rcnn.
So how to extract ANNOTATED FEATURE in fc7 layer of fast rcnn?

I am looking forward to your reply. Thanks!

[jpeyre]

Hi z-kun,

In Fast-RCNN you would have to provide the region of interests yourself, as there is no RPN module like in Faster-RCNN to generate the regions of interest. For example, in this project, we use selective search to generate the candidate regions. To extract the features of annotated bounding boxes, instead of using the candidate regions provided by selective search, you can provide as input the annotated bounding boxes. Thus, you can use exactly the same network to extract the appearance features.

I hope I got your question right !

[z-kun]

@jpeyre
Thanks for your reply.

Indeed, I have extracted the candidate features by faster rcnn with RPN.
Is there any method to extract annotated fc7 features with the same network (faster rcnn)? I think the main problem is pool5(roi pooling) layer which also exists in fast rcnn.

[jpeyre]

Ah I see your problem. I am not aware of a simple way to do this... I think you would have to manually modify Faster-RCNN code so that it extract features for your own proposals (the annotated boxes) instead of using the ROIs from RPN. In your Faster-RCNN implementation, check how the rois are given to roi_pooling layer and see whether you can modify the code to give your own proposals as done in Fast-RCNN.

[z-kun]

@jpeyre
It seems that extracting annotated features in Faster-RCNN is more difficult than Fast-RCNN. I will try to analyze the input structure of roipooling layer.
Thanks anyway.

[z-kun]

@jpeyre
And I call to mind another question.
when we extract the candidate features, we use the fc7 features whose corresponding bounding boxes are not processed by Bbox regression at the end of Fast-RCNN, thus, the bounding box may be just a part of one object.
But when we extract the annotated features, we use the bounding boxes after Bbox regression.
I wonder if this difference between annotated and candidates features will affect the relationship detection.

[jpeyre]

Well I haven't studied this question, but intuitively I would say that the feature extracted "on the fly" of detection, and a feature re-extracted for the same box but after box regression should be quite similar... but once again, I am not a specialist in this question :) Anyway, it's something you could definitely try !