Reduce the inference cost of Faster R-CNN with FPN

[wkcn]

Hi, there.

In the implementation of Faster R-CNN with FPN, it takes much time in RoIPooling.
The reason is that RoIPooling computes the feature of duplicated proposals.
https://github.com/dmlc/gluon-cv/blob/master/gluoncv/model_zoo/faster_rcnn/faster_rcnn.py#L298

There are some procedures to reduce the inference cost.

1. Modify the implementation of RoIPooling and RoIAlign
   https://github.com/apache/incubator-mxnet/blob/master/src/operator/roi_pooling.cc#L75
   For invalid indexes, we should ignore its output and gradient.

2. Change some indexes of rpn_rois to -1 when they are not in the specific roi_level
   RoIPooling only computes the feature of proposals, which are in the specific roi level.

3. Use assigment rather than sum to aggregate pooled features
   pooled_roi_feats = F.ElementWiseSum(*pooled_roi_feats) need extra addition to aggregate features.

[zhreshold]

@Jerryzcn is working on speeding up that.

[Jerryzcn]

The main reason FPN is slower is actually the num of RCNN samples. FPN network uses 512, whereas non-FPN version uses 128. That being said, I have modified ROIAlign, the gain is not significant. We are working on other aspects of the model to speedup the training. We are hoping to train Mask-RCNN in 3.5 hours w/ 12 epochs schedule.

[Jerryzcn]

Regarding 3., the issue is that the number of rois for each level is dynamic, so it is tricky to use assignment.

[Jerryzcn]

The change for ROIAlign is in
apache/mxnet#16215

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.