Performance Regression from #12019

[pengzhao-intel]

The performance on CPU dropped about 30-50% for the inference in resnet151 and other networks till
#12019

Please reproduce the issue by benchmark_score.py.

Below data shows the fps changes, such as first row from 33.89 drops to 19.91, about -41%.

@azai91 @zheng-da @eric-haibin-lin

647559 resnet-152#3#224#224#1 647559
COMPLETED 1516 s RESNET-152 1 FORWARD (images/sec) 19.91 ( -41.25% ) 33.89
BACKWARD (images/sec) 2.09 ( 0.26% ) 2.08
TOTAL (images/sec) 1.89 ( -3.67% ) 1.96
647559 resnet-152#3#224#224#2 647559
COMPLETED 1516 s RESNET-152 2 FORWARD (images/sec) 30.03 ( -34.54% ) 45.87
BACKWARD (images/sec) 4.09 ( 3.72% ) 3.94
TOTAL (images/sec) 3.60 ( -0.87% ) 3.63
647559 resnet-152#3#224#224#4 647559
COMPLETED 1516 s RESNET-152 4 FORWARD (images/sec) 42.07 ( -28.76% ) 59.05
BACKWARD (images/sec) 6.71 ( -8.32% ) 7.32
TOTAL (images/sec) 5.79 ( -11.13% ) 6.52
647559 resnet-152#3#224#224#8 647559
COMPLETED 1516 s RESNET-152 8 FORWARD (images/sec) 50.32 ( -25.80% ) 67.82
BACKWARD (images/sec) 12.81 ( -6.65% ) 13.72
TOTAL (images/sec) 10.21 ( -10.54% ) 11.41
647559 resnet-152#3#224#224#16 647559
COMPLETED 1516 s RESNET-152 16 FORWARD (images/sec) 50.81 ( -33.99% ) 76.98
BACKWARD (images/sec) 24.34 ( 10.02% ) 22.13
TOTAL (images/sec) 16.46 ( -4.24% ) 17.19
647559 resnet-152#3#224#224#32 647559
COMPLETED 1516 s RESNET-152 32 FORWARD (images/sec) 51.26 ( -35.19% ) 79.10
BACKWARD (images/sec) 36.72 ( 10.40% ) 33.26
TOTAL (images/sec) 21.39 ( -8.63% ) 23.41
647559 resnet-152#3#224#224#64 647559
COMPLETED 1516 s RESNET-152 64 FORWARD (images/sec) 50.46 ( -36.77% ) 79.81
BACKWARD (images/sec) 56.14 ( 16.53% ) 48.18
TOTAL (images/sec) 26.57 ( -11.54% ) 30.04
647559 resnet-152#3#224#224#128 647559
COMPLETED 1516 s RESNET-152 128 FORWARD (images/sec) 51.30 ( -34.70% ) 78.57
BACKWARD (images/sec) 73.81 ( 10.29% ) 66.92
TOTAL (images/sec) 30.27 ( -16.25% ) 36.14
647559 resnet-152#3#224#224#256 647559
COMPLETED 1516 s RESNET-152 256 FORWARD (images/sec) 51.44 ( -33.42% ) 77.27
BACKWARD (images/sec) 80.00 ( 1.38% ) 78.92

[vandanavk]

@mxnet-label-bot [Performance]

[wikier]

Have you monitored the memory? I think we've run into a similar issue because inputs caching (#12746). That may be related with the performance regression.

[pengzhao-intel]

@wikier Thanks for the information.
Do you have an example? I'd like to see the problem and figure out a long-term solution.

[pengzhao-intel]

fix is merged so close.