[QUANTIZE] Memorizing the quantize node mapping

[ZihengJiang]

To avoid duplicated simulated quantize.

@tqchen @vinx13 @jwfromm

[ZihengJiang]

Also, let's remove skip_k_conv since it can be expressed by skip_conv_layers, and remove use_stop_fusion since it is mainly for store_lowbit_output. @vinx13 @jwfromm

[vinx13]

Fusion of resnet-50 is broken after this pr.
Previously it is ensured that every subfunction after has int8 output type. This need special handling of residual block

conv - add(bn) - conv
    |---add ------|

If simulated_quantize is not memorized, there will be two simulated_quantize following the first conv2d.
After realize, we have:

conv - i8 - stop_fusion - i32 - add(bn) ... conv
     -i8  - stop_fusion - i32 -----------add |

CSE will merge both i8 and stop_fusion, but not i32 cast:

conv - i8 - stop_fusion - i32 - add(bn) ... conv
                   |-----i32 -------------add |

In this way we ensure that the output is int8.

However, with simulated_quantize memorized, the two i32 cast will be merged because memorization (we call ExprMutator::Mutate with the same simulated_quantize) as a result subfunctions will have i32 output type.
Maybe we can do something with multiref trigger in realize?

[tqchen]

@vinx13 would be great if we can find alternates. Ideally, we want to deal with both cases.
It would also be helpful if you can elaborate it a bit, I did not quite get your example.

[ZihengJiang]

@vinx13 I may not get the point. Does it matter to merge those two i32 cast? The main residual block still will output 8bit (in front of the stop_fusion). Will it influent the accuracy or just performance?

[vinx13]

@ZihengJiang It will impact the performance. Although Stop_fusion can make sure that conv2d + fused ops produce int8 result, if the int32 casts are merged, it will be put into a separate sub function

[tqchen]

@ZihengJiang @vinx13 can you please followup now that #3280 is merged?

[vinx13]

@ZihengJiang please rebase against master