[BYOC][TensorRT] Add TensorRT own int8 calibration support to TensorRT BYOC integration

[tiandiao123]

Hi, this commit is used to add calibration and tensorrt int8 mode in the original src/runtime/contrib/tensorrt (thanks for @trevor-m help). Currently, I tested several models including mobilenet_v2 and resnet34, and all of them have greater performance gain. Here is my GitHub gist for testing trt int8 result: https://gist.github.com/tiandiao123/67adb11ab3d73df8e83a1469707d7db4

Firstly, users need to provide calibration data to do calibration, then use the following command line to run your program:

TVM_TENSORRT_USE_INT8=1 TENSORRT_NUM_CALI_INT8=10 python test_trt.py

, where TVM_TENSORRT_USE_INT8 denotes whether to use trt int8, and TENSORRT_NUM_CALI_INT8 denotes how many times we need to do data calibration (we'll do 10-times data calibration in the example).

Here are some testing results (based on T4 machine, and TRT 7):

                                   INT8 TRT                    FP16 TRT
resnet18                    0.38039684295654297 ms             0.5254888534545898 ms
super_resolution            0.2799248695373535 ms              0.44 ms

resnet18 python testing script: https://gist.github.com/tiandiao123/c535b62918677d0a7c45fc005a44f27d
super_resolution testing script: https://gist.github.com/tiandiao123/de52ef96f574645c2dccb3544b291487

Also, in my script, I used a single image for calibrating data for 10-times (just for demonstration). In real project, we should use authors' own data to do calibration.

Also, I used this test file (https://github.com/tiandiao123/tvm/blob/pr_trt_int8/tests/python/contrib/test_tensorrt_int8_exp.py) to test trt int8 precision result with respect to different kinds of resnet. I tested several cosine distances between outputs of the original torch models and the corresponding trt int8 tvm ouputs. I think all of them are under expectation since their similarities are greater than 0.99. The calibration results really depend on which models we choose, and what kind of sample data we choose to do calibration, and how many times do we need to do calibrations. In my experience, the number of calibration times also impact final calibration results. We need to make sure the calibration data is from actual models' training data or evaluation data, otherwise the cosine difference will be impacted. @jcf94

                      cosine distance
resnet34             0.005518794059753418
resnet18             0.005487024784088135
resnet50             0.002280294895172119

I also used the python script above to test densenet161 and mobilenet_v2, and both of them's cosine distance around 0.02~0.03. Since my image used is based on tutorial (https://tvm.apache.org/docs/tutorials/frontend/from_pytorch.html#sphx-glr-tutorials-frontend-from-pytorch-py), and the sample image is actually from resnet evaluation data, probably we need to use actual sample data belonging to densenet161 and mobilenet_v2 to get actual cosine distance, but overall the results are under expectation.

[comaniac]

@tiandiao123 thanks for adding this support. Please add details to the PR description.

@trevor-m it would be great if you could help sharpen this PR :)

[vinx13]

cc @FrozenGene

[Laurawly]

Hi, this commit is used to add calibration and tensorrt int8 mode in the original src/runtime/contrib/tensorrt. Currently, I tested several models including mobilenet_v2 and resnet34, and all of them have greater performance gain. Here is my GitHub gist for testing trt int8 result: https://gist.github.com/tiandiao123/67adb11ab3d73df8e83a1469707d7db4

Firstly, users need to provide calibration data to do calibration, then use the following command line to run your program:

TVM_TENSORRT_USE_INT8=1 TENSORRT_NUM_CALI_INT8=10 python test_trt.py

, where TVM_TENSORRT_USE_INT8 denotes whether to use trt int8, and TENSORRT_NUM_CALI_INT8 denotes how many times do we need to do data calibration (we'll do 10-times data calibration in the example).

Please add some performance numbers for the models you tested.

[FrozenGene]

@jcf94 please help to review it too as you have implemented similar functionality

[jcf94]

Thanks! @tiandiao123 @FrozenGene

It's great to see guys also working on tvm trt, the overall implementation is good, but let's make it better to be accept.

Also a basic UT is needed in tests/python/contrib/test_tensorrt.py. 😄

[jcf94]

Suggest to create calibrator and process the input information here, if there is still some remaining calibration batches, just return.

This can get rid of building an engine then destory it and build one again.

[trevor-m]

We have to build the engine first to know the input binding order that tensorrt assigns to the inputs. It might not match TVM input signature directly.
This is the same strategy used by TF-TRT

[jcf94]

We have to build the engine first to know the input binding order that tensorrt assigns to the inputs. It might not match TVM input signature directly.
This is the same strategy used by TF-TRT

Yeah, I mean we can also just put all of the input information check here, use something like unordered_map<string, void*>, unordered_map<string, size> to create the calibrator and set_batch here. During real inference, these information will be useless.

The trt get_batch function has a parameter const char* names[], that can be used when getting data out. So the order does not matter in this way.

At least in TF-2.4.1, they used:

  // Construct a calibrator for future calibration.
  TRTInt8Calibrator(
      const std::unordered_map<string, std::pair<void*, size_t>>& dev_buffers,
      int batch_size, string engine_name);

  // Feed calibration data to the calibrator, and return true if the data is
  // accepted. Return false if the calibrator has been terminated.
  bool setBatch(const std::unordered_map<string, void*>& data,
                const cudaStream_t stream);

I think that would be a good example for reference.

[Laurawly]

@trevor-m @jcf94 The CI is green now, please give another round of review, thanks!

[jcf94]

Thanks! @tiandiao123 LGTM now.

[Laurawly]

@FrozenGene @trevor-m Could you update your reviews?

[comaniac]

Overall LGTM. Left two nits and I think they can be fixed in a follow-up PR.

[comaniac]

IIUC, use_int8_ is exclusive with use_fp16_? If so, we should combine them to be a single variable like target_dtype.

I dismissed this change request to unblock this PR first as @trevor-m is on PTO. We could fix additional comments in follow-up PRs if needed.

[comaniac]

Thanks @tiandiao123 @trevor-m @FrozenGene @jcf94 @vinx13 @Laurawly