Drop QDQ around more nodes - #21376
Conversation
With matching quantization parameters: DequantizeLinear ∘ Flatten ∘ QuantizeLinear is equivalent to just the Flatten, and it saves some floating- point computations. There's already support for a similar optimization for an equivalent Reshape: this change extends the existing optimization to also recognize Flatten. microsoft#21167
Currently, the DropQDQNodesRules optimization removes QuantizeLinear and DequantizeLinear nodes from DequantizeLinear∘MaxPool∘QuantizeLinear. However, if the x_scale/y_scale values are non-positive, this changes the ordering of the elements in the input value, so this optimization is changing the results. This change adds a check for whether the scale in the QuantizeLinear (or DequantizeLinear) is a positive scalar, and a new selector to disallow removing the QDQ around MaxPool if it is not. microsoft#21176
Only does so if the scale is positive
No integer implementations are present, so they need to stay in floating-point. microsoft#21287
Don't expect the drop qdq optimization to work for multiple inputs for now.
Apparently the type constraints for these ops don't include 16-bit integers.
Results don't appear to match
|
Changes are on top of #21182, since it also needs to check for positive scale when dropping QDQ around ReduceMin and ReduceMax |
|
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline |
|
/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline |
|
/azp run Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
Azure Pipelines successfully started running 9 pipeline(s). |
|
Azure Pipelines successfully started running 10 pipeline(s). |
I guess these are no longer lined up anyway after moving some to previous line.
|
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline |
|
/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline |
|
/azp run Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline |
|
Azure Pipelines successfully started running 3 pipeline(s). |
|
Azure Pipelines successfully started running 9 pipeline(s). |
|
Azure Pipelines successfully started running 10 pipeline(s). |
|
I had missed the reason why the windows builds were failing last few commits (currently don't have a Windows system to try locally), sorry I guess since it's I'll try adding the onnxruntime/cmake/onnxruntime_unittests.cmake Line 892 in b9f3a5d |
Seeing: fatal error C1128: number of sections exceeded object file format limit: compile with /bigobj so apparently these additional tests are pushing this file over the limit. Given there's already a statement setting /bigobj for sibling graph_transform_test, simply copy-pasting that for qdq_transformer_test
|
/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows CPU CI Pipeline,Windows GPU CI Pipeline,Windows GPU TensorRT CI Pipeline,ONNX Runtime Web CI Pipeline,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline |
|
/azp run Linux OpenVINO CI Pipeline,Linux QNN CI Pipeline,MacOS CI Pipeline,orttraining-amd-gpu-ci-pipeline,orttraining-linux-ci-pipeline,orttraining-linux-gpu-ci-pipeline,orttraining-ortmodule-distributed,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline |
|
/azp run Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline |
|
Azure Pipelines successfully started running 2 pipeline(s), but failed to run 1 pipeline(s). |
|
Azure Pipelines successfully started running 9 pipeline(s). |
|
Azure Pipelines successfully started running 10 pipeline(s). |
|
/azp run Windows GPU CUDA CI Pipeline,Windows GPU DML CI Pipeline,Windows GPU Doc Gen CI Pipeline |
|
Azure Pipelines successfully started running 3 pipeline(s). |
) ## Summary - Register `Flatten` in `QLinearOpsRegistry` (`Direct8BitOp`) and `QDQRegistry` (`QDQDirect8BitOp`) so the Python QDQ quantizer no longer emits a redundant `DequantizeLinear -> Flatten -> QuantizeLinear` pair around layout-only `Flatten` nodes. - Add `test_op_flatten.py` covering both `QOperator` and `QDQ` formats in u8/u8 and s8/s8. ## Motivation Fixes #21375. The C++ runtime optimizer was updated in #21376 to drop redundant Q/DQ pairs around `Flatten`, mirroring its existing `Reshape` handling. The Python-layer quantizer was never updated, so `quantize_static` / `quantize_dynamic` still emitted the redundant Q/DQ pair around `Flatten` in QDQ format. This PR aligns the Python tooling with the runtime behavior. `Flatten` is a layout-only op with no arithmetic — its quantization semantics are identical to `Reshape`, `Squeeze`, and `Unsqueeze`, all of which are already registered as `Direct8BitOp` / `QDQDirect8BitOp`. ## Changes - `onnxruntime/python/tools/quantization/registry.py`: add `"Flatten": Direct8BitOp` to `QLinearOpsRegistry` and `"Flatten": QDQDirect8BitOp` to `QDQRegistry`. - `onnxruntime/test/python/quantization/test_op_flatten.py`: new test file modeled on `test_op_reshape.py` covering u8/u8 and s8/s8 in both QOperator and QDQ formats; asserts no extra `QuantizeLinear`/`DequantizeLinear` is inserted around `Flatten`, and verifies numerical correctness via `check_model_correctness`. ## Test Plan - `python -m pytest onnxruntime/test/python/quantization/test_op_flatten.py -v` — 2 tests pass. - `python -m pytest onnxruntime/test/python/quantization/test_op_reshape.py -v` — regression: 2 tests pass. - `python -m pytest onnxruntime/test/python/quantization/test_op_squeeze_unsqueeze.py -v` — regression: 2 tests pass. - `lintrunner -a` clean on changed files.
2 participants
Description
Extends the Drop QDQ optimization to remove DequantizeLinear and QuantizeLinear nodes from around operators:
Motivation and Context
To reduce floating-point conversions in quantize inference. Mainly motivated by the Flatten case, since that will show up in graphs exported from PyTorch to ONNX. But to make the change complete, extending to a larger set of ops for which this optimization is valid.
#21375