Skip to content

Modify scale & offset of WhereDummyDq #27109

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Edward Chen (edgchen1) merged 6 commits into from
Apr 10, 2026
Merged

Modify scale & offset of WhereDummyDq #27109

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
6f8c58f
Modify scale & offset of WhereDummyDq
qti-hungjuiw Jan 22, 2026
53ff9a0
Use graph_utils::IsConstantInitializer to check constant
qti-hungjuiw Mar 23, 2026
2b1ca7b
Add testcases where DQ and Q have different datatypes
qti-hungjuiw Mar 23, 2026
2e5279b
Check whether the fields are null in where_dummy_dq before optimization
qti-hungjuiw Apr 7, 2026
17d165c
Add testcases to cover multi-consumer & non-quantize consumer scenarios
qti-hungjuiw Apr 7, 2026
473c6af
Simplify condition check and add motivation in the header
qti-hungjuiw Apr 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
190 changes: 132 additions & 58 deletions onnxruntime/core/optimizer/qdq_transformer/where_dummy_dq.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,112 +14,186 @@

namespace onnxruntime {
bool WhereDummyDq::SatisfyCondition(const Graph& graph, const Node& node) const {
// This transformer targets a very specific pattern around `Where` when used in a QDQ graph:
// cond, DQ(xq), const_scalar -> Where -> Q(yq)
// or
// cond, const_scalar, DQ(xq) -> Where -> Q(yq)
//
// When one `Where` branch is a scalar initializer (no producer node), WhereNodeGroupSelector
// requires both data branches to be produced by DQ nodes so the `Where` can be grouped into a
// single node-unit. We insert a "dummy" DQ for the scalar branch to satisfy that requirement.
if (!(node.OpType() == "Where")) {
return false;
}

// ONNX Where inputs: [0]=condition, [1]=X, [2]=Y
const auto& where_inputs = node.InputDefs();
const auto& where_outputs = node.OutputDefs();

const Node* parent_node_1 = graph.GetProducerNode(where_inputs[1]->Name());
const Node* parent_node_2 = graph.GetProducerNode(where_inputs[2]->Name());

bool is_p1_dq = (parent_node_1 && parent_node_1->OpType() == QDQ::DQOpName);
bool is_p2_dq = (parent_node_2 && parent_node_2->OpType() == QDQ::DQOpName);
// Only apply when the `Where` output is immediately consumed by a single QuantizeLinear.
// If there are multiple consumers (or not a Q), inserting an extra DQ would not help form a
// clean QDQ node-unit and may create additional overhead.
std::vector<const Node*> child_nodes = graph.GetConsumerNodes(where_outputs[0]->Name());
if (child_nodes.size() != 1 || child_nodes[0]->OpType() != QDQ::QOpName) {
return false;
}
Comment thread
qti-hungjuiw marked this conversation as resolved.

// WhereDummyDq focus on WhereOp with one DQ input and one scalar initializer input
if (is_p1_dq && !parent_node_2) {
return (where_inputs[2]->Shape()->dim_size() == 0);
const bool is_p1_dq = (parent_node_1 && parent_node_1->OpType() == QDQ::DQOpName);
const bool is_p2_dq = (parent_node_2 && parent_node_2->OpType() == QDQ::DQOpName);

// We require exactly one branch to be fed by a DQ and the other branch to be a scalar initializer
// (represented as a NodeArg with rank 0 shape and no producer node).
Comment thread
qti-hungjuiw marked this conversation as resolved.
if (is_p1_dq && graph_utils::IsConstantInitializer(graph, where_inputs[2]->Name(), true)) {
return where_inputs[2]->HasTensorOrScalarShape() ? (where_inputs[2]->Shape()->dim_size() == 0) : false;
}
if (!parent_node_1 && is_p2_dq) {
return (where_inputs[1]->Shape()->dim_size() == 0);
if (graph_utils::IsConstantInitializer(graph, where_inputs[1]->Name(), true) && is_p2_dq) {
return where_inputs[1]->HasTensorOrScalarShape() ? (where_inputs[1]->Shape()->dim_size() == 0) : false;
}

return false;
}

Status WhereDummyDq::InsertDummyDQ(Node& node, Graph& graph, bool& modified, const logging::Logger& logger) const {
// Inserts a DeQuantizeLinear node on the scalar initializer branch of `Where` so that both
// data branches (X and Y) are produced by DQ nodes, enabling downstream QDQ grouping.
const auto& where_inputs = node.InputDefs();
const auto& where_outputs = node.OutputDefs();
const Node* parent_node_1 = graph.GetProducerNode(where_inputs[1]->Name());
const Node* parent_node_2 = graph.GetProducerNode(where_inputs[2]->Name());
const Node* child_node = graph.GetConsumerNodes(where_outputs[0]->Name())[0];

// With SatisfyCondition, we must have one DQ and one initializer
// From SatisfyCondition():
// - exactly one of parent_node_1/parent_node_2 is a DQ node
// - the other input is a scalar initializer (rank-0 tensor) with no producer node
const Node* dq_node = parent_node_1 ? parent_node_1 : parent_node_2;
int const_idx = parent_node_1 ? 2 : 1;
const int const_idx = parent_node_1 ? 2 : 1;

// Guardrail: only insert dummy DQ when the quantized dtype matches the output Q's dtype.
// If they differ, we cannot safely synthesize quantization parameters.
const int32_t dt_input = dq_node->InputDefs()[0]->TypeAsProto()->tensor_type().elem_type();
const int32_t dt_output = child_node->OutputDefs()[0]->TypeAsProto()->tensor_type().elem_type();
if (dt_input != dt_output) {
Comment thread
qti-hungjuiw marked this conversation as resolved.
LOGS(logger, WARNING) << "WhereDummyDq: skip inserting dummy DQ due to mismatched quantized dtype between input DQ "
"and output Q. DQ input dtype="
<< dt_input << ", Q output dtype=" << dt_output;
return Status::OK();
}

const ONNX_NAMESPACE::TensorProto* dq_node_scale_proto = nullptr;
graph.GetInitializedTensor(dq_node->InputDefs()[1]->Name(), dq_node_scale_proto);
if (!graph.GetInitializedTensor(dq_node->InputDefs()[1]->Name(), dq_node_scale_proto) ||
dq_node_scale_proto == nullptr) {
LOGS(logger, WARNING) << "WhereDummyDq expects dq branch to have an initializer scale. "
<< "DQ: " << dq_node->Name();
return Status::OK();
};

const ONNX_NAMESPACE::TensorProto* dq_node_zp_proto = nullptr;
graph.GetInitializedTensor(dq_node->InputDefs()[2]->Name(), dq_node_zp_proto);
if (!graph.GetInitializedTensor(dq_node->InputDefs()[2]->Name(), dq_node_zp_proto) ||
dq_node_zp_proto == nullptr) {
LOGS(logger, WARNING) << "WhereDummyDq expects dq branch to have an initializer zero point. "
<< "DQ: " << dq_node->Name();
return Status::OK();
};

// Create initializers for the dummy DQ input triplet: (xq, scale, zero_point).
// We choose values so that DeQuantizeLinear(dummy_xq, dummy_scale, dummy_zp) reconstructs
// the original scalar float value as closely as possible.
//
// Note: We only support float scalar constants currently.
// Dummy data initializer.
ONNX_NAMESPACE::TensorProto dummy_data_proto;
dummy_data_proto.set_name(graph.GenerateNodeArgName(node.Name() + "_dummy_data"));
ONNX_NAMESPACE::TensorProto dummy_xq_proto;
dummy_xq_proto.set_name(graph.GenerateNodeArgName(node.Name() + "_dummy_xq"));
// Set data type to dq node's zp dtype
dummy_data_proto.set_data_type(dq_node_zp_proto->data_type());
dummy_xq_proto.set_data_type(dq_node_zp_proto->data_type());

// Dummy zero point initializer.
ONNX_NAMESPACE::TensorProto dummy_zp_proto;
dummy_zp_proto.set_name(graph.GenerateNodeArgName(node.Name() + "_dummy_zp"));
dummy_zp_proto.set_data_type(dq_node_zp_proto->data_type());

// Dummy scale initializer.
ONNX_NAMESPACE::TensorProto dummy_scale_proto;
dummy_scale_proto.set_name(graph.GenerateNodeArgName(node.Name() + "_dummy_scale"));
dummy_scale_proto.set_data_type(dq_node_scale_proto->data_type());
Comment thread
qti-hungjuiw marked this conversation as resolved.

// Get original float input
const ONNX_NAMESPACE::TensorProto* const_node_data_proto = nullptr;
if (!graph.GetInitializedTensor(where_inputs[const_idx]->Name(), const_node_data_proto) ||
const_node_data_proto == nullptr) {
// The scalar branch is not backed by an initializer (e.g., it's a graph input). Skip transform.
Comment thread
edgchen1 marked this conversation as resolved.
Outdated
LOGS(logger, WARNING) << "WhereDummyDq expects scalar branch to be an initializer. "
<< "DQ: " << dq_node->Name() << ", scalar(input): " << where_inputs[const_idx]->Name();
return Status::OK();
}
Initializer initializer(graph, *const_node_data_proto, graph.ModelPath());
if (dq_node_scale_proto->data_type() != const_node_data_proto->data_type()) {
// WhereDummyDq fills the const value to the dummy DQ's scale
LOGS(logger, WARNING) << "Currently only support existing DQ's scale with same datatype as scalar. "
<< "DQ: " << dq_node->Name() << ", scalar(const): " << where_inputs[const_idx]->Name();
return Status::OK();
}
float dummy_xf = 0;
switch (initializer.data_type()) {
case ONNX_NAMESPACE::TensorProto_DataType_FLOAT: {
dummy_xf = *initializer.data<float>();
break;
}
default:
LOGS(logger, WARNING) << "Unsupported dtype of constant input. "
<< "DQ: " << dq_node->Name() << ", scalar(const): " << where_inputs[const_idx]->Name();
return Status::OK();
}

// TensorProto stores INT8/UINT8/INT16/UINT16 values via `int32_data`.
// Keep values in-range for unsigned cases (0..255 / 0..65535) before writing.
int32_t dummy_zp_i32 = 0;
int32_t dummy_xq_i32 = 0;
float dummy_scale = 1.0f;

switch (dummy_zp_proto.data_type()) {
case ONNX_NAMESPACE::TensorProto_DataType_INT8: {
int8_t zp = 0;
int8_t dummy_data = 1;
utils::SetRawDataInTensorProto(dummy_zp_proto, &zp, 1);
utils::SetRawDataInTensorProto(dummy_data_proto, &dummy_data, 1);
dummy_zp_i32 = 0;
dummy_xq_i32 = (dummy_xf > 0) ? 127 : ((dummy_xf == 0) ? dummy_zp_i32 : -128);
dummy_scale = (dummy_xf == 0) ? 1 : (float)dummy_xf / (dummy_xq_i32 - dummy_zp_i32);
break;
}
case ONNX_NAMESPACE::TensorProto_DataType_UINT8: {
uint8_t zp = 0;
uint8_t dummy_data = 1;
utils::SetRawDataInTensorProto(dummy_zp_proto, &zp, 1);
utils::SetRawDataInTensorProto(dummy_data_proto, &dummy_data, 1);
dummy_zp_i32 = 127;
dummy_xq_i32 = (dummy_xf > 0) ? 255 : ((dummy_xf == 0) ? dummy_zp_i32 : 0);
dummy_scale = (dummy_xf == 0) ? 1 : (float)dummy_xf / (dummy_xq_i32 - dummy_zp_i32);
break;
}
case ONNX_NAMESPACE::TensorProto_DataType_INT16: {
int16_t zp = 0;
int16_t dummy_data = 1;
utils::SetRawDataInTensorProto(dummy_zp_proto, &zp, 2);
utils::SetRawDataInTensorProto(dummy_data_proto, &dummy_data, 2);
dummy_zp_i32 = 0;
dummy_xq_i32 = (dummy_xf > 0) ? 32767 : ((dummy_xf == 0) ? dummy_zp_i32 : -32768);
dummy_scale = (dummy_xf == 0) ? 1 : (float)dummy_xf / (dummy_xq_i32 - dummy_zp_i32);
break;
}
case ONNX_NAMESPACE::TensorProto_DataType_UINT16: {
uint16_t zp = 0;
uint16_t dummy_data = 1;
utils::SetRawDataInTensorProto(dummy_zp_proto, &zp, 2);
utils::SetRawDataInTensorProto(dummy_data_proto, &dummy_data, 2);
dummy_zp_i32 = 32767;
dummy_xq_i32 = (dummy_xf > 0) ? 65535 : ((dummy_xf == 0) ? dummy_zp_i32 : 0);
dummy_scale = (dummy_xf == 0) ? 1 : (float)dummy_xf / (dummy_xq_i32 - dummy_zp_i32);
break;
}
default:
LOGS(logger, WARNING) << "Currently support existing DQ's zero point with INT8, UINT8, INT16, UINT16";
LOGS(logger, WARNING) << "Currently support existing DQ's zero point with INT8, UINT8, INT16, UINT16. "
<< "DQ: " << dq_node->Name() << ", scalar(const): " << where_inputs[const_idx]->Name();
return Status::OK();
}

// Set dummy scale to the original value
const ONNX_NAMESPACE::TensorProto* const_node_data_proto = nullptr;
graph.GetInitializedTensor(where_inputs[const_idx]->Name(), const_node_data_proto);
Initializer initializer(graph, *const_node_data_proto, graph.ModelPath());
if (dq_node_scale_proto->data_type() != const_node_data_proto->data_type()) {
// WhereDummyDq fills the const value to the dummy DQ's scale
LOGS(logger, WARNING) << "Currently only support existing DQ's scale with same datatype as scalar";
return Status::OK();
}

// Dummy scale initializer.
ONNX_NAMESPACE::TensorProto dummy_scale_proto;
dummy_scale_proto.set_name(graph.GenerateNodeArgName(node.Name() + "_dummy_scale"));
dummy_scale_proto.set_data_type(dq_node_scale_proto->data_type());
switch (initializer.data_type()) {
case ONNX_NAMESPACE::TensorProto_DataType_FLOAT: {
float* where_const_scalar = initializer.data<float>();
utils::SetRawDataInTensorProto(dummy_scale_proto, where_const_scalar, sizeof(float));
break;
}
default:
LOGS(logger, WARNING) << "Currently support scalar with FLOAT";
return Status::OK();
}
dummy_zp_proto.add_int32_data(dummy_zp_i32);
dummy_xq_proto.add_int32_data(dummy_xq_i32);
dummy_scale_proto.add_float_data(dummy_scale);

// Start editing the graph
NodeArg& dummy_data_arg = graph_utils::AddInitializerWithOrtValue(graph, dummy_data_proto);
// Start editing the graph:
// - add the initializers
// - add a DeQuantizeLinear node consuming them
// - rewire the scalar branch of `Where` to use the DQ output
// - drop the original scalar initializer if it becomes unused
NodeArg& dummy_xq_arg = graph_utils::AddInitializerWithOrtValue(graph, dummy_xq_proto);
NodeArg& dummy_scale_arg = graph_utils::AddInitializerWithOrtValue(graph, dummy_scale_proto);
NodeArg& dummy_zp_arg = graph_utils::AddInitializerWithOrtValue(graph, dummy_zp_proto);

Expand All @@ -132,7 +206,7 @@ Status WhereDummyDq::InsertDummyDQ(Node& node, Graph& graph, bool& modified, con
graph.GenerateNodeArgName(node.Name() + "_dummy_dq"),
QDQ::DQOpName,
"DeQuantizeLinear from WhereDummyDq GraphTransformer",
{&dummy_data_arg, &dummy_scale_arg, &dummy_zp_arg},
{&dummy_xq_arg, &dummy_scale_arg, &dummy_zp_arg},
{&dummy_dq_arg},
node,
nullptr,
Expand Down
24 changes: 22 additions & 2 deletions onnxruntime/core/optimizer/qdq_transformer/where_dummy_dq.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,27 @@ namespace onnxruntime {
@Class WhereDummyDq

Graph transformer that inserts a dummy DQ on Where node's initializer input
to form Node Unit when Where node has one DQ and one scalar initializer input
to form Node Unit when Where node has one DQ and one scalar initializer input.

If `Where` gets a float scalar `xf` and a `DequantizeLinear` as its two data inputs,
`WhereDummyDq` inserts a dummy DQ so that `xf ≈ DQ(xq, scale, zp)`.

The `xq`, `zp` are chosen per the table below (by the dtype of the existing DQ's zero-point),
and `scale` is computed from them:

| | uint8 | uint16 | int8 | int16 |
|-----------------|--------|--------|-------|--------|
| xf > 0 | | | | |
| xq | 255 | 65535 | 127 | 32767 |
| zp | 127 | 32767 | 0 | 0 |
| xf < 0 | | | | |
| xq | 0 | 0 | -128 | -32768 |
| zp | 127 | 32767 | 0 | 0 |
| xf = 0 | | | | |
| xq | 127 | 32767 | 0 | 0 |
| zp | 127 | 32767 | 0 | 0 |

scale = xf / (xq - zp) if (xq != zp) else 1
Comment thread
edgchen1 marked this conversation as resolved.
*/
class WhereDummyDq : public GraphTransformer {
public:
Expand All @@ -23,4 +43,4 @@ class WhereDummyDq : public GraphTransformer {
bool SatisfyCondition(const Graph& graph, const Node& node) const;
Status InsertDummyDQ(Node& node, Graph& graph, bool& modified, const logging::Logger& logger) const;
};
} // namespace onnxruntime
} // namespace onnxruntime
Loading