Enable BF16 for opset 28 operators by justinchuby · Pull Request #32378 · microsoft/onnxruntime

Justin Chu (justinchuby) · 2026-09-02T06:03:40Z

Summary

enable CPU BF16 execution for opset-28 Compress, OneHot, ReverseSequence, Unique, Celu, and Einsum
preserve legacy type coverage with v28 registration splits; add CUDA BF16 support only for executable Compress, OneHot, and ReverseSequence paths
cap unsupported BF16 registrations for CUDA Einsum, JS, and WebGPU while preserving their existing v28 types
add focused BF16 execution coverage, including Celu FP16/BF16/double and Einsum matmul, reduction, and diagonal paths

Validation

onnxruntime_providers target builds successfully with all changed provider translation units
all six focused provider-test translation units compile successfully
onnxruntime_provider_test cannot link because the base branch's ONNX archive has unresolved kDoc_DepthToSpace_ver28 and kDoc_SpaceToDepth_ver28 symbols; this also prevents running the tests and regenerating OperatorKernels.md locally
git diff --check passes
independent review completed; initial registration-boundary findings were addressed, with no remaining issues

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e64e522-b07f-4c1c-abf1-3985268beb7e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Xavier Dupré (xadupre) · 2026-09-02T08:42:46Z

einsum for BFloat16 is not going to be very efficient since it is not using mlas.

Justin Chu (justinchuby) · 2026-09-02T17:22:24Z

einsum for BFloat16 is not going to be very efficient since it is not using mlas.

True. I will create a follow up PR for this

Copilot

🟡 Changes recommended

Legacy Unique coverage is broadened, generated kernel documentation and OneHot coverage are incomplete, and BF16 Einsum matmul ignores CPU parallelism.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Enables opset-28 BF16 execution across CPU and selected CUDA operators while preserving older registration boundaries.

Changes:

Adds opset-28 CPU/CUDA kernel registrations.
Implements BF16 Celu, Unique, and Einsum execution paths.
Adds focused BF16 provider tests.

File summaries

File	Description
`onnxruntime/test/providers/cpu/tensor/unique_op_test.cc`	Tests BF16 Unique.
`onnxruntime/test/providers/cpu/tensor/reverse_sequence_test.cc`	Tests BF16 ReverseSequence.
`onnxruntime/test/providers/cpu/tensor/onehot_op_test.cc`	Tests BF16 OneHot.
`onnxruntime/test/providers/cpu/tensor/compress_op.test.cc`	Tests BF16 Compress.
`onnxruntime/test/providers/cpu/math/einsum_test.cc`	Tests BF16 Einsum paths.
`onnxruntime/test/providers/cpu/activation/activation_op_test.cc`	Tests new Celu types.
`onnxruntime/core/providers/webgpu/webgpu_execution_provider.cc`	Splits WebGPU Einsum registration.
`onnxruntime/core/providers/webgpu/math/einsum.cc`	Registers WebGPU Einsum v28.
`onnxruntime/core/providers/js/operators/einsum.cc`	Splits JS Einsum registration.
`onnxruntime/core/providers/js/js_execution_provider.cc`	Registers JS Einsum v28.
`onnxruntime/core/providers/cuda/tensor/reverse_sequence.cc`	Adds CUDA BF16 ReverseSequence.
`onnxruntime/core/providers/cuda/tensor/onehot.cu`	Instantiates CUDA BF16 OneHot.
`onnxruntime/core/providers/cuda/tensor/onehot.cc`	Registers CUDA BF16 OneHot.
`onnxruntime/core/providers/cuda/tensor/compress.cc`	Splits CUDA Compress registration.
`onnxruntime/core/providers/cuda/math/einsum.cc`	Caps CUDA Einsum v28 types.
`onnxruntime/core/providers/cuda/cuda_execution_provider.cc`	Registers CUDA opset-28 kernels.
`onnxruntime/core/providers/cpu/tensor/unique.cc`	Implements CPU BF16 Unique.
`onnxruntime/core/providers/cpu/tensor/reverse_sequence.cc`	Splits CPU ReverseSequence registration.
`onnxruntime/core/providers/cpu/tensor/onehot.cc`	Registers CPU BF16 OneHot.
`onnxruntime/core/providers/cpu/tensor/compress.cc`	Splits CPU Compress registration.
`onnxruntime/core/providers/cpu/math/einsum.cc`	Registers CPU BF16 Einsum.
`onnxruntime/core/providers/cpu/math/einsum_utils/einsum_auxiliary_ops.cc`	Implements BF16 Einsum helpers.
`onnxruntime/core/providers/cpu/cpu_execution_provider.cc`	Registers CPU opset-28 kernels.
`onnxruntime/core/providers/cpu/activation/activations.h`	Implements FP16/BF16 Celu.
`onnxruntime/core/providers/cpu/activation/activations.cc`	Registers Celu v28 types.

Review details

Files reviewed: 25/25 changed files
Comments generated: 4
Review effort level: Balanced

💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

G. Ramalingam (gramalingam) · 2026-10-01T18:10:24Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2026-10-01T18:10:31Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

G. Ramalingam (gramalingam) · 2026-10-01T18:10:52Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2026-10-01T18:10:59Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) · 2026-10-01T18:22:59Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2026-10-01T18:23:06Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) · 2026-10-01T18:37:34Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2026-10-01T18:37:41Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

G. Ramalingam (gramalingam) · 2026-10-01T18:38:07Z

/azp run Linux Android Emulator QNN CI Pipeline

azure-pipelines · 2026-10-01T18:38:15Z

Azure Pipelines will not run the associated pipelines, because the pull request was updated after the run command was issued. Review the pull request again and issue a new run command.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

Warning

Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.

Copilot review overview

Review effort: Lite
Findings: 1 · 3 · 1

Open (5)

This BF16 Einsum test will fail under CUDA builds because CUDA Einsum v28 is registered without… · New
The PR title/summary focuses on opset-28 BF16 enablement, but the generated docs update introduces… · New
The accelerated BF16 MatMul path materializes full-size float buffers for all batches… · New
The BF16 ReduceSum adapter allocates float_input using input.Shape() even when… · New
This build-time type guard always checks EnabledUniqueDataTypes28, even though the kernel is… · New

Resolved since last review (4)

This is the only BF16 OneHot execution test, but it uses int64 indices/depth and a nonzero…
BFloat16 is being added to the ALL_OPSETS type list, while both the new [11, 27] registration…
The BF16 matmul fallback discards the supplied thread pool and executes the full O(batch·M·N·K)…
These CPU/CUDA registration changes are not reflected in docs/OperatorKernels.md; for example,…

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

### Description  Reintegrate the ONNX 1.23 base plumbing originally introduced by #32359 and reverted by #33073, now using ONNX **1.23.2**. - Coordinate source/submodule, FetchContent, vcpkg, and Python dependency pins and raise the relevant opset ceilings to 28. - Restore Float6 public element-type mappings and conversion coverage, preserving subsequent independent type-support changes. - Update existing-type Cast registrations, CPU/CUDA Mod boundaries, signed BitShift support, provider compatibility, backend filters/tolerances, and operator documentation. - Materialize the removed upstream node-test corpus with exact-version and minimum-case guards, and cover backend serialization and released-opset policy. - Keep matching FetchContent/vcpkg compatibility patches; remove the released-opset correction now fixed upstream in 1.23.2. - Address review feedback: OpenVINO rejects BitShift-28; CPU/CUDA legacy SpaceToDepth kernels stop at 27; DirectML accepts only safe DCR; unverified graph providers defer SpaceToDepth-28. Full builds use ONNX function expansion for CRD, with literal-output and routing regressions. Avoid creating unsupported SpaceToDepth-28 nodes in the post-expansion Slice/Concat fusion. - Use shared ABI-safe tensor-element conversion in the native ONNX node runner, fixing INT2/UINT2/FLOAT8E8M0 corpus decoding after the main merge. ### Motivation and Context  This is a redo of #32359 after its rollback in #33073. ONNX 1.23.2 includes the external tensor read/re-save fix from onnx/onnx#8472 for onnx/onnx#8471 and the released-opset metadata fix from onnx/onnx#8544 for onnx/onnx#8529. The redundant local `LastReleaseVersionMap` patch has been removed. Remaining FetchContent/vcpkg patches are byte-identical and apply against the pinned release. **Scope:** This PR establishes dependency/runtime base integration. Full per-operator opset-28 support remains separate follow-up work; see #32362, #32369, #32372, #32378, #32381, and #32393. These references describe staged scope, not a claim that every follow-up has landed or every opset-28 operator is supported. Runtime function expansion is a full-build fallback, not minimal/ORT-format runtime coverage. ### Local validation Completed on Windows with retained CPU-only Ninja/MSVC `RelWithDebInfo` builds and strict released-opset policy: - Built `onnxruntime_provider_test`, `onnxruntime_test_all`, `onnx_test_runner`, and Python bindings against ONNX 1.23.2; retained build remains available for incremental work. - Source-built and installed the exact patched ONNX 1.23.2 Python wheel. - Separate `--minimal_build extended` shared runtime configured, compiled, and linked successfully with ML/contrib ops and unit tests disabled. This exercises `ONNX_MINIMAL_BUILD=ON`; it is not a minimal-runtime function-op execution test. - Mod/BitShift/SpaceToDepth/Cast provider tests: **247 passed**, no skips. SpaceToDepth checks default DCR, explicit DCR, and CRD with distinct literal expected outputs, expansion, and legacy opset boundaries. - Slice/Concat fusion and type-info/tensor-proto framework tests: **27 passed**, no skips. - Python backend tests: **22 passed**, plus **2 subtests passed**. - Full Python backend series: **4,066 tests**, **2,685 skipped**, zero failures. Skipped/unsupported coverage is not a pass. - Materialized **1,884 node cases**. Native CPU runner with `-m 1500` collected **1,760 cases**: **1,590 succeeded**, **170 not implemented**, **zero failed**, exit 0. These results do not establish complete opset-28 support. - Generated CPU operator documentation matches the rebuilt kernel registry; verified archive hashes, patch identity/application, submodule SHA, and changed-file lint. **Remaining validation/infrastructure limits:** GPU and other EP execution relies on CI; optional-provider routing checks are compiled into the relevant builds but were unavailable locally. Full unfiltered C++ suites and the ancillary broad `ALL` target were not rerun. The oracle-dependent corpus equivalence test skips because upstream no longer ships the corpus. The new ONNX 1.23.2 tarball currently returns **404 from the Microsoft vcpkg asset mirror**; it still needs Terrapin/infra seeding before read-only `x-block-origin` vcpkg legs can download it. FetchContent and the successful minimal build are mirror-independent. New-head CI is pending. --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com> Co-authored-by: gramalingam <10075881+gramalingam@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: gramalingam <10075881+gramalingam@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Ti-Tai Wang (titaiwangms)

I wonder if we need to trigger that python package run as well. Since we are bumping opset for some ops, there should be tests updated.

Copilot

🟡 Changes recommended

The new FP16 and BF16 Celu implementations silently replace NaN inputs with zero.

1 open finding

Celu incorrectly converts NaN inputs to zero · New

5 resolved since last review

This BF16 Einsum test will fail under CUDA builds because CUDA Einsum v28 is registered without…
The BF16 ReduceSum adapter allocates float_input using input.Shape() even when…
The accelerated BF16 MatMul path materializes full-size float buffers for all batches…
The PR title/summary focuses on opset-28 BF16 enablement, but the generated docs update introduces…
This build-time type guard always checks EnabledUniqueDataTypes28, even though the kernel is…

🧠 Review effort: Balanced

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) · 2026-10-14T20:11:09Z

I wonder if we need to trigger that python package run as well. Since we are bumping opset for some ops, there should be tests updated.

Started: https://aiinfra.visualstudio.com/Lotus/_build/results?buildId=1481778&view=results

Justin Chu (justinchuby) and others added 7 commits September 1, 2026 13:02

Integrate ONNX 1.23.0 base plumbing

4f215a6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Harden ONNX node corpus version checks

1b70a86

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add FP6 TensorProto element sizes

9285320

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6e64e522-b07f-4c1c-abf1-3985268beb7e

Map ONNX Float6 element types

2eabd0e

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Wire materialized node tests to runner

a38811d

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Test ONNX Float6 type mappings

6602645

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Enable opset 28 BF16 operators

4ed5f23

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Xavier Dupré (xadupre) requested a balanced review from Copilot September 9, 2026 10:57

Copilot started reviewing on behalf of Xavier Dupré (xadupre) September 9, 2026 10:58

View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Base automatically changed from justinchuby-onnx-1-23-opset-28-base to main September 30, 2026 15:34

G. Ramalingam (gramalingam) self-assigned this Oct 1, 2026

Fix C++ formatting for BF16 opset 28

0cd0e44

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Address BF16 opset 28 review feedback

f7e841c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) and others added 4 commits October 1, 2026 12:52

Merge main into BF16 opset 28 branch

3ae6b75

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Fix CI for opset 28 feedback changes

8b1ecb8

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Fix unused capture in Celu test

53474d4

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Update generated operator documentation

215d135

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) requested a balanced review from Copilot October 2, 2026 16:20

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Copilot started reviewing on behalf of G. Ramalingam (gramalingam) October 2, 2026 16:59

View session

Address BF16 opset 28 review feedback

fdd2faf

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) mentioned this pull request Oct 6, 2026

Reintegrate ONNX 1.23.2 base plumbing #33134

Merged

Merge main after ONNX reintegration

32aa764

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) assigned Copilot Oct 12, 2026

Copilot started work to fix failing Actions workflow(s) on behalf of G. Ramalingam (gramalingam) October 12, 2026 20:37

View session

Add ONNX version metadata for CI parser

8b993c3

Co-authored-by: gramalingam <10075881+gramalingam@users.noreply.github.com>

Copilot finished work on behalf of G. Ramalingam (gramalingam) October 12, 2026 20:46

Copilot AI requested a review from G. Ramalingam (gramalingam) October 12, 2026 20:46

Fix BF16 Einsum provider test setup

a0d2921

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Ti-Tai Wang (titaiwangms) requested a balanced review from Copilot October 14, 2026 17:23

Copilot started reviewing on behalf of Ti-Tai Wang (titaiwangms) October 14, 2026 17:24

View session

Ti-Tai Wang (titaiwangms) previously approved these changes Oct 14, 2026

View reviewed changes

Copilot AI reviewed Oct 14, 2026

View reviewed changes

Comment thread onnxruntime/core/providers/cpu/activation/activations.h Outdated

Preserve NaN inputs in FP16 and BF16 Celu

4126f60

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

G. Ramalingam (gramalingam) dismissed Ti-Tai Wang (titaiwangms)’s stale review via 4126f60 October 14, 2026 20:03

Copilot started work to fix failing Actions workflow(s) on behalf of G. Ramalingam (gramalingam) October 14, 2026 23:02

View session

Copilot finished work on behalf of G. Ramalingam (gramalingam) October 14, 2026 23:04

Copilot started work to fix failing Actions workflow(s) on behalf of G. Ramalingam (gramalingam) October 15, 2026 01:22

View session

Copilot finished work on behalf of G. Ramalingam (gramalingam) October 15, 2026 01:24

Conversation

Justin Chu (justinchuby) commented Sep 2, 2026

Summary

Validation

Uh oh!

Xavier Dupré (xadupre) commented Sep 2, 2026

Uh oh!

Justin Chu (justinchuby) commented Sep 2, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

G. Ramalingam (gramalingam) commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

G. Ramalingam (gramalingam) commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

G. Ramalingam (gramalingam) commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

G. Ramalingam (gramalingam) commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

G. Ramalingam (gramalingam) commented Oct 1, 2026

Uh oh!

azure-pipelines Bot commented Oct 1, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Ti-Tai Wang (titaiwangms) left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

G. Ramalingam (gramalingam) commented Oct 14, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants