Skip to content

Add VarlenNGramHashMapping for packed/varlen batching support in DeepSeek Engram - #32358

Merged
kunal-vaishnavi merged 55 commits into
mainfrom
copilot/update-ngramhashmapping-for-ogaengine
Oct 1, 2026
Merged

kunal-vaishnavi merged 55 commits into
mainfrom
copilot/update-ngramhashmapping-for-ogaengine

Conversation

Copilot AI commented Sep 1, 2026 •

Copy link
Copy Markdown
Contributor

Description

NGramHashMapping operates on a dense (batch_size, sequence_length) layout and mixes n-gram hash inputs across positions within a row. Packing multiple sequences into one flat buffer (as required by ragged/packed batching engines like OgaEngine) and reshaping to (1, num_tokens) would silently leak hash-mix inputs across sequence boundaries at every packed request boundary.

This adds a new com.microsoft op, VarlenNGramHashMapping, modeled on VarlenCausalConvWithState, which accepts a cumulative_sequence_length (shape (batch_size + 1), device-resident int32) offsets tensor and clamps the n-gram window — and past_ids/present_ids history — at each packed request's own boundary instead of the flat buffer's.

  • Schema: New op in bert_defs.cc (registered in ms_opset.h) taking token-major packed input_ids ((total_tokens)) plus cumulative_sequence_length; past_ids/present_ids are now indexed per-request instead of per fixed-stride batch row.
  • CPU kernel: Parallelizes over requests via ThreadPool::TryParallelFor; validates offsets host-side.
  • CUDA kernel: Block-per-request design (mirrors VarlenCausalConvKernel); each block validates its own offsets before touching memory.
  • WebGPU kernel: Workgroup-per-request dispatch for hash ids, with a separate program for present_ids.
  • Prerequisite port: NGramHashMapping/EngramGate (from unmerged PR Add DeepSeek Engram contrib ops (EngramGate, NGramHashMapping) #32268) were not yet on this branch; selectively ported only the NGramHashMapping-related CPU/CUDA/WebGPU kernels, engram_helper files, and tests needed here (EngramGate and unrelated CausalConvWithState dilation/channels_last changes are out of scope).
  • Docs: Hand-updated docs/ContribOperators.md and docs/OperatorKernels.md to match gen_doc output.
  • Tests: Added coverage in engram_ops_test.cc for packed-vs-per-sequence equivalence, sequence-boundary no-leakage, packed chunked/decode equivalence across concurrent requests, negative ids, and invalid vocab_sizes.
# Equivalent results, but VarlenNGramHashMapping never reads across the sequence boundary
NGramHashMapping(seq_a) + NGramHashMapping(seq_b)
  == VarlenNGramHashMapping(concat(seq_a, seq_b), cumulative_sequence_length=[0, len(seq_a), len(seq_a)+len(seq_b)])

Motivation and Context

NGramHashMapping's dense (batch_size, sequence_length) layout is incompatible with OgaEngine's packed/ragged batching model (continuous batching / paged attention), where tokens from multiple in-flight requests are concatenated into a single flat, unpadded buffer. Without this change, models using DeepSeek Engram would silently corrupt n-gram hash ids at every sequence boundary whenever more than one request is batched together.

Copilot AI and others added 27 commits August 25, 2026 20:18
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
…e for NGramHashMapping/ShortConv (schema, CPU, CUDA)

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
…op docs

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
…ipliers check, fused CUDA reduction

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
T1: templatize the chunked NGramHashMapping equivalence test over int32/int64
so the WebGPU past_ids/present_ids state shaders get execution coverage.

Q1: vectorize EngramGateScalarProgram, EngramGateProgram, ShortConvInvRmsProgram
and ShortConvPresentStateProgram over the contiguous hidden dimension using
GetMaxComponents + Flatten bindings + SumVector reductions. ShortConvProgram
stays scalar because weight is laid out (channel, 1, kernel_size).

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
… CUDA grid/shared-mem, WebGPU empty input guard, negative-id tests

- Reject non-positive vocab_sizes on CPU and document the requirement in the schema
- Explain in the ShortConv doc why it is separate from CausalConvWithState
- CPU ShortConv: allocate inv_rms via GetTempSpaceAllocator; hoist row base pointers
- CUDA: raise GridSize cap from 65535 to the real 2^31-1 grid.x limit
- CUDA: stage multipliers/vocab_sizes in shared memory, mark read-only pointers __restrict__
- WebGPU: do not bind zero-sized input_ids when sequence_length == 0
- Tests: negative ids/pad_id coverage for PositiveMod and WrappedMultiply, plus vocab_sizes rejection

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
…traints

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
…s and docs

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Agent-Logs-Url: https://github.com/microsoft/onnxruntime/sessions/780c3ba6-6eaf-48ed-9080-88cac877e87c

Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Franklin Moormann (ooples) pushed a commit to ooples/AiDotNet that referenced this pull request Oct 13, 2026
Updated
[Microsoft.ML.OnnxRuntime](https://github.com/Microsoft/onnxruntime)
from 1.30.0 to 1.31.0.

<details>
<summary>Release notes</summary>

_Sourced from [Microsoft.ML.OnnxRuntime's
releases](https://github.com/Microsoft/onnxruntime/releases)._

## 1.31.0

ONNX Runtime 1.31.0 stabilizes model-package and EPContext data APIs,
improves CPU model loading and quantized MoE inference, expands
device-based execution-provider selection, and strengthens model-loading
and runtime reliability. These notes cover changes since ONNX Runtime
1.30.0. CUDA and WebGPU kernel updates are summarized here; detailed
provider notes are covered by their separate plugin EP releases.

## Highlights

- Promoted the model-package API and EPContext data callbacks to stable
C and C++ APIs, with EPContext callback support added across language
bindings
([#​33166](microsoft/onnxruntime#33166),
[#​32265](microsoft/onnxruntime#32265)).
- Reduced CPU session-initialization overhead by parallelizing eligible
weight prepacking, and accelerated block-wise INT4/INT8 QMoE experts
with MLAS QNBit kernels and grouped expert dispatch
([#​31691](microsoft/onnxruntime#31691),
[#​32644](microsoft/onnxruntime#32644),
[#​32668](microsoft/onnxruntime#32668)).
- Added x86 FP16 LayerNorm/RMSNorm acceleration, Arm KleidiAI SVE2.1
FP16 GEMM support, and additional RISC-V vector kernels
([#​32715](microsoft/onnxruntime#32715),
[#​32670](microsoft/onnxruntime#32670),
[#​32710](microsoft/onnxruntime#32710)).
- Enabled CoreML participation in device-based EP selection, including
Apple Neural Engine selection through `PREFER_NPU`
([#​31975](microsoft/onnxruntime#31975)).
- Added workspace-memory accounting and verification for
constrained-memory graph partitioning, alongside broader model, graph,
and tensor validation
([#​31962](microsoft/onnxruntime#31962),
[#​32189](microsoft/onnxruntime#32189)).

## Announcements & Compatibility

- [CUDA Plugin EP
v0.3.0](https://github.com/microsoft/onnxruntime/releases/tag/plugin-ep-cuda/v0.3.0)
and WebGPU Plugin EP v0.5.0 are matched with ONNX Runtime 1.31.0. For
**contrib-op compatibility**, core and the plugin must use the same
contributed-operator schemas; building both from the **same ONNX Runtime
revision** is recommended. Schema mismatches can cause incorrect
execution or crashes and may not be detected during plugin registration.
- **`GatherBlockQuantized` behavior change:** out-of-range indices now
produce zero output slices across CPU, CUDA, and WebGPU for both integer
and floating-point quantized formats. This deliberately differs from
ONNX `Gather` error semantics
([#​32480](microsoft/onnxruntime#32480)).
- **Experimental API migration:** model-package consumers should use
`OrtApi::GetModelPackageApi` and the stable `OrtModelPackageApi` table.
EPContext callback registration now uses stable APIs; the corresponding
experimental lookup entries were removed. The stable native callback
contract carries callback/state pointers, with payload limits enforced
by application callbacks, bindings, or EPs rather than a native
read-options policy
([#​33166](microsoft/onnxruntime#33166),
[#​32265](microsoft/onnxruntime#32265)).
- **ACL EP is deprecated** and will be removed in a future release.
Source builds using `--use_acl` now receive a deprecation warning
([#​32564](microsoft/onnxruntime#32564)).
- The ONNX dependency remains at **1.22.0**. The initial ONNX 1.23
integration was reverted pending upstream fixes; this release does not
ship that integration's new opset or FLOAT6 support
([#​33073](microsoft/onnxruntime#33073)).
- Supported native builds now default to **1DS telemetry**. Windows
TraceLogging remains selectable with `--use_windows_telemetry` or
`onnxruntime_USE_WINDOWS_TELEMETRY=ON`; telemetry can be disabled with
`--no_telemetry`
([#​32384](microsoft/onnxruntime#32384)).

## New Features

### Core APIs & Runtime

- Stabilized the model-package C and C++ APIs for loading packages,
selecting compatible model variants, and creating sessions
([#​33166](microsoft/onnxruntime#33166)).
- Stabilized named EPContext data read/write callbacks for
application-managed storage. Added EP support negotiation so registered
callbacks do not silently fall back to filesystem I/O
([#​32265](microsoft/onnxruntime#32265)).
- Extended the Compile API to write external initializers and EP context
data to buffers, enabling in-memory compilation workflows for models
larger than 2 GB
([#​31347](microsoft/onnxruntime#31347)).
- Added workspace-memory accounting, source reporting, and
post-partition reservation verification. Set
`session.strict_workspace_verification=1` to fail initialization on
workspace declaration overruns or nonzero orphaned reservations; strict
verification is not supported for ORT-format model loads
([#​31962](microsoft/onnxruntime#31962),
[#​32189](microsoft/onnxruntime#32189)).
- Added the opt-in source-build option
`onnxruntime_DISABLE_DEVICE_DISCOVERY=ON` to skip platform accelerator
probing and report only the CPU device
([#​32592](microsoft/onnxruntime#32592)).

### Shared Contrib Operators

These updates affect core operator schemas and cross-provider behavior,
not only the separately released GPU plugins.

- Extended `EngramGate` and `NGramHashMapping` for Qwen4-Exp, and added
`VarlenNGramHashMapping` for packed DeepSeek Engram batching with
request-local history. Packed hashing also supports Qwen4-Exp head
offsets, EOS resets, and segment boundaries
([#​32285](microsoft/onnxruntime#32285),
[#​32358](microsoft/onnxruntime#32358),
[#​32467](microsoft/onnxruntime#32467)).
- Added the `activation` attribute to `GatedRMSNorm` for SiLU, its Swish
alias, and Sigmoid gating across CPU, CUDA, and WebGPU. SiLU remains the
default
([#​32512](microsoft/onnxruntime#32512)).
- Added weightless hyper-connection operators `BranchwiseRMSNorm`,
`ScaledSiLU`, `HyperConnectionPreMix`, and `HyperConnectionPostMix`,
with CPU, CUDA, and WebGPU implementations
([#​32687](microsoft/onnxruntime#32687)).
- Extended `GatherBlockQuantized` to FP8/FP4 data, full-axis blocks, and
scale broadcasting, and improved CPU UINT8 dequantization. See the
out-of-range-index behavior change above
([#​32480](microsoft/onnxruntime#32480),
[#​32621](microsoft/onnxruntime#32621)).

### Plugin Execution Provider Infrastructure

- Added support for statically linked plugin EPs, with a static WebGPU
plugin build configuration
([#​32395](microsoft/onnxruntime#32395)).
- Preserved caller streams in single-tensor plugin data transfers and
handled null plugin allocator returns
([#​32666](microsoft/onnxruntime#32666),
[#​32683](microsoft/onnxruntime#32683)).
- Deduplicated AutoEP custom-op domains and corrected EP assignment
reporting after NHWC layout transformation
([#​32711](microsoft/onnxruntime#32711),
[#​33184](microsoft/onnxruntime#33184)).
- Added reference guidance for compiled-model compatibility validation,
including multi-device scenarios
([#​29168](microsoft/onnxruntime#29168)).

## Execution Provider Updates

### CUDA EP

CUDA updates include sparse attention and packed continuous batching,
expanded paged speculative decoding, INT2 and mixed-width QMoE
execution, opt-in MatMul auto-tuning, improved MXFP4 support on
Blackwell consumer GPUs, and reliability fixes.
 ... (truncated)

Commits viewable in [compare
view](microsoft/onnxruntime@v1.30.0...v1.31.0).
</details>

Updated
[Swashbuckle.AspNetCore](https://github.com/domaindrivendev/Swashbuckle.AspNetCore)
from 10.2.3 to 10.3.0.

<details>
<summary>Release notes</summary>

_Sourced from [Swashbuckle.AspNetCore's
releases](https://github.com/domaindrivendev/Swashbuckle.AspNetCore/releases)._

## 10.3.0

## What's Changed

* Change nullability handling for schemas with `anyOf`, `allOf`, `oneOf`
or `type = null` by @​ldeluigi in
domaindrivendev/Swashbuckle.AspNetCore#3976
* Handle leading slash on route prefix during index.html GET by
@​superstator in
domaindrivendev/Swashbuckle.AspNetCore#4084
* Move NRT handling to DataContractResolver by @​desjoerd in
domaindrivendev/Swashbuckle.AspNetCore#3998
* Clarify nullable behaviour by @​martincostello in
domaindrivendev/Swashbuckle.AspNetCore#4094
* Fix review issues for caching and encoding by @​martincostello in
domaindrivendev/Swashbuckle.AspNetCore#4093
* Support multiple response types per status code by @​DeagleGross in
domaindrivendev/Swashbuckle.AspNetCore#4100
* Add coverage for `AddServer` across multiple documents by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4115
* Fix Swagger sort key fallback for empty endpoint tags metadata by
@​AleksaMCode in
domaindrivendev/Swashbuckle.AspNetCore#4125
* Fix parameter matching to consider location when names collide by
@​nonsoikem in
domaindrivendev/Swashbuckle.AspNetCore#4126
* Emit form encoding for properties of a complex form parameter by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4128
* Keep a property's own summary when comments come from several XML
files by @​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4129
* Strip query string from relative document URLs in Swagger UI by
@​GabeSilvaDev in
domaindrivendev/Swashbuckle.AspNetCore#4152
* Document that System.Text.Json ignores `[EnumMember]` by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4127
* Drop the regex route constraint from the default Swagger route by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4130
* Serialize custom values in the UI `ConfigObject.AdditionalItems` by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4132
* Use Minimal APIs JSON options when MVC is not registered by
@​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4116
* Resolve enum contracts against the underlying type under source
generation by @​RaphaelFakhri in
domaindrivendev/Swashbuckle.AspNetCore#4131

## New Contributors

* @​ldeluigi made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#3976
* @​superstator made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4084
* @​desjoerd made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#3998
* @​RaphaelFakhri made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4115
* @​AleksaMCode made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4125
* @​nonsoikem made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4126
* @​GabeSilvaDev made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4152
* @​JulianThoms made their first contribution in
domaindrivendev/Swashbuckle.AspNetCore#4182

**Full Changelog**:
domaindrivendev/Swashbuckle.AspNetCore@v10.2.3...v10.3.0


Commits viewable in [compare
view](domaindrivendev/Swashbuckle.AspNetCore@v10.2.3...v10.3.0).
</details>

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore <dependency name> major version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's major version (unless you unignore this specific
dependency's major version or upgrade to it yourself)
- `@dependabot ignore <dependency name> minor version` will close this
group update PR and stop Dependabot creating any more for the specific
dependency's minor version (unless you unignore this specific
dependency's minor version or upgrade to it yourself)
- `@dependabot ignore <dependency name>` will close this group update PR
and stop Dependabot creating any more for the specific dependency
(unless you unignore this specific dependency or upgrade to it yourself)
- `@dependabot unignore <dependency name>` will remove all of the ignore
conditions of the specified dependency
- `@dependabot unignore <dependency name> <ignore condition>` will
remove the ignore condition of the specified dependency and ignore
conditions


</details>

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
This was referenced Oct 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants