Track L — genuine gated delta rule + Mamba2 SSM (L0–L4.1) - #77
Merged
Merged
Conversation
Frontier hybrids (Qwen3.6, Nemotron-3, Mellum2) need a dual-form linear mixer the existing surface lacks. L0 found the shipped gated_deltanet / kimi_delta / modified_delta reference is gated *linear* attention — it omits the DeltaNet (I − β·k·kᵀ) erase term — so this adds the genuine rule, each step gated by an oracle. - L0: lock the linear-vs-delta distinction; honesty note at the mislabel source (_delta_attention_impl). The "Mamba2 op landed" drift was a misquote — CLAUDE.md is honest (selective_ssm op is pending). - L1: gated_delta_rule_recurrent — true erase recurrence, fp32 state, validated vs an independent (I−βkkᵀ) brute-force; erase=False ≡ existing. - L2: gated_delta_rule_chunked — chunk-parallel UT-transform; the only non-GEMM step is the (I+Ã)⁻¹ forward substitution; chunk ≡ recurrent across all gating modes, chunk-size-invariant. - L1.1/L2.1: tessera_apple_gpu_gated_delta_rule[_chunked]_f32 — the rule on Metal (per-(b,h) MSL scan + threadgroup chunk kernel), DESIL-gated Metal ≡ numpy ≡ chunked. Finding: the rule needs L2-normalized keys (β·‖k‖²<1, a contraction); unnormalized keys → genuine f32 ill-conditioning. - L3: tessera.stdlib.hybrid — HybridSchedule makes layer_types first-class (Qwen3.6 [lin×3,full]·N); reference stack threads the dual cache (recurrent Ŝ for linear layers + KV for full layers); oracle: streaming decode ≡ full recompute. 42 oracles across 3 new test files; full Track L surface + regression green. Survey + ladder: docs/audit/roadmap/FRONTIER_MODEL_SURVEY_2026.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The coverage registry asserted a "dedicated Mamba2 Graph IR op landed (2026-05-18) as tessera.selective_ssm" and set graph_ir_lowering="registered" — but NO such op existed in TesseraOps.td (only op_catalog had the name). Registry intent that outran the compiler surface (Decision #25/#26). This makes the claim true. - Tessera_SelectiveSsmOp in TesseraOps.td: x, a, b, c, delta (+ optional gate / init-state), chunk_size attr; SSD recurrence documented. - SelectiveSsmOp::verify in TesseraOps.cpp: rank-3 x / shape-equal delta / matching b,c / A rank-1|2 / optional gate shape-equal x / state rank-3. - Lit fixture tests/tessera-ir/model_class/selective_ssm.mlir (roundtrip + optional gate(...)/init(...)); tessera-opt rebuilds clean under MLIR 22.1.6; model_class lit sweep 5/5. - primitive_coverage.py: corrected the false "landed" comment; the graph_ir_lowering="registered" flag is now backed by a real op. The chunked-parallel SSD lowering (_mamba_ssd.py, chunk ≡ sequential) and the closed-form JVP already existed and stay green (test_mamba_ssd_gpu.py). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Drift-gate follow-up: the Track L commits added Apple GPU C-ABI symbols (gated_delta_rule + gated_delta_rule_chunked) and the SelectiveSsmOp verifier without regenerating the script-owned dashboards. Regenerated via scripts/check_generated_docs.sh --write: - runtime_abi: gated_delta_rule[_chunked]_f32 - verifier_coverage: SelectiveSsmOp (real) - test_coverage: new Track L test files Drift check: 16 generated docs in sync. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ties L4 (selective_ssm op) into the L3 dual-cache hybrid stack so a Nemotron-shaped model — Mamba SSM layers + sparse attention anchors — is expressible end-to-end. - HybridConfig.linear_mixer = "delta" | "ssm" (+ ssm_state N); mixer_for(i) returns full / delta / ssm. - Refactored the stack to per-mixer *span functions* so hybrid_forward (one span) and hybrid_decode (streamed spans) share identical per-layer code — the dual-cache oracle now covers delta Ŝ, SSM h[D,N], and attention KV. - _ssm_scan: the Mamba-2 SSD recurrence that reproduces the shipped tessera.ops.selective_ssm AND returns the carried state (the public reference does not), so streaming SSM decode is exact. Oracles (test_stdlib_hybrid.py, +5): _ssm_scan ≡ selective_ssm; Nemotron dual-cache decode ≡ recompute across prefill points; all-SSM stack. Existing L3 delta oracles unchanged (the refactor preserves numerics). Regenerated test_coverage dashboard (drift gate). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Track L makes the dual-form linear mixer + hybrid layer schedule a first-class, executable surface — the highest-leverage gap from the 2026-H1 frontier-model survey (it unblocks Qwen3.6, Nemotron-3, Mellum2, LFM2.5). Along the way it fixes two genuine registry/prose drifts where intent had outrun the compiler surface.
Two flagship hybrids are now expressible end-to-end against numpy oracles: Qwen3.6 (gated-DeltaNet hybrid) and Nemotron-3 (Mamba-SSM hybrid).
The core finding (L0)
The shipped
gated_deltanet/kimi_delta/modified_deltareference computes gated linear attention — its state updateŜ_t = α·Ŝ_{t-1} + β·k·vᵀomits the DeltaNet(I − β·k·kᵀ)erase term. Every parity test passed because the GPU path faithfully matched the mislabeled reference. Track L adds the genuine rule beside it, each step gated by an oracle, and leaves the shipped op's numerics untouched (changing them is tracked as L3.1).What landed
gated_delta_rule_recurrent— true erase recurrence (decode form), fp32 state; validated vs an independent(I−βkkᵀ)brute-force;erase=False≡ existing refgated_delta_rule_chunked— chunk-parallel UT-transform (prefill form); only non-GEMM step is the(I+Ã)⁻¹forward substitution; chunk ≡ recurrent, chunk-size-invarianttessera_apple_gpu_gated_delta_rule_f32— the rule on Metal (per-(b,h) MSL scan); DESIL Metal ≡ numpytessera_apple_gpu_gated_delta_rule_chunked_f32— threadgroup-per-(b,h) chunk kernel, on-device UT solve; Metal chunked ≡ numpy ≡ Metal recurrenttessera.stdlib.hybrid—HybridSchedulemakeslayer_typesfirst-class; reference stack threads the dual cache (recurrent Ŝ + KV); oracle: streaming decode ≡ full recomputeTessera_SelectiveSsmOp— materialized the Mamba2 ODS op the registry falsely claimed had landed; verifier + lit fixture;tessera-optrebuilds clean (MLIR 22.1.6)linear_mixer="ssm"Mamba mixer in the hybrid stack;_ssm_scan≡ shippedselective_ssm+ returns carried state → Nemotron dual-cache decode ≡ recomputeNumerics finding
The delta rule is only well-conditioned with L2-normalized keys — then
β·‖k‖²=β<1makes(I−βkkᵀ)a contraction and f32≡f64. Unnormalized keys (‖k‖²≫1) make the recurrence expand and f32 legitimately diverges from f64 (~10%). Caught during the L1.1 Metal smoke; encoded in the tests and the hybrid mixers, noted as anumeric_policycontract (key-normalization + fp32 state accumulation).Drift fixes (Decision #25/#26)
primitive_coverage.pyasserted a "Mamba2 Graph IR op landed (2026-05-18)" +graph_ir_lowering="registered"while no ODS op existed. L4 makes it true; comment corrected._delta_attention_impl(it's linear attention, not the delta rule).Tests & verification
test_stdlib_delta_rule.py,test_apple_gpu_gated_delta_rule.py,test_stdlib_hybrid.py; all Metal DESIL tests run (not skipped) on Apple Silicon.tessera-optrebuilds clean under MLIR 22.1.6;model_classlit sweep 5/5 (incl. newselective_ssm.mlir).test_mamba_ssd_gpu.py(chunk ≡ sequential) green; L3 delta oracles preserved through the L4.1 refactor.runtime_abi,verifier_coverage,test_coverage); drift gate clean.Not in scope (tracked)
L2.2 (cooperative-parallel chunk kernel — perf), L3.1 (promote
gated_deltanetODS → true rule, a shipped-numerics decision), L5 (LFM2.5 LIV mixer), MoE/MTP composition into the hybrid stack.Full ladder + survey:
docs/audit/roadmap/FRONTIER_MODEL_SURVEY_2026.md.🤖 Generated with Claude Code