Repository navigation
perf(exl3): accelerate fat-expert prefill - #77
Conversation
Corrected zero-cache E2 field matrixPlease treat this comment as a correction to, and replacement for, the cold-prefill performance evidence currently in the PR body. After auditing the earlier 64K/100K receipts, those runs had APC hits and should not be cited as cold-prefill evidence. The implementation/GPU parity checks are unaffected; only the earlier performance methodology is being corrected here. I reran a four-arm matrix on the local 2× DGX Spark deployment. The same image was used for every arm. Within each fixed-MNBT pair, only the three E2 rollout flags changed; MNBT intentionally differed between the 2048 and 3584 pairs. Protocol and validity gate
A cold row was accepted only when all four independently reported conditions held:
Missing fields now fail validation. I re-read the stored receipts using that hardened rule: all 30 cold rows contained every required field and passed all four checks. Every cold response returned Controlled four-arm resultsPrefill throughput in local-compute tok/s:
¹ The 3584-on values are the median of two separate boots. The other three arms are one cold ladder each. The 3584-on repeat drift was −3.48%, −2.32%, −1.07%, −1.10%, −0.53%, and +0.57% from 8K through 300K respectively; from 16K onward it stayed within about ±1.1%. This matrix validates the complete E2-on stack against the legacy flags-off path. It does not separately re-isolate E1 from E2. Decode checkMedian of 5×400-token runs:
² Central value across the two 3584-on boots. Structured decode is effectively unchanged, consistent with E2 not changing the decode path. Prose moved in opposite directions in the two fixed-MNBT comparisons and tracked DFlash acceptance, so this matrix does not establish a consistent E2 prose cost. One 2048-off prose run triggered the benchmark's lexical README comparison — context only, not a causal A/BThe existing README cold rows are lower than this deployment's current 2048-E2 measurements:
This README comparison is descriptive only: the README rows mix dates and DFlash TP geometry (8K/16K/100K are TP2 C4 measurements; 12K/256K/300K are from the older TP1 ladder). The controlled flags-off/flags-on table above is the causal evidence for E2. Conclusion and rollout recommendationOn this exact deployment, fully uncached prefill improved by:
However, 3584 improves over the already-fast 2048-E2 arm by only 1.8%–3.0% at 16K–300K and regresses 8K by 7.9%. I therefore restored production to MNBT 2048 with E2 enabled. MNBT 3584 remains reasonable only for a dedicated long-prefill workload where the extra 2%–3% matters more than short-prompt latency. These figures are deployment-specific. A separate deployment did not reproduce the 2048 gain, so the PR should retain opt-in rollout flags and should not present the result as universal across GB10 configurations. Limitations / deviation
The main corrected result is therefore narrower but defensible: on this deployment, E2 is a real approximately 18%–19% fully uncached 100K–300K prefill improvement at the retained MNBT 2048 setting, while decode remains effectively unchanged. |
EXL3_FAT_KERNEL experiments had no way to prove which tier actually ran: the label survived decode calls, shared-SUH loss silently dropped the kernel to sorted, and the only symbol check fired mid-prefill. Add a stable schema (EXL3_FAT_DIAG_KEYS/SCHEMA v1, exl3_fat_diag()) covering configured/effective tier, reason, shared-SUH, symbol availability, capability, TP identity, fat layers/experts/max rows, direct/scatter calls, per-tier fallback calls+reasons, and scratch alloc/current/peak bytes. Counters are integer bumps on the prefill path only; the label resets every call. Logging stays at load and the existing 42-layer cadence, never per expert. Tier resolution happens at weight load, checkpoint cap first: without shared SUH the tier caps at sorted (the stacked gate+up GEMM shares one input Hadamard, and sorted needs no fat symbols), whatever the image. EXL3_FAT_KERNEL=1 with shared SUH but missing fat symbols fails closed at load with the missing symbol named. Lower tiers stay legitimate defaults.
_check_e2_diag_static pins the schema key set and the full tier map, including the precedence: shared-SUH loss returns sorted/ shared_suh_absent even when both fat symbols are missing, while EXL3_FAT_KERNEL=1 with shared SUH and missing symbols raises (covers E1 images too). _check_e2_diag measures one accepted E2 fat prefill from a cleared counter window and asserts every counter exactly: prefill_layer_calls 1, fallback kernel=1, direct=2, scatter=2, fat_expert_runs=2, scratch_allocs=1, bytes==peak>0. It then proves a decode call cannot inherit the kernel label, shared-SUH loss reports sorted+degraded_shared_suh with no kernel calls, and row tiles preempt the tier visibly.
PR #77 local promotion sequence — valid on/off/on evidenceThis replaces the earlier under-repeated local promotion claim. The original APC-contaminated table remains withdrawn. Exact arm contract
Distribution summaryCompute throughput is
Boot-specific gains versus the intervening off control:
Every rung has complete distribution separation: the slowest on observation is faster than the fastest off observation. Engagement diagnosticsFinal cumulative receipts were identical across TP ranks within each boot:
All accepted cold observations
All 45 rows returned HTTP 200, ConclusionOn this deployment, E2 is engaged on both ranks and produces a repeatable ~20–21% fully uncached prefill gain at 8K–300K with complete distribution separation. The kernel is not silently falling back here. Rollout should remain opt-in until the maintainer deployment runs this same diagnostic image/protocol; a neutral result there with |
Maintainer kit — E2 at MNBT=7168 (2026-09-01)
One-shot unique-salt
7168 ≈ 3548 on 100k–300k (≤1%). 2048 loses on 100k/256k. Vs this kit’s C4 keep, 7168 E2 is about +15% at 100k and +18% at 300k. Direct/scatter counters were nonzero on every E2 boot ( Follow-up on this branch: default |
GHCR :exl3 predates exl3_fat_gemm, so EXL3_FAT_KERNEL=1 was a no-op until BUILD=1. Stamp the image from the overlay recipe and rebuild when git pull changes it. 7168 was the best E2 one-shot on this kit. Co-authored-by: Cursor <cursoragent@cursor.com>
Independent reproduction on a second 2× DGX Spark deployment — E2 gain confirmedWe independently reproduced this PR's fat-expert tiers on our own 2× DGX Spark GB10 deployment Method (mirrors this PR's protocol)
Results (medians of accepted cold rows per boot)
Pooled kernel vs pooled off: +24.5% @8k (median-of-medians +22.2%), +21.6% @50k, +21.0% @100k, Engagement and safety receipts
MNBT cross-check (kernel tier; 5 cold reps/rung, 3 at 300K where shown)Since main just defaulted
On this deployment, 7168 measured slower than every 2048 boot: −1.2% to −1.8% at 50K ConclusionOn this second, independently operated 2× DGX Spark deployment, E2 is engaged on both ranks and Raw per-row JSONL, boot logs, diag lines and the derivation script are archived on our side and can Submitted by nood-co1 on behalf of Blockbrain Labs (https://x.com/blockbrain_labs, GitHub org blockbrain-ai). Measured on our 2× NVIDIA DGX Spark deployment. |
plotarmordev
added
the
reviewed
Summary
Accelerate EXL3 routed-expert prefill without changing the decode path:
EXL3_FAT_SORTED,EXL3_FAT_BATCHED, andEXL3_FAT_KERNELrollout switches; higher tiers imply their prerequisites.Implementation
exl3_fat_gemmfor SM121.Verification
Executed from clean PR head
4b8d3c7based onorigin/main493cb88:A local diagnostic follow-up (
86e09e0+f1d1d83, not yet pushed) adds a stable 32-key receipt, both-rank tier/symbol/fallback/call/workspace counters, and fail-closed symbol handling. Its GPU self-check and live on/off/on campaign passed.Performance evidence — local promotion sequence complete
The original 64K/100K figures remain withdrawn because they included APC hits. The earlier corrected one-shot table was diagnostic only and is superseded by this repeated sequence.
Protocol:
glm53-pr77-diag:e7e1286-v3, derived from provenglm53-fat-e2:validated-20260901by replacing onlyvllm/.../quantization/exl3.pyand its narrow E2 self-check.cached_tokens=0, prefix-cache hit delta 0, local-cache-hit delta 0, andlocal_compute == prompt_tokens; missing fields fail closed.local_compute / TTFT.Boot-specific gains versus the intervening off control were +19.56%/+21.09% at 8K, +21.04%/+21.67% at 100K, and +20.13%/+21.27% at 300K. Every rung has complete separation: the slowest on observation exceeded the fastest off observation.
Engagement and fallback proof
Final receipts were identical across both TP ranks within each boot:
direct_calls=scatter_calls=555140; kernel calls43569; batched/sorted/legacy calls0; one 130027520-byte scratch allocation.0; legacy calls43568; kernel/batched/sorted calls0; scratch allocations0.direct_calls=scatter_calls=560536; kernel calls43566; batched/sorted/legacy calls0; one 130027520-byte scratch allocation.42/42, capability 12.1,max_rows=2048, zero row-tile calls, and no unintended lower-tier fallback.All samples, per-arm distributions, exact cache deltas, and diagnostic receipts are in the latest PR comment.
Scope of the conclusion
On this deployment, E2 is genuinely engaged and yields a repeatable ~20–21% fully uncached prefill gain at 8K–300K. The result is not a silent fallback artifact.
A separate deployment previously did not reproduce the 2048 gain. Rollout therefore remains opt-in pending the maintainer deployment running the same diagnostic image/protocol. If its direct/scatter counters are nonzero but performance is neutral, the effect is geometry-sensitive; zero calls or a fallback reason indicates integration drift.
Rollout
Flags select the highest enabled tier:
EXL3_FAT_SORTED=1enables sorted routing.EXL3_FAT_BATCHED=1enables E1 batching and implies sorted routing.EXL3_FAT_KERNEL=1enables E2 and implies E1 batching and sorted routing.Setting all three to
0restores the legacy fat-expert path.