[justinchuby]

Summary First falsifiable consumer increment for #738, using the truthful native KV admission geometry merged in #644. - Preserves the typed SchedulerAdmissionError::ByteBudget classification across the engine-driver boundary. - Maps only that memory-admission refusal to HTTP 429 with Retry-After: 1 and stable resource_limit_error JSON naming the KV memory resource limit. - Leaves unrelated generation failures as HTTP 500; BatchFull is not treated as memory overload. - Reuses the existing request middleware accounting, so each 429 increments onnx_genai_rejections_total exactly once. - Adds no budget, concurrency threshold, queue cap, preemption, or swap state. This advances #738 and #706 but intentionally does not close the full #738 preemption/state-machine issue. ## Tests - HTTP status/header/body mapping test. - Typed scheduler-error classification test (KV byte budget vs unrelated batch-full). - Rejection metric increments exactly once for 429 and not for unrelated 500. - Sabotage: changed memory mapping back to 500; the HTTP test failed (500 vs 429), then restored. ## Validation - cargo fmt --all -- --check ✅ - cargo clippy --locked --all-targets -p onnx-genai-server -- -D warnings ✅ - cargo clippy --locked --all-targets -p onnx-genai-engine --features native-backend -- -D warnings ✅ - Targeted server refusal tests ✅ - Full server lib target: 152 passed, 2 ignored; one unrelated fixture-path failure (vision.onnx absent in the compatibility-package fixture). - CUDA/native server build ✅ ## Governed qwen14b-zp E2E Command: native CUDA server with --vram-limit 6000000000. - First accepted request: HTTP 200, response OK, 197.21 s. - Admission estimate remained 196,608 B/token (192 KiB/token). - 16 concurrent identical over-budget requests: 16× HTTP 429, each in 1.68–3.82 s, all with Retry-After: 1 and resource_limit_error. - onnx_genai_requests_waiting: max 0, final 0. - onnx_genai_rejections_total: 0 → 16 (then 17 after a separate body-capture probe), exactly one per 429. - /metrics: 29 overload samples, maximum response latency 0.292 s. ## Review fixes - Streaming completion/chat now await the first driver event before constructing SSE. A first-event KV refusal returns normal HTTP 429; an accepted token/result is buffered and forwarded without loss. Chat emits no role chunk before acceptance. - Audio transcription now uses the same typed generation-failure mapping (KV refusal 429, unrelated failure 500). - Public overload bodies are fixed and omit scheduler IDs, usage, limits, and configuration internals; full details are logged server-side. - Sabotaging the handshake to accept a first-event error made streaming_completion_refusal_is_http_overload_before_sse fail, then was restored. Validation after review fixes: - server format/clippy gate ✅ - streaming completion/chat refusal tests ✅ - transcription mapping tests ✅ - accepted completion/chat stream order and first-token preservation ✅ - governed native CUDA E2E: accepted chat stream returned 200 with role → OK → finish → [DONE]; strict governed overload probe returned HTTP 429 in 0.036 s with Retry-After: 1, exact stable JSON body, rejections_total 0 → 1, requests_waiting=0, and /metrics responding in 83 ms. The original 6 GB governed acceptance/overload evidence remains above; the stricter 1 GB run isolates and falsifies the streaming refusal path. ## Final handshake correction The first-event preflight described above has been replaced by an explicit, exactly-once driver admission oneshot. Engine generation APIs notify it immediately after scheduler admission and before backend execution. Refusals resolve Err(DriverFailure) without starting the backend; accepted requests resolve Ok(()) before tokens/results. Dropped senders resolve as internal errors rather than hanging. Continuous batching now keeps the handshake pending after manager.submit: it resolves only when that handle is assigned a physical decode row. Row-assignment failures resolve before HTTP headers as internal errors (not KV-memory 429s), and canceled pending handles are removed. FIM uses the engine admission callback and no longer waits for its final result before returning SSE headers. Additional validation: - delayed-first-token admission test returns promptly without consuming/buffering any event; - dropped admission sender resolves promptly; - FIM stream returns headers through the admission handshake; - accepted zero-visible-output stream returns headers and terminates with [DONE]; - native rejection test confirms the admission callback is not fired before a KV refusal; - native CUDA accepted chat stream: HTTP 200 headers in 0.026 s, first/final output completed in 24.760 s, preserving role → content → finish → [DONE] order; - native CUDA streaming overload: HTTP 429 in 0.089 s, Retry-After: 1, stable body, rejections_total 0 → 1, requests_waiting=0, /metrics responsive in 131 ms; - server and native-engine -D warnings clippy gates pass.

Continuous-row admission correction

After rebasing onto current main (0f0b6c0e), continuous batching now distinguishes queue insertion, physical row assignment, and token output:

• ContinuousBatchManager emits handle-specific Assigned/Rejected admission notifications.
• The driver acknowledges SSE only after successful assign_row, before backend prefill/token generation.
• Per-handle row failures are returned before headers and remain HTTP 500, never KV-memory 429.
• Pending canceled handles are removed without leaking a route or permit.

Final validation:

• cargo fmt --all -- --check ✅
• server all-target clippy with -D warnings ✅
• native-engine all-target clippy with native-backend and -D warnings ✅
• 4 manager admission tests, 17 server admission tests, 5 streaming admission tests, and 4 scheduled continuous-batch integration tests ✅
• focused FIM pre-header handshake test ✅ (the full server lib run also has the already documented missing vision.onnx fixture; its 100 ms FIM timing assertion passed when run without full-suite contention)
• rebuilt native CUDA CLI on current main ✅
• strict governed streaming E2E (--vram-limit 1000000000): accepted stream HTTP 200 headers in 0.017 s, completion in 9.373 s; 16 concurrent over-budget streams returned 16×429 in 0.066–0.279 s, each with Retry-After: 1 and the stable body; rejections_total 0→16, requests_waiting peak 0, /metrics max latency 0.033 s.

The 6 GB Qwen evidence above remains the requested governed-path check; the stricter limit makes the refusal predicate independently falsifiable without relying on concurrent reservation timing. Full #738 remains open.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.23%. Comparing base (0f0b6c0) to head (4cd720c).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #743      +/-   ##
==========================================
- Coverage   80.93%   80.23%   -0.71%     
==========================================
  Files         331      331              
  Lines      134104   134104              
  Branches   134104   134104              
==========================================
- Hits       108543   107599     -944     
- Misses      21298    22246     +948     
+ Partials     4263     4259       -4     

Flag	Coverage Δ
cli-ort-linux	83.37% <ø> (ø)
cli-ort-windows	82.96% <ø> (ø)
mlas	80.96% <ø> (ø)
offline	80.09% <ø> (-0.75%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 6 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	add/large_f32_threads=1-internal/4194304	39.39 ms	43.77 ms	+11.1%
✅	sampling_latency/top_p_per_token	357.85 µs	366.53 µs	+2.4%
✅	qwen3_sampling_processors/top_k_partial_selection	133.10 µs	135.62 µs	+1.9%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	72.63 µs	73.87 µs	+1.7%
✅	sampling_latency/min_p_per_token	193.78 µs	196.19 µs	+1.2%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	1.99 ms	2.01 ms	+1.0%
✅	gather/small_bf16_threads=1-internal/4096	438.7 ns	441.5 ns	+0.6%
✅	reduce_mean/medium_f32_threads=1-internal/65536	232.99 µs	234.23 µs	+0.5%
✅	sampling_latency/greedy_per_token	2.99 µs	2.99 µs	-0.2%
✅	sampling_latency/top_k_per_token	49.85 µs	49.53 µs	-0.6%
✅	gather/small_f16_threads=1-internal/4096	434.0 ns	430.8 ns	-0.7%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.30 ms	3.26 ms	-1.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	490.73 µs	481.74 µs	-1.8%
✅	reduce_mean/large_f32_threads=1-internal/262144	939.40 µs	919.54 µs	-2.1%
✅	add/medium_f32_threads=1-internal/262144	2.78 ms	2.71 ms	-2.4%
✅	gather/medium_f32_threads=1-internal/32768	3.53 µs	3.43 µs	-2.7%
✅	add/small_bf16_threads=1-internal/1024	13.90 µs	13.50 µs	-2.8%
✅	gather/small_f32_threads=1-internal/4096	620.8 ns	597.7 ns	-3.7%
✅	add/medium_bf16_threads=1-internal/262144	2.58 ms	2.48 ms	-3.8%
✅	add/medium_f16_threads=1-internal/262144	2.79 ms	2.65 ms	-5.3%
✅	add/large_f16_threads=1-internal/4194304	43.20 ms	40.37 ms	-6.5%
✅	qwen3_sampling_processors/top_k_top_p_fast	663.03 µs	616.76 µs	-7.0%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.00 ms	1.86 ms	-7.0%
✅	add/small_f16_threads=1-internal/1024	14.66 µs	13.37 µs	-8.8%
✅	reduce_mean/small_f32_threads=1-internal/4096	15.28 µs	13.84 µs	-9.4%
✅	matmul/medium_generic_f16_threads=1/32x512x512	30.36 µs	27.47 µs	-9.5%
✅	add/large_bf16_threads=1-internal/4194304	43.20 ms	38.87 ms	-10.0%
✅	tokenization/decode_tokens_per_second	6.37 ms	5.69 ms	-10.6%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	5.94 ms	5.30 ms	-10.8%
✅	logit_processing/seven_processor_chain_per_step	337.90 µs	300.45 µs	-11.1%
✅	grammar_masking/llguidance_compute_mask/32	80.96 µs	71.37 µs	-11.9%
✅	kv_cache/alloc_dealloc_pages	39.15 µs	34.28 µs	-12.4%
✅	matmul/small_generic_f32_threads=1/1x256x256	42.54 µs	37.04 µs	-12.9%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	94.62 µs	81.57 µs	-13.8%
🟢	matmul/large_generic_f32_threads=1/32x1024x1024	10.51 ms	8.91 ms	-15.2%
🟢	add/small_f32_threads=1-internal/1024	261.2 ns	221.5 ns	-15.2%
🟢	gather/medium_bf16_threads=1-internal/32768	2.59 µs	2.18 µs	-15.8%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	87.73 µs	73.23 µs	-16.5%
🟢	matmul/small_generic_bf16_threads=1/1x256x256	35.66 µs	29.55 µs	-17.1%
🟢	matmul/medium_generic_f32_threads=1/32x512x512	2.63 ms	2.17 ms	-17.5%
🟢	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	709.70 µs	579.24 µs	-18.4%
🟢	tokenization/encode_tokens_per_second	434.73 µs	346.93 µs	-20.2%
🟢	matmul/small_generic_f16_threads=8/1x256x256	36.70 µs	29.04 µs	-20.9%
🟢	matmul/small_generic_bf16_threads=8/1x256x256	39.62 µs	30.75 µs	-22.4%
🟢	matmul/medium_generic_bf16_threads=1/32x512x512	651.86 µs	500.25 µs	-23.3%
🟢	matmul/small_generic_f16_threads=1/1x256x256	37.02 µs	28.34 µs	-23.4%
🟢	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	491.85 µs	365.61 µs	-25.7%
🟢	gather/medium_f16_threads=1-internal/32768	3.12 µs	2.18 µs	-30.0%
🟢	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	60.61 µs	41.09 µs	-32.2%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	5.37 ms	3.64 ms	-32.2%
🟢	gather/large_f16_threads=1-internal/131072	14.75 µs	9.67 µs	-34.5%
🟢	gather/large_f32_threads=1-internal/131072	36.42 µs	23.76 µs	-34.8%
🟢	matmul/large_generic_bf16_threads=8/32x1024x1024	2.05 ms	1.32 ms	-35.6%
🟢	matmul/small_generic_f32_threads=8/1x256x256	52.38 µs	32.39 µs	-38.2%
🟢	matmul/medium_generic_f32_threads=8/32x512x512	1.48 ms	893.86 µs	-39.5%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	47.12 µs	28.05 µs	-40.5%
🟢	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	73.51 µs	43.63 µs	-40.6%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	705.69 µs	378.32 µs	-46.4%
🟢	gather/large_bf16_threads=1-internal/131072	18.24 µs	9.61 µs	-47.3%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 2.48 3.05 4.29 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[justinchuby]

Final review passed after four rounds.

Contract now enforced

Streaming/non-streaming admission separates three events:

1. scheduler/governor KV byte grant;
2. continuous-batch decode-row assignment;
3. token/output production.

HTTP/SSE headers follow (1)+(2), never wait for (3), and never acknowledge mere insertion into an unbounded queue.

• memory admission refusal → pre-header HTTP 429, Retry-After: 1, stable resource_limit_error body;
• unrelated generation/row errors remain 500;
• detailed scheduler internals are server logs only, not public response;
• audio transcription uses the same typed mapping;
• cancellation while pending removes the request and resolves the handshake without leaking a row;
• errors after accepted streaming remain stream errors, not false rejections.

Measured E2E

Governed native CUDA server, qwen2.5-14b:

• accepted streaming headers: 17 ms, before first token/output;
• 16 concurrent over-budget requests: 16/16 HTTP 429;
• rejections_total delta: 16, exactly once per refusal;
• requests_waiting peak/final: 0/0;
• metrics remained responsive.

Review bugs closed

1. Streaming originally returned HTTP 200 before admission and emitted refusal as SSE error.
2. Audio kept admission as 500.
3. Public body exposed IDs/limits/internal scheduler text.
4. First fix waited for first token/result rather than admission, blocking FIM/zero-output streams.
5. Continuous batching first acknowledged queue insertion rather than actual decode-row assignment.

All are covered by targeted tests and final review found no significant issue.

This is the first increment of #738, not full preemption/swap. It closes the unbounded-queue behaviour measured in #706 without inventing a private budget or magic concurrency cap.

Merging.