feat(server): report a turn's private thinking on its own channel

[justinchuby]

Symptom

Driving this model from OpenCode, a plain hello shows nothing at all for minutes, and no reasoning is ever displayed.

Measured against this server:

first_visible_text = 280.4s   total = 280.8s   finish = stop

280 seconds of complete silence on the wire. A client cannot tell that apart from a hung server.

Cause

#1249 stopped this package's private channel being served as an answer, which was right — but it did so by discarding it. The model addresses to=self while it works and only later opens to=user, so for the whole of the thinking there is nothing to send, and the connection simply sits there.

Thinking was never the problem. Passing it off as an answer was.

Fix

The private channel is now reported on the channel that exists for it — reasoning_content, beside content rather than inside it — on the streaming delta and on a buffered message alike. Nothing moves into content, so the answer is byte for byte what it was.

The gate keeps one transcript and asks the channel definitions what each has revealed since the last emission, so the two channels cannot disagree about where a token belongs. A model that declares no private channel reports no thinking rather than having one invented for it.

Reasoning streams even on the tool-detection path, where the answer must be buffered until the turn is known not to be a tool call — that is exactly the agent case, and exactly where the silence was worst.

Verification

Same prompt, same server, after the change:

first_reasoning = 6.9s   first_content = 7.8s   total = 8.3s   finish = stop
answer: Hello! 👋 How can I help you today?

• 220 lib + 31 http + 11 bundle tests pass; clippy --all-targets clean.
• Four new gate tests: thinking is reported as it is produced while the answer stays gated; a turn that never answers still reports what it was thinking; thinking resumed after a tool call extends rather than replaces; an unarmed gate reports none.
• End to end from OpenCode: reasoning is displayed live, the tool calls dispatch, and the answer is unchanged.

[justinchuby]

Review — verified against #1256 acceptance criteria ✅ merging

Reviewed in a dedicated worktree off freshly-fetched main, rebased PR head. Reasoning is now reported on BOTH the streamed and the buffered paths, on its own reasoning_content channel, with no compatibility regression for ordinary clients. I added two missing end-of-path test guards (pushed as 0dfe9e23) and falsified every behavioural assertion.

Wire format (the "on its own channel" claim)

Reasoning rides on a separate reasoning_content field beside content, not spliced into it — the DeepSeek/vLLM-style OpenAI-compatible convention. Both sse::Delta (streaming) and types::ChatMessage (buffered) gained reasoning_content: Option<String> with skip_serializing_if = "Option::is_none". Captured shapes:

• Streaming (armed): {"delta":{"reasoning_content":"..."}} — a distinct chunk, no content key.
• Buffered (armed): message.reasoning_content set; message.content is only the user-channel answer.
• Unarmed (plain qwen, exercised end-to-end against qwen05b-q4): the field is omitted entirely on every streamed delta and on the final message. An existing OpenAI client sees byte-for-byte the previous shape. Verified live on both stream:true and stream:false.

Criteria

1. Reasoning present on both paths — ✅. Streaming routed via PrivateChannelGate (ChannelDelta.reasoning, emitted as reasoning_chunk); buffered via atem_reasoning_content → message.reasoning_content. The buffered path had no test (the exact Making reasoning visible spans #1197, not just #1224 — the buffered path filters independently #1256 trap: it touches atem_visible_content/parse_assistant_output but nothing guarded the behaviour). I added buffered_reasoning_tests (mirrors run_chat_completion's composition) and reasoning_wire_tests (serialized SSE shape).
2. Now-wrong tests handled — ✅. The design keeps arming and redirects reasoning to its own channel rather than discarding it, so the old tests were correctly repurposed, not deleted: reasoning_never_streams/a_tool_channel_never_streams/a_self_opened_turn_still_hides_reasoning/an_armed_gate_withholds_an_unaddressed_turn/an_unspellable_token_is_withheld now assert reasoning stays out of content (the compat guarantee), paired with new tests asserting it is present on reasoning_content. registry.rs arming-detection tests remain valid because arming is still needed. No test asserts the old "reasoning discarded entirely" behaviour.
3. Client compatibility — ✅, see wire format above; confirmed live with plain qwen.
4. Budget behaviour untouched — ✅ by test run: output_budget_tests all green incl. an_unspecified_budget_is_the_server_cap (the fix(server): give a reasoning turn room to finish, and keep it private #1224 falsification anchor). Diff touches no budget code.
5. Falsification — streaming: neutralising growth() reasoning → 3 gate tests RED, restored GREEN. Buffered: neutralising atem_reasoning_content → both buffered tests RED, restored GREEN. Wire tests green/red verified against the serde attrs.

Suite / lint

• cargo test -p onnx-genai-server: 224 lib (220 + my 4) + 31 http + 11 bundle, 0 failed. (Two lib tests — production_pipeline_loads_reserve_all_components_on_shared_ledger, fim_stream_returns_headers_before_generation_finishes — flaked under parallel host load and pass isolated; unrelated to this change.)
• cargo clippy -p onnx-genai-server --all-targets -- -D warnings: clean.
• The 4 files under onnx-runtime-ep-cuda/onnx-runtime-session are pure rustfmt churn (line-wrapping + import ordering) from style: apply rustfmt across the tree #1260 — confirmed formatting-only.

Honest limitation

No model available on this box declares a private channel (qwen05b-q4 has no response_template.reasoning_content.open_pattern), so the armed reasoning path could not be exercised over a real wire. It is covered by the unit + falsification evidence above and the captured serialized shapes; the live end-to-end run validated the unarmed/no-regression case.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 80.06%. Comparing base (7d669c7) to head (0dfe9e2).
⚠️ Report is 26 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1257      +/-   ##
==========================================
+ Coverage   79.69%   80.06%   +0.36%     
==========================================
  Files         358      363       +5     
  Lines      155271   160240    +4969     
  Branches   155271   160240    +4969     
==========================================
+ Hits       123746   128292    +4546     
- Misses      26940    27293     +353     
- Partials     4585     4655      +70     

Flag	Coverage Δ
mlas	85.19% <ø> (?)
offline	79.96% <ø> (+0.26%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.
see 28 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

🔴 Benchmark Regression Detected

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
🔴	matmul/medium_generic_bf16_threads=8/32x512x512	390.44 µs	841.37 µs	+115.5%
🔴	gather/large_bf16_threads=1-internal/131072	13.14 µs	26.86 µs	+104.4%
🔴	matmul/small_generic_f32_threads=1/1x256x256	37.16 µs	50.46 µs	+35.8%
🔴	gather/medium_f16_threads=1-internal/32768	2.39 µs	3.20 µs	+33.5%
⚠️	block_quantized_matmul_cached_dense/mxfp4_preexpanded_dense_oncelock_like_proxy/1x1024x1024	68.56 µs	81.35 µs	+18.7%
⚠️	gather/large_f16_threads=1-internal/131072	15.58 µs	18.25 µs	+17.2%
✅	gather/large_f32_threads=1-internal/131072	32.26 µs	36.15 µs	+12.1%
✅	matmul/small_generic_bf16_threads=8/1x256x256	36.00 µs	39.78 µs	+10.5%
✅	gather/medium_f32_threads=1-internal/32768	4.92 µs	5.22 µs	+6.1%
✅	kv_cache/alloc_dealloc_pages	40.20 µs	42.41 µs	+5.5%
✅	qwen3_sampling_processors/top_p_full_sort_after_top_k_baseline	3.88 ms	4.07 ms	+4.9%
✅	grammar_masking/llguidance_compute_mask/32	77.55 µs	81.29 µs	+4.8%
✅	add/small_f16_threads=1-internal/1024	514.9 ns	537.6 ns	+4.4%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.28 ms	1.33 ms	+4.1%
✅	add/medium_f16_threads=1-internal/262144	111.46 µs	115.36 µs	+3.5%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	657.34 µs	676.77 µs	+3.0%
✅	gather/small_f16_threads=1-internal/4096	528.0 ns	543.1 ns	+2.9%
✅	logit_processing/seven_processor_chain_per_step	330.83 µs	337.69 µs	+2.1%
✅	add/medium_f32_threads=1-internal/262144	28.55 µs	29.14 µs	+2.1%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.67 ms	9.87 ms	+2.0%
✅	qwen3_sampling_processors/top_p_fast_after_top_k	538.64 µs	545.63 µs	+1.3%
✅	qwen3_sampling_processors/top_k_top_p_fast	693.09 µs	700.76 µs	+1.1%
✅	qwen3_sampling_processors/top_k_full_sort_baseline	2.25 ms	2.26 ms	+0.4%
✅	gather/medium_bf16_threads=1-internal/32768	2.56 µs	2.57 µs	+0.2%
✅	matmul/small_generic_bf16_threads=1/1x256x256	36.81 µs	36.56 µs	-0.7%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.55 ms	2.51 ms	-1.8%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.56 ms	1.53 ms	-2.0%
✅	qwen3_sampling_processors/top_k_top_p_full_sort_baseline	6.09 ms	5.97 ms	-2.0%
✅	sampling_latency/min_p_per_token	236.22 µs	228.46 µs	-3.3%
✅	block_quantized_matmul_cached_dense/mxfp4_cached_dense_repeated_call/1x1024x1024	75.43 µs	72.28 µs	-4.2%
✅	add/small_bf16_threads=1-internal/1024	492.0 ns	466.3 ns	-5.2%
✅	add/medium_bf16_threads=1-internal/262144	126.17 µs	118.49 µs	-6.1%
✅	block_quantized_moe_cached_dense/mxfp4_cached_dense_expert_repeated_call/rows=1,H=256,I=256,E=4,top_k=1	88.58 µs	83.10 µs	-6.2%
✅	add/small_f32_threads=1-internal/1024	268.3 ns	251.6 ns	-6.2%
✅	block_quantized_matmul_cached_dense/mxfp4_uncached_dequant_each_call/1x1024x1024	879.18 µs	822.39 µs	-6.5%
✅	matmul/medium_generic_f16_threads=1/32x512x512	34.99 µs	32.41 µs	-7.4%
✅	block_quantized_moe_cached_dense/mxfp4_uncached_expert_dequant_each_call/rows=1,H=256,I=256,E=4,top_k=1	482.41 µs	445.78 µs	-7.6%
✅	sampling_latency/top_k_per_token	56.50 µs	51.82 µs	-8.3%
✅	gather/small_f32_threads=1-internal/4096	798.0 ns	730.7 ns	-8.4%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.16 ms	1.95 ms	-9.7%
✅	tokenization/decode_tokens_per_second	6.81 ms	6.13 ms	-9.9%
✅	sampling_latency/top_p_per_token	428.27 µs	385.11 µs	-10.1%
✅	sampling_latency/greedy_per_token	3.40 µs	2.94 µs	-13.5%
✅	qwen3_sampling_processors/top_k_partial_selection	186.08 µs	159.12 µs	-14.5%
✅	matmul/small_generic_f16_threads=8/1x256x256	53.65 µs	45.71 µs	-14.8%
🟢	matmul/small_generic_f16_threads=1/1x256x256	40.38 µs	34.25 µs	-15.2%
🟢	gather/small_bf16_threads=1-internal/4096	661.1 ns	555.7 ns	-15.9%
🟢	add/large_f16_threads=1-internal/4194304	2.21 ms	1.81 ms	-18.3%
🟢	matmul/large_generic_f16_threads=1/32x1024x1024	110.85 µs	87.01 µs	-21.5%
🟢	matmul/small_generic_f32_threads=8/1x256x256	58.90 µs	45.87 µs	-22.1%
🟢	reduce_mean/large_f32_threads=1-internal/262144	1.36 ms	1.02 ms	-25.6%
🟢	add/large_bf16_threads=1-internal/4194304	2.36 ms	1.75 ms	-25.8%
🟢	reduce_mean/small_f32_threads=1-internal/4096	22.21 µs	15.92 µs	-28.3%
🟢	matmul/large_generic_f32_threads=8/32x1024x1024	6.04 ms	4.27 ms	-29.4%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	50.76 µs	35.80 µs	-29.5%
🟢	matmul/large_generic_f16_threads=8/32x1024x1024	129.32 µs	90.35 µs	-30.1%
🟢	add/large_f32_threads=1-internal/4194304	1.06 ms	717.35 µs	-32.0%
🟢	tokenization/encode_tokens_per_second	546.96 µs	363.74 µs	-33.5%
🟢	reduce_mean/medium_f32_threads=1-internal/65536	407.67 µs	247.69 µs	-39.2%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.5.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 5.36 3.82 6.19 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)