[justinchuby]

"cli的stats应该默认要打开" — @justinchuby

The gap

The REPL has shown stats by default on a TTY since #289. One-shot generate did not — it emitted nothing unless --profile was passed explicitly. So the command being used for native-vs-ORT benchmarking printed no timing, no token counts, and no context usage unless you already knew to ask for it.

Wrong default for a tool whose charter (#271) is explicitly a developer instrument rather than a consumer product.

Change

Text generate now defaults to the compact stats line on a TTY.

Reuses rather than reimplements, which was the main constraint:

• the REPL's repl_input_mode / initial_repl_show_stats for TTY detection — a second TTY check that could disagree with the first is the failure mode here
• the shared emit_stats_line — one format, not two

generate --no-stats opts out, matching run --no-stats. --profile suppresses the compact line so the two cannot double-print.

The stdout/stderr contract

Stats go to stderr; generated text stays on stdout. That is what keeps piped generate byte-stable for the scripts and e2e tests that consume it — a stats line on stdout would break every script capturing model output.

Covered by a test asserting piped stdout is byte-identical. This is not theoretical: the same class of regression broke #289 on its first pass, and a flaky test that compared merged streams had to be fixed earlier today.

Image, audio, and transcribe deliberately excluded

They do not get the compact token stats. Tokens per second is meaningless for image generation, so those paths keep --profile for steps, RTF and audio metrics rather than gaining a line that reads as authoritative and is not.

Stating this explicitly so it reads as a decision rather than an oversight.

Gates

cargo build / test --lib / fmt --check / clippy -D warnings pass. Model-loading e2e could not run locally (ORT API mismatch 27 vs 17 — #315 addresses that); the cli-ort CI lane covers Linux and Windows.

Part of the CLI improvement track — see #271.

[codecov]

Codecov Report

❌ Patch coverage is 97.59036% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 80.28%. Comparing base (41c14e6) to head (c9dee17).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
crates/onnx-genai-cli/src/generate.rs	98.33%	0 Missing and 1 partial ⚠️
crates/onnx-genai-cli/src/lib.rs	92.85%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #325      +/-   ##
==========================================
- Coverage   81.04%   80.28%   -0.76%     
==========================================
  Files         306      306              
  Lines      116658   116736      +78     
  Branches   116658   116736      +78     
==========================================
- Hits        94544    93724     -820     
- Misses      18200    19103     +903     
+ Partials     3914     3909       -5     

Flag	Coverage Δ
cli-ort-linux	81.96% <97.59%> (+0.31%)	⬆️
cli-ort-windows	81.34% <97.59%> (+0.32%)	⬆️
mlas	77.91% <ø> (-0.82%)	⬇️
offline	80.25% <ø> (-0.80%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
crates/onnx-genai-cli/src/output.rs	69.69% <100.00%> (+1.54%)	⬆️
crates/onnx-genai-cli/src/generate.rs	81.77% <98.33%> (+5.66%)	⬆️
crates/onnx-genai-cli/src/lib.rs	94.32% <92.85%> (-0.03%)	⬇️

... and 7 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[justinchuby]

我说错了，repl有stats就行了。oneshot不需要默认展示

[github-actions]

✅ Benchmarks — No Regression

Comparison of criterion micro-benchmarks: PR head vs merge-base, measured on the same runner in the same job (base first → PR second).

ℹ️ Absolute times are informational only — they vary with runner load. The % change column is the reliable signal because both sides ran under identical conditions.

Status	Scenario	Base	PR	Change
✅	matmul/large_generic_f16_threads=1/32x1024x1024	75.95 µs	78.62 µs	+3.5%
✅	grammar_masking/llguidance_compute_mask/32	69.45 µs	71.00 µs	+2.2%
✅	kv_cache/alloc_dealloc_pages	35.32 µs	35.38 µs	+0.2%
✅	sampling_latency/greedy_per_token	3.01 µs	3.01 µs	+0.1%
✅	tokenization/decode_tokens_per_second	5.85 ms	5.81 ms	-0.6%
✅	sampling_latency/top_p_per_token	917.03 µs	910.47 µs	-0.7%
✅	logit_processing/seven_processor_chain_per_step	1.09 ms	1.09 ms	-0.8%
✅	add/medium_f32_threads=1-internal/262144	2.57 ms	2.54 ms	-1.0%
✅	sampling_latency/min_p_per_token	317.22 µs	313.77 µs	-1.1%
✅	sampling_latency/top_k_per_token	440.96 µs	435.32 µs	-1.3%
✅	gather/medium_f32_threads=1-internal/32768	3.67 µs	3.57 µs	-2.7%
✅	tokenization/encode_tokens_per_second	362.61 µs	348.89 µs	-3.8%
✅	matmul/large_generic_f16_threads=8/32x1024x1024	82.00 µs	78.84 µs	-3.8%
✅	matmul/large_generic_f32_threads=1/32x1024x1024	9.08 ms	8.61 ms	-5.2%
✅	matmul/medium_generic_f32_threads=1/32x512x512	2.29 ms	2.16 ms	-5.5%
✅	gather/small_f32_threads=1-internal/4096	699.9 ns	660.2 ns	-5.7%
✅	matmul/small_generic_bf16_threads=8/1x256x256	31.39 µs	29.58 µs	-5.8%
✅	matmul/medium_generic_f16_threads=1/32x512x512	30.12 µs	28.35 µs	-5.9%
✅	reduce_mean/large_f32_threads=1-internal/262144	1.10 ms	1.03 ms	-6.9%
✅	gather/small_bf16_threads=1-internal/4096	490.0 ns	455.5 ns	-7.0%
✅	matmul/large_generic_bf16_threads=8/32x1024x1024	1.34 ms	1.24 ms	-7.7%
✅	matmul/small_generic_f16_threads=8/1x256x256	31.05 µs	28.48 µs	-8.3%
✅	gather/medium_bf16_threads=1-internal/32768	2.57 µs	2.35 µs	-8.5%
✅	matmul/medium_generic_f32_threads=8/32x512x512	1.02 ms	937.94 µs	-8.5%
✅	gather/small_f16_threads=1-internal/4096	518.2 ns	471.6 ns	-9.0%
✅	gather/large_bf16_threads=1-internal/131072	13.64 µs	12.37 µs	-9.4%
✅	matmul/medium_generic_bf16_threads=1/32x512x512	535.22 µs	485.08 µs	-9.4%
✅	matmul/small_generic_bf16_threads=1/1x256x256	32.04 µs	28.82 µs	-10.1%
✅	add/large_f32_threads=1-internal/4194304	42.00 ms	37.50 ms	-10.7%
✅	matmul/small_generic_f32_threads=1/1x256x256	39.63 µs	35.02 µs	-11.6%
✅	matmul/large_generic_f32_threads=8/32x1024x1024	3.99 ms	3.53 ms	-11.6%
✅	matmul/large_generic_bf16_threads=1/32x1024x1024	2.07 ms	1.82 ms	-12.1%
✅	add/medium_bf16_threads=1-internal/262144	2.84 ms	2.43 ms	-14.6%
✅	reduce_mean/medium_f32_threads=1-internal/65536	279.33 µs	238.36 µs	-14.7%
🟢	add/large_bf16_threads=1-internal/4194304	47.82 ms	40.58 ms	-15.1%
🟢	gather/medium_f16_threads=1-internal/32768	2.73 µs	2.31 µs	-15.6%
🟢	matmul/small_generic_f16_threads=1/1x256x256	33.14 µs	27.62 µs	-16.6%
🟢	matmul/medium_generic_f16_threads=8/32x512x512	33.50 µs	27.86 µs	-16.8%
🟢	gather/large_f16_threads=1-internal/131072	11.92 µs	9.87 µs	-17.2%
🟢	add/small_bf16_threads=1-internal/1024	14.41 µs	11.85 µs	-17.8%
🟢	gather/large_f32_threads=1-internal/131072	31.77 µs	25.58 µs	-19.5%
🟢	add/large_f16_threads=1-internal/4194304	48.46 ms	38.76 ms	-20.0%
🟢	add/small_f16_threads=1-internal/1024	15.56 µs	12.11 µs	-22.2%
🟢	matmul/medium_generic_bf16_threads=8/32x512x512	451.80 µs	350.54 µs	-22.4%
🟢	add/medium_f16_threads=1-internal/262144	3.17 ms	2.38 ms	-24.9%
🟢	add/small_f32_threads=1-internal/1024	265.4 ns	186.3 ns	-29.8%
🟢	reduce_mean/small_f32_threads=1-internal/4096	20.36 µs	14.27 µs	-29.9%
🟢	matmul/small_generic_f32_threads=8/1x256x256	47.23 µs	32.37 µs	-31.5%

Visual flags: ⚠️ ≥ 15% slower, 🔴 ≥ 30% slower — calibrated against measured runner noise (~27% worst-case on multi-threaded matmul)

Host info

CPU: Apple M1 (Virtual)
Cores: 3
OS: Darwin 25.4.0 arm64
Rust: rustc 1.97.1 (8bab26f4f 2026-07-14)
Load avg: { 3.07 3.40 6.27 }

What this cannot catch

• Regressions in code paths not covered by these benchmarks (e.g., end-to-end decode with a real model)
• Sub-threshold regressions that compound over multiple PRs
• Performance changes that only manifest under GPU execution
• Latency changes in the ORT integration path (these benchmarks exercise the native Rust kernels)

[Copilot]

Pull request overview

This PR updates the onnx-genai CLI so generate (text mode) shows the same compact per-turn stats line by default on an interactive TTY, matching the REPL behavior, while preserving the stdout/stderr contract for scriptable usage.

Changes:

• Add generate --no-stats opt-out and default-on compact stats for TTY text generation (stats on stderr, generated text on stdout).
• Reuse the existing REPL TTY detection (repl_input_mode / initial_repl_show_stats) and the shared stats emitter (emit_stats_line) to avoid divergent behavior/formatting.
• Add regression coverage ensuring piped generate stdout remains byte-identical with and without the default-stats behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
README.md	Documents the unified “stats on TTY, stderr-only” behavior and the piping contract.
crates/onnx-genai-cli/tests/multimodal_cli_e2e.rs	Adds an e2e regression test asserting piped generate stdout stays pure generated text.
crates/onnx-genai-cli/src/output.rs	Refactors stats emission gating into a helper and adds a small unit test.
crates/onnx-genai-cli/src/lib.rs	Introduces --no-stats for generate and adds a CLI parsing test.
crates/onnx-genai-cli/src/generate.rs	Implements TTY-gated default stats for text generate, and explicitly excludes image/audio outputs.
crates/onnx-genai-cli/README.md	Updates CLI crate docs to reflect default stats behavior and stderr routing.
.squad/agents/zhora/history.md	Records the design decision/contract for default stats in generate.