fix(test): bfs_latency_v3 smoke flake — bump iterations + decouple from gate verdict (#792)

[robotrocketscience]

Closes #792

Two atomic commits unblock the pytest (3.13) smoke flake that's been silently blocking PRs (most recently #790).

Root cause

test_main_end_to_end_smoke calls main([... "--queries", "4", "--iterations", "2", "--warmup", "1", ...]) → 8 timed samples per arm. _percentile uses nearest-rank semantics; with n=8 the p95 index resolves to ceil(0.95*8) = 8 = max sample. So p95 = p99 = max_sample, and a single noisy timer reading on a shared CI runner trips the delta_p95 ≤ 50ms gate. The test then trips on assert rc == 0.

The file docstring already declares the intent:

"These tests exercise the corpus generator, query generator, percentile helper, and end-to-end CLI flow on a tiny (40 beliefs / 4 topics) corpus. Latency numbers are not asserted — only schema, determinism, and gate-result shape."

But assert rc == 0 is precisely a latency-numbers assertion. Defence in depth fixes both halves.

Commit 1 — bump iterations 2→10, warmup 1→2

40 samples per arm → p95 = sample[37], no longer max. Local pytest-3.13 runtime: 0.79s (no measurable regression; the earlier 16.7s was a cold-venv build, not the test itself).

Commit 2 — decouple smoke from gate verdict

• assert rc == 0 → assert rc in (0, 1) (both = harness ran end-to-end and emitted JSON).
• Add shape assertions on payload["gate"]: every key is present, numeric values are numeric, *_pass keys are bool.
• Cross-check (rc == 0) is gate["passed"] so a flake can't decouple the two without something more interesting going wrong.

Dedicated test_evaluate_gate_* fixtures above already cover gate logic deterministically with hand-rolled ArmResult objects — gate verdict is not the smoke's job.

Verification

• uv run --python 3.13 pytest tests/test_bfs_latency_v3.py -q → 11 passed in 1.10s.
• Both commits SSH-signed, no Co-Authored-By lines per repo convention.
• Diff: 1 file, +23/-6.
• Discretion grep on diff vs github/main: CLEAN.

Out of scope

• Production-bench gate thresholds (GATE_DELTA_P50_MS, GATE_DELTA_P95_MS, GATE_MAX_OVER_MEDIAN_RATIO) — those are the real benchmark's contract for ≥1000-sample runs.
• _percentile semantics — nearest-rank is correct at production sample sizes.

[coderabbitai]

Warning

Rate limit exceeded

@robotrocketscience has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 3 minutes and 52 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5b646012-1ed9-494c-9933-2d95374c419d

📥 Commits

Reviewing files that changed from the base of the PR and between eff8a1c and b2287ef.

📒 Files selected for processing (1)

• tests/test_bfs_latency_v3.py

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/issue-792-bfs-latency-smoke-flake

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[sourcery-ai]

Sorry @robotrocketscience, you have reached your weekly rate limit of 500000 diff characters.

Please try again later or upgrade to continue using Sourcery

[robotrocketscience]

[claim:review:jojo:2026-05-14T16:15:44Z]

[robotrocketscience]

Review — approving

Targeted defence-in-depth on the bench-smoke flake. Both halves are necessary and sufficient:

• Iterations 2→10 / warmup 1→2 ⇒ 40 samples per arm. Nearest-rank p95 index resolves to ceil(0.95*40) = 38, so p95 is no longer max-of-sample. Doesn't eliminate runner noise, but breaks the "one slow sample dominates" mode.
• assert rc == 0 → assert rc in (0, 1) restores the smoke's stated contract (file docstring: "Latency numbers are not asserted — only schema, determinism, and gate-result shape"). Gate-verdict assertions move to the dedicated test_evaluate_gate_* fixtures above, which use hand-rolled ArmResult objects and are timing-independent.

The (rc == 0) is gate["passed"] invariant is the right load-bearing check — it pins rc/payload consistency without binding either to a specific value, so the smoke catches a real bug (rc and JSON drift apart) while tolerating runner noise.

Two atomic commits, both SSH-signed; +23/-6 in one file; discretion grep on rebased diff clean; CI all green on the rebased SHAs (f7bcf2b3 / b2287ef6). Rebased onto current main (post-#790) because the original tip was behind 81b458a.

Minor nit (not blocking): the new shape assertion enumerates delta_p50_ms / delta_p95_ms / tail_ratio / *_pass / passed but doesn't cover anything else the gate dict might grow later. The schema is currently complete, so this is forward-only.

Labeling ready-to-merge.

[github-actions]

merge-train: merged b2287ef → main via FF push.

[robotrocketscience]

[release:review:jojo:2026-05-14T16:21:03Z]