docs(bench): RFC 0031 §9.13 comparative entry draft (runs #8–#18) — maintainer-gated fold-in

[jensholdgaard]

What

The docs/benchmarks.md §9.13 draft: the RFC 0031 comparative program's honest record, compiling runs #8–#18 (incl. the run #18 latency channel) on otel-demo-v8 against digest-pinned grafana/loki 3.5.3.

Maintainer-gated fold-in. Recording comparative numbers in §9 has been reserved as a maintainer step throughout the program — this PR is the prepared draft, reviewed for correctness, held for your merge (or edits) rather than auto-merged on the clean gate.

What the entry carries

• Purpose framing: §7 calibration inputs — margins are provisional defaults, every pass/fail "reported, not asserted" (the §5 gate scenarios remain red stubs).
• Run-provenance table (feat(miner): implement tokenize() — RFC0001.3 (first red → green step) #8–feat(miner): add Drain prefix tree skeleton (RFC 0001 §6.2 step 3) #17 with workflow ids and the single harness delta each run tested); integrity preconditions (per-pair multiset equivalence at 4.9M records in every counted run; OTLP partialSuccess asserted).
• The §3.6 metric amendment (total bytes = count + materialize + registry) and the explicit retirement of runs docs(rfc-0001): fill in drafted-bar content for the template miner #5–chore(miner): add ourios-miner crate with §5 acceptance stubs (specified → red) #7's biased count-scan-only figures as not citable.
• Per-class results: L1 77.2×/1,816.5× PASS, L3 21.9×/514.6× PASS (with the no-bloom 1.41× before-picture and the RFC 0005 §3.6 amendment linked), L2 0.67×→1.08× storage ledger (explicitly not a 10× pass; named levers: RFC 0033 registry constant = 20–29% of small-answer queries, write-side sizing), L6 published loss (0.003–0.018 vs floor 0.33) with run feat(miner): add Drain prefix tree skeleton (RFC 0001 §6.2 step 3) #17's diagnostic: resource-scoping −22% and processed-floor flip, no bloom collapse (one row group per hour holds all services — write-side layout is the tier lever, hazard docs: apply RFC maturity-model amendments #4, RFC-level).
• Ourios determinism vs Loki's 2.67–3.35 MB run wobble; all four documented in-Loki's-favour deviations quoted with flag names; the §7 freeze inputs listed as open maintainer decisions.

Checks run

mdbook build (clean; §9.13 anchor renders), no code touched. Arithmetic cross-checks close exactly (component sums and quoted ratios reproduce from raw bytes).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y

Summary by CodeRabbit

• Documentation
  • Added benchmark results for RFC 0031 comparing Ourios with Grafana Loki using the otel-demo-v8 capture.
  • Published provisional calibration outcomes for template-exact lookup (L1), trace correlation (L3), severity predicate (L2), and time-window browsing (L6), with linked harness run IDs.
  • Included the shared test setup (including failure-mode flags), determinism notes, and documented flag deviations (all favoring Loki).
  • Clarified these are calibration inputs pending final scenario/margin and must-win decisions.

[coderabbitai]

📝 Walkthrough

Walkthrough

Adds Section 9.13 to document provisional RFC 0031 comparisons between Ourios and Grafana Loki on the otel-demo-v8 corpus, including methodology, run identifiers, L1–L3 and L6 outcomes, determinism observations, configuration deviations, and deferred calibration decisions.

Changes

RFC 0031 benchmark documentation

Layer / File(s)	Summary
Document calibration results docs/benchmarks.md	Adds Section 9.13 with the reference setup, equivalence checks, harness run identifiers, provisional L1, L2, L3, and L6 results, determinism notes, Loki configuration deviations, and deferred maintainer decisions.

Estimated code review effort: 1 (Trivial) | ~5 minutes

Possibly related PRs

• jensholdgaard/ourios#464: Introduces the RFC 0031 comparative benchmarking methodology documented here.
• jensholdgaard/ourios#479: Adds storage-side fetched-bytes reporting reflected in these results.
• jensholdgaard/ourios#480: Adds related RFC 0031 harness reporting for the L2 and L6 outcomes.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly describes the RFC 0031 §9.13 benchmark draft and maintainer-gated merge intent.
Description check	✅ Passed	The description covers the main summary, RFC context, and checks run, but it does not follow the template's explicit Related and Checklist sections.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rfc0031-s9-draft

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Adds a new draft benchmark results entry to docs/benchmarks.md documenting RFC 0031’s comparative program runs (#8–#17) of Ourios vs Grafana Loki (otel-demo-v8, Loki 3.5.3), including metric definition notes and per-class outcome tables intended as §7 calibration inputs (not final gate verdicts).

Changes:

• Add §9.13 “Results — 2026-07-12” narrative framing + run provenance table for comparative dispatch runs.
• Document the amended §3.6 “total bytes” metric and retire earlier biased (count-scan-only) run figures as non-citable.
• Record L1/L2/L3/L6 per-class result tables and Loki configuration deviations used for replay integrity.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/benchmarks.md`:
- Around line 1370-1375: Update the “Determinism note” in docs/benchmarks.md to
scope Ourios’s byte-identical claim to repeated measurements of the same fixed
build and configuration, rather than every optimization run in the ledger.
Preserve the explanation that this enables reliable repetition comparisons,
while acknowledging that totals may differ between optimization runs such as
`#8`–#10 and the L6 `#8/`#10 entries.
- Around line 1220-1224: Update the “Reference system” section to replace the
truncated Loki image digest with the complete sha256 digest, or link directly to
the committed configuration containing the full digest. Preserve the existing
image tag, deployment mode, endpoint, and deviation details.
- Around line 1244-1251: Clarify the benchmark pass-streak accounting in the
surrounding RFC0031.1 results text: either add the omitted run `#16` to the table
with its relevant outcomes, or explicitly state that omitted no-delta runs count
toward consecutive-pass streaks. Ensure the “third consecutive” L1 and “three in
a row” L3 claims can be verified from the documented runs.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 89157814-e5d1-4850-b973-74cf97096bcc

📥 Commits

Reviewing files that changed from the base of the PR and between 844d413 and bcf860e.

📒 Files selected for processing (1)

• docs/benchmarks.md

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.