fix(bench): RFC 0031 — query ingesters regardless of range age (L3 flicker diagnosed)

[jensholdgaard]

What

The run #11/#13 L3 intermittency, diagnosed by the #488 post-mortem dump and fixed with three small harness changes:

1. -querier.query-ingesters-within=0 (third documented ingest-replay deviation, in Loki's favour). The failing query's stats showed ingester.totalReached: 0 with 2.1 GB of structured metadata scanned and zero matches: queries over the corpus's weeks-old time range skip the ingesters (default cutoff 3h), so rows still in unflushed low-volume chunks are invisible. High-volume streams (kafka) always flush fast — which is exactly why only the 9-row trace pair flickered across runs while the severity/window pairs never did. Disabling the cutoff makes Loki's answer to old-range queries complete and deterministic.
2. Poll interval 2 s → 10 s. The L3 poll costs Loki ~19 s of engine time per query; polling at 2 s queued the deadline away (queueTime: 321s in the dump).
3. OTLP partialSuccess assert on the push path. A 2xx can carry silently-rejected records; that would unequalize the corpora and surface later as a baffling equivalence failure. Now it fails at ingest, loudly, with Loki's own error message.

Sequencing

Merges ahead of the bloom PR (#489), which then rebases and re-runs the pre-merge bench (run #14) under the measure-before-merge workflow — the L3 pair must measure deterministically for the bloom's before/after to mean anything.

Invariants / hazards

Bench harness only. Anti-strawman: the new flag is ingest-replay accommodation on the query path — it lets Loki see all its own data; the stock query configuration otherwise stands.

Checks run

cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings (workspace-wide), cargo nextest run -p ourios-bench (133 passed, 13 skipped).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 24 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 995bbe88-22e9-4b5b-8da5-e76cca1adb1c

📥 Commits

Reviewing files that changed from the base of the PR and between 6e91c0e and feb2f98.

📒 Files selected for processing (1)

• crates/ourios-bench/tests/rfc0031_comparative.rs

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rfc0031-ingester-visibility

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Updates the RFC 0031 comparative benchmark harness to eliminate Loki L3 intermittency by ensuring old-range queries still consult ingesters, reducing self-induced query queueing, and failing loudly on OTLP partial-success ingestion.

Changes:

• Assert on OTLP partial_success in successful push responses to catch silently rejected log records at ingest time.
• Increase Loki measurement poll interval from 2s to 10s to avoid piling up expensive L3 queries in Loki’s queue.
• Start Loki with -querier.query-ingesters-within=0 so ingesters are queried even for weeks-old replay windows.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.