feat(querier): RFC 0033 green (freshness) — cached read path + write-through, .2/.4/.5

[jensholdgaard]

What

RFC 0033's freshness + write-through green slice — the cache enters the query path, discharging scenarios RFC0033.2, .4, .5.

• load_or_derive per §3.3: LIST → GET → frontier set-equality. One listing serves both the validity check and the fallback fold, made structural by audit_scan::ResolvedAuditSet. IO per outcome documented and tested: hit = 1 LIST + 1 artifact GET (zero audit GETs); miss/stale = the fold over the same listing + best-effort write-through publish (CAS at the observed ETag; LostRace fine; publish failure is telemetry, never a query failure). Tenant-mismatched body fails loudly, no fallback.
• Write-through abstains when the artifact wouldn't beat the audit bytes folded (§3.2) — load-bearing: empty/small-audit buckets stay artifact-free, which is exactly what keeps every pinned byte-contract green.
• One acquisition per query: the resolves_to compile path and collect_records' rendering share a single AcquiredTemplateMap — one frontier, both folds, per query.
• registry_bytes_read contract amended to template-map acquisition bytes (audit fold on miss — byte-for-byte the pre-0033 figure — artifact GET on hit), with the RFC 0031 §3.6 dated amendment block.
• Scenarios: .2 staleness probed through the query answer (an alias assertion flips resolves_to row counts; republish frontier asserted; removal direction proves set-equality-not-subset); .4 advisory (no-artifact acquisition == summed audit bytes on disk, exact; deletion mid-flight costs only re-derivation; audit walk blind to the artifact); .5 tenant isolation end-to-end (planted foreign artifact fails the victim's query loudly, the other tenant unaffected).

Pinned contracts proved undisturbed

rfc0017_6 (limited.stats == counted.stats), the RFC 0031 single-pass byte-total tests, and the bench comparative fixture tests — all unmodified, all green (330/330 across querier+bench, re-verified after rebasing over the artifact PR's hardening rounds and the publish merge).

Remaining

.6 (comparative cold-vs-warm — the dispatch measurement that unfreezes M_L2) and .7 (observability) — next slices.

Checks run

cargo fmt --all --check, cargo clippy --all-targets --all-features -- -D warnings (workspace-wide), strict rustdoc (-D private-intra-doc-links, clean), cargo nextest run -p ourios-querier -p ourios-bench (330 passed).

🤖 Generated with Claude Code

https://claude.ai/code/session_01WQY9wfrfRggqSpMLH8Xj3Y

Summary by CodeRabbit

• Performance

  • Queries reuse a single acquired template map to avoid redundant registry and audit derivation.
  • Cache freshness is validated automatically; stale or missing artifacts are refreshed before results are returned.
  • Reported registry_bytes_read now reflects whether registry data came from an artifact GET (cache hit) or audit-based derivation (cache miss).

• Reliability

  • Improved audit-file set and frontier resolution for both local and remote storage.
  • Stronger tenant isolation to prevent using mismatched cached artifacts.

• Documentation

  • Clarified cached template-map behavior and the meaning/source of registry-byte accounting.

• Tests

  • Expanded end-to-end cache scenario coverage, including miss, refresh, hit, and tenant isolation.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 15ab3eaf-6114-46b0-90df-ae6299126cb3

📥 Commits

Reviewing files that changed from the base of the PR and between f886a0d and 1037466.

📒 Files selected for processing (2)

• crates/ourios-querier/src/template_map.rs
• crates/ourios-querier/tests/it/rfc0033_cached_template_map.rs

🚧 Files skipped from review as they are similar to previous changes (2)

• crates/ourios-querier/tests/it/rfc0033_cached_template_map.rs
• crates/ourios-querier/src/template_map.rs

---

📝 Walkthrough

Walkthrough

The querier now resolves audit frontiers once, classifies and reuses cached template maps, derives aliases and registries from a shared scan on misses, threads acquired maps through query rendering, and enables RFC0033 tests for staleness, byte accounting, and tenant isolation.

Changes

Cached template-map query flow

Layer / File(s)	Summary
Audit set resolution and captured reads crates/ourios-querier/src/audit_scan.rs	Local and remote audit sets resolve into backend-specific paths or keys with sorted tenant-relative frontiers, shared reads, byte accounting, and tenant validation.
Template-map cache classification and folding crates/ourios-querier/src/template_map.rs, crates/ourios-querier/src/alias_store.rs	Cache outcomes are classified explicitly; non-hit paths fold registry and alias maps from one audit scan and conditionally publish refreshed artifacts.
Query acquisition and row-materialization reuse crates/ourios-querier/src/lib.rs	Query execution acquires a template map once, passes it through execution, reuses it during record rendering, and reports its measured acquisition bytes.
RFC documentation and integration validation crates/ourios-querier/tests/it/rfc0033_cached_template_map.rs, crates/ourios-querier/src/template_registry.rs, docs/rfcs/0031-comparative-evaluation-loki.md	Integration tests cover tenant-aware frontiers, stale artifacts, write-through, acquisition bytes, and tenant isolation; documentation describes cached derivation and acquisition-byte semantics.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Query
  participant TemplateMap
  participant AuditStore
  participant Renderer
  Query->>TemplateMap: load_or_derive tenant map
  TemplateMap->>AuditStore: resolve frontier and read events
  AuditStore-->>TemplateMap: events and measured bytes
  TemplateMap-->>Query: template map and acquisition outcome
  Query->>Renderer: render rows with acquired map
  Renderer-->>Query: records and registry bytes

Loading

Possibly related PRs

• jensholdgaard/ourios#277: Introduces the query record-materialization path reused by this template-map integration.
• jensholdgaard/ourios#414: Also changes template and registry derivation within collect_records.
• jensholdgaard/ourios#464: Introduces the RFC bytes-read metric extended here for template-map acquisition.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the RFC 0033 freshness cache path and write-through changes.
Description check	✅ Passed	The description is detailed and relevant, but it does not follow the required Summary/Related/Checklist template sections.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rfc0033-green-freshness

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Implements the RFC 0033 “freshness + write-through” cached template-map acquisition in the querier query path, and updates the comparative-evaluation byte-contract wording to match the new acquisition semantics.

Changes:

• Adds load_or_derive (LIST → GET → frontier set-equality → fold fallback + best-effort write-through) and threads a single acquired TemplateMap through both resolves_to compilation and record rendering.
• Refactors audit scanning to make “one listing, shared by freshness check + fold” structural via ResolvedAuditSet.
• Activates and implements RFC0033 scenarios .2/.4/.5 integration tests; amends RFC 0031 doc text for registry_bytes_read.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
docs/rfcs/0031-comparative-evaluation-loki.md	Documents amended meaning of registry_bytes_read as template-map acquisition bytes post-RFC0033.
crates/ourios-querier/tests/it/rfc0033_cached_template_map.rs	Implements RFC0033.2/.4/.5 integration scenarios for freshness, advisory behavior, and tenant isolation.
crates/ourios-querier/src/template_registry.rs	Updates registry derivation docs to reflect cached template-map acceleration/fallback semantics.
crates/ourios-querier/src/template_map.rs	Introduces cached read path (load_or_derive), cache outcome types, and write-through abstention logic.
crates/ourios-querier/src/lib.rs	Wires cached acquisition into query compilation + rendering with a single per-query acquisition.
crates/ourios-querier/src/audit_scan.rs	Adds ResolvedAuditSet to couple a single listing with the subsequent fold read.
crates/ourios-querier/src/alias_store.rs	Updates alias-store derivation docs to reflect cached template-map acceleration/fallback semantics.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

🧹 Nitpick comments (3)

crates/ourios-querier/src/template_map.rs (2)

239-278: 🩺 Stability & Availability | 🔵 Trivial

Real backend errors and genuine corruption collapse into the same Unreadable/MissReason::Torn outcome.

Both branches map any non-not-found error (a transient S3 timeout, permission issue, etc.) to Unreadable, which is later reported as MissReason::Torn. This is safe by design (RFC 0033 §1: the fresh fold is always correct), but it means a sustained backend outage silently degrades every query on that tenant to a full audit-tree fold with no distinguishing signal from genuine artifact corruption. Since lookup-outcome telemetry is already deferred to the RFC0033 §3.7 slice (per the module docs and PR scope), this is just a heads-up for that follow-up: consider surfacing the underlying error kind (not just the discarded classification) so the eventual metrics/logs can distinguish "backend flaky" from "artifact torn".

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ourios-querier/src/template_map.rs` around lines 239 - 278, Preserve
the current safe fallback behavior in fetch_artifact, but retain the underlying
non-not-found backend error information instead of discarding it when
classifying failures as Unreadable. Carry enough error-kind/context through the
eventual MissReason::Torn and RFC 0033 §3.7 lookup telemetry path to distinguish
backend failures from genuine artifact corruption, without expanding the current
telemetry scope beyond that follow-up.

---

566-601: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

Add a property test for the TemplateMap reconstruction round trip.

write_through/publish_bytes now make to_json → (store) → from_json a hot-path operation on effectively every cache-miss query, rather than an occasionally-exercised admin path. As per path instructions, **/crates/ourios-{miner,parquet,querier}/**/*.rs: "Use property tests (proptest) for anything with an invariant... Reconstruction is always a property test." The to_json/from_json pair (lines 416-527) is exactly this kind of reconstruction invariant (arbitrary TemplateRegistry/AliasMap combinations should serialize and parse back to an equivalent TemplateMap), and I don't see proptest coverage for it in the reviewed context.

proptest! {
    #[test]
    fn template_map_json_round_trips(registry in arb_registry(), aliases in arb_alias_map(), files in arb_folded_files()) {
        let tenant = tenant();
        let map = TemplateMap { tenant: tenant.clone(), folded_files: files, registry, aliases };
        let bytes = map.to_json().expect("serialize");
        let read = TemplateMap::from_json(&bytes, &tenant).expect("parse");
        prop_assert!(matches!(read, ArtifactRead::Valid(rt) if rt.folded_files() == map.folded_files()
            && rt.registry() == map.registry()
            && rt.alias_map().classes(&tenant) == map.alias_map().classes(&tenant)));
    }
}

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ourios-querier/src/template_map.rs` around lines 566 - 601, Add a
proptest property covering the TemplateMap JSON reconstruction invariant. Reuse
or define generators for TemplateRegistry, AliasMap, and folded files, construct
a TemplateMap with a tenant, serialize it via to_json, parse it with from_json,
and assert a valid ArtifactRead preserves the folded files, registry, and
tenant-scoped alias classes.

Source: Path instructions

crates/ourios-querier/src/lib.rs (1)

816-843: 📐 Maintainability & Code Quality | 🔵 Trivial

Record CacheOutcome at both template-map acquisition sites (crates/ourios-querier/src/lib.rs:834 and :1197). load_or_derive already classifies hit/miss/stale, but both call sites discard _outcome, so the query path still drops the signal needed for cache-hit telemetry.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/ourios-querier/src/lib.rs` around lines 816 - 843, Retain the
CacheOutcome returned by template_map::load_or_derive at both template-map
acquisition sites in the query paths, replacing the discarded _outcome binding.
Record or propagate that outcome through the existing query telemetry mechanism
so cache hit, miss, and stale classifications are emitted for each acquisition,
while preserving the shared acquired map behavior.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@crates/ourios-querier/src/lib.rs`:
- Around line 816-843: Retain the CacheOutcome returned by
template_map::load_or_derive at both template-map acquisition sites in the query
paths, replacing the discarded _outcome binding. Record or propagate that
outcome through the existing query telemetry mechanism so cache hit, miss, and
stale classifications are emitted for each acquisition, while preserving the
shared acquired map behavior.

In `@crates/ourios-querier/src/template_map.rs`:
- Around line 239-278: Preserve the current safe fallback behavior in
fetch_artifact, but retain the underlying non-not-found backend error
information instead of discarding it when classifying failures as Unreadable.
Carry enough error-kind/context through the eventual MissReason::Torn and RFC
0033 §3.7 lookup telemetry path to distinguish backend failures from genuine
artifact corruption, without expanding the current telemetry scope beyond that
follow-up.
- Around line 566-601: Add a proptest property covering the TemplateMap JSON
reconstruction invariant. Reuse or define generators for TemplateRegistry,
AliasMap, and folded files, construct a TemplateMap with a tenant, serialize it
via to_json, parse it with from_json, and assert a valid ArtifactRead preserves
the folded files, registry, and tenant-scoped alias classes.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 99f19bbf-01e3-40c2-bdb7-6597c133ebab

📥 Commits

Reviewing files that changed from the base of the PR and between 3aefac6 and f886a0d.

📒 Files selected for processing (7)

• crates/ourios-querier/src/alias_store.rs
• crates/ourios-querier/src/audit_scan.rs
• crates/ourios-querier/src/lib.rs
• crates/ourios-querier/src/template_map.rs
• crates/ourios-querier/src/template_registry.rs
• crates/ourios-querier/tests/it/rfc0033_cached_template_map.rs
• docs/rfcs/0031-comparative-evaluation-loki.md

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated no new comments.