feat(querier): partition-level time pruning (RFC 0007)

[jensholdgaard]

What

Implements partition-level time pruning in the querier (RFC 0007's explicitly-deferred refinement) — directly motivated by the otel-demo run.

Why (the finding that drove it)

Running b2/otel-demo on the real frozen corpus (#116) showed a template-exact query prunes nothing on real logs — v4 scanned 5/5 row groups. A template recurs in every hour partition, so every row group's template_id min/max matches the queried id. The inverted-index-collapse locality (B2's thesis) only materialises when a time-range predicate skips whole partitions before any footer is opened. The querier had this deferred ("today the time bound is a column predicate").

How

• ourios-parquet — hour_partition_in_window(dir, start_ns, end_ns) -> bool: does a …/year=…/month=…/day=…/hour=HH leaf's [hour_start, +1h) UTC span overlap the half-open window? Conservative by design: any unparseable/foreign path or non-instant returns true (do not prune), so pruning can never drop in-window data.
• ourios-querier — resolve_live_files now takes the query's time_range and skips a leaf partition's files when its hour can't overlap, so DataFusion never opens those footers. It layers on the time_unix_nano column predicate, which remains the row-level correctness authority — pruning is a pure optimisation.

Result

A time-windowed query's scanned row groups track the window, not the corpus's time span — the missing piece for B2 on real logs.

Tests

• rfc0007_2_time_window_prunes_whole_partitions (integration): the same in-window result over a 10× larger time span (2 vs 20 filler hours) hands DataFusion the same tiny row-group set — the out-of-window partitions are pruned at the directory level and never reach the engine (scanned + pruned <= 2, not 21). This is the deterministic analogue of the real-corpus B2 win.
• Unit tests for the overlap math (inside / boundary / before / after) and the conservative fallbacks (non-leaf, foreign, non-canonical paths never pruned).
• Local: cargo fmt --all --check, cargo clippy --workspace --all-targets --all-features -D warnings, cargo test --workspace all green.

Invariants (CLAUDE.md §3.7, RFC0007.5 / §3.9)

Tenant isolation is untouched (resolution still rooted at the tenant dir). Pruning is conservative and never drops in-window data; the column predicate is unchanged, so row-level results are identical with or without pruning. Within RFC 0007's accepted scope (the doc anticipated this), not a new RFC.

Descent-level pruning (skipping whole year/month subtrees) is a further optimisation; leaf pruning already removes the footer reads that dominate.

Part of epic #82 (querier) / epic #81 (thesis-gate validation).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added hour-level partition pruning that filters partitions based on time ranges, reducing the amount of data scanned during queries and improving query performance.

• Tests

  • Added test coverage for time window filtering and partition pruning behavior.

[jensholdgaard]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 50 minutes and 55 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: ea071668-cb2d-4701-b90a-795c418fbbe5

📥 Commits

Reviewing files that changed from the base of the PR and between 6d18d07 and 6e95dc2.

📒 Files selected for processing (4)

• crates/ourios-parquet/src/lib.rs
• crates/ourios-parquet/src/partition.rs
• crates/ourios-querier/src/lib.rs
• crates/ourios-querier/tests/execution.rs

📝 Walkthrough

Walkthrough

This PR implements hour-level partition pruning for time-windowed Parquet queries. A new hour_partition_in_window function checks whether an hour partition might overlap a time window; the querier integrates this to skip out-of-window year/month/day/hour directories before reading manifests or glob files, reducing I/O overhead while preserving row-level filtering correctness.

Changes

Hour partition pruning feature

Layer / File(s)	Summary
Hour partition overlap logic crates/ourios-parquet/src/partition.rs	New hour_partition_in_window function parses Hive year=/month=/day=/hour= directory segments and conservatively determines overlap with a [start_ns, end_ns) time window. Private helpers compute the UTC nanosecond span for each hour; tests cover boundary cases and conservative "never prune" behavior on unparseable/invalid paths.
Public API re-export crates/ourios-parquet/src/lib.rs	hour_partition_in_window is added to the pub use partition::{...} re-export list, making it available for downstream consumers.
Querier read-path integration crates/ourios-querier/src/lib.rs	resolve_live_files signature now accepts an optional time window; when present, each partition directory is tested via hour_partition_in_window and skipped if outside the window, before reading manifest or glob files. Querier::run passes request.time_range into resolve_live_files. Existing unit tests updated to call the new signature with None.
End-to-end partition pruning test crates/ourios-querier/tests/execution.rs	New corpus_over_hours helper generates hour-partitioned corpora with configurable filler hours. Test rfc0007_2_time_window_prunes_whole_partitions verifies that a time-windowed query returns stable row counts and bytes read across different corpus sizes, confirming partition-level pruning skips out-of-window directories before manifest access.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• jensholdgaard/ourios#96: Modifies resolve_live_files to read per-partition manifest.json-listed files; the partition-pruning logic in this PR gates that manifest-driven resolution.
• jensholdgaard/ourios#87: Applies time_range filter for DataFusion query execution; this PR additionally uses that time_range to prune hour partition directories before I/O.
• jensholdgaard/ourios#89: Extends ourios-querier/tests/execution.rs with hour-partitioned corpora and assertions about row-group/bytes work consistency, similar patterns to the new pruning test.

Poem

A rabbit hops through partitions with care,
Hour by hour, skipping what's not there.
Time windows guide each leap so precise,
No needless reads—partition pruning's nice! 🐰✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately captures the main change: partition-level time pruning in the querier, and references the RFC scope (0007).
Description check	✅ Passed	The description includes all required sections: a detailed what/why/how, test coverage, invariant guarantees, and addresses all template requirements comprehensively.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/querier-partition-time-pruning

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

This PR implements RFC 0007’s deferred refinement: partition-level time pruning in the querier, allowing queries with a time window to skip entire year/month/day/hour partitions before DataFusion opens Parquet footers. This makes scanned row groups and bytes read track the query window rather than the corpus’s total time span.

Changes:

• Add hour_partition_in_window(dir, start_ns, end_ns) -> bool in ourios-parquet to conservatively decide whether an hour=HH partition can overlap a half-open time window.
• Update ourios-querier’s resolve_live_files to accept the query time_range and skip out-of-window hour partitions when resolving live Parquet files.
• Add integration + unit tests asserting directory-level pruning keeps DataFusion work flat as the corpus time span grows.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File	Description
crates/ourios-querier/tests/execution.rs	Adds an integration test proving time-windowed queries prune entire out-of-window hour partitions before DataFusion sees them.
crates/ourios-querier/src/lib.rs	Threads time_range into live-file resolution and applies hour-partition overlap pruning prior to DataFusion execution.
crates/ourios-parquet/src/partition.rs	Implements conservative hour-partition/window overlap predicate + unit tests for boundary and fallback cases.
crates/ourios-parquet/src/lib.rs	Re-exports hour_partition_in_window for use by the querier.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[jensholdgaard]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Copilot]

Pull request overview

Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.