feat(storage): effective-timestamp column + windowing fallback (RFC0005.13)

[jensholdgaard]

Implements the effective-timestamp amendment merged in #178 (RFC 0005 §3.2/§3.4/§3.6/§3.9 + RFC 0002 §6.2, main @ 407c12a) across ourios-parquet, ourios-querier, and ourios-bench. This unblocks B1, the last unmeasured thesis gate: ~15 % of the v5/v6 OTel-Demo corpora are observed-only-timestamp records, which were previously unaddressable by time and tripped the bench's zero-timestamp guard.

What lands

ourios-parquet — the stored column via one shared derivation

• choose_partition_timestamp is extracted as the public effective_time_unix_nano() (RFC 0005 §3.2 rule 1). Both the §3.4 partition tuple (PartitionKey::derive) and the new stored column (the record-batch writer) call this one function, so the spec's "the partition tuple and the stored column never disagree" rule is structural, not a convention.
• New effective_time_unix_nano column: OPTIONAL TIMESTAMP(NANOS, UTC) / INT64, placed after observed_time_unix_nano, stats-bearing with dictionary off (§3.6 — same encoding row as time_unix_nano). The writer always populates it; NULL exists only in pre-amendment files.
• MinedRecord is unchanged (RFC 0003 §6.6 note: receiver contract untouched); the reader maps the column to nothing — it is derivable, not carried, and outside the RFC0005.1 round-trip surface.
• Schema pin (RFC0005.10) updated in lockstep with the RFC §3.2 table.

ourios-querier — the window + the §3.9 carve-out

• Both window paths (QueryRequest.time_range and the DSL range(...) stage) compile through one shared time_window_filter: half-open [from, to) over effective_time_unix_nano (RFC 0002 §6.2 amendment). Bare ts is untouched — it still resolves to the verbatim wire time_unix_nano (RFC0001.10).
• The §3.9 rule-2 carve-out (the explicit exception to the absent-OPTIONAL ⇒ predicate-false convention of RFC0007.4):
  • column absent from the union schema ⇒ every file predates the amendment ⇒ the window filters time_unix_nano directly, exactly as before;
  • column present in a mixed scan ⇒ DataFusion fills pre-amendment rows with NULL, which fails both window bounds — the forbidden silent-hiding outcome. The filter compiles to (eff >= lo AND eff < hi) OR (eff IS NULL AND ts >= lo AND ts < hi): post-amendment writers never store NULL, so IS NULL identifies exactly the old rows needing the fallback. The OR shape (not coalesce) keeps the predicate inside DataFusion's pruning grammar — min/max stats prune the effective branch, null counts collapse the fallback branch (RFC 0005 §3.2 rule 3, the B1 mechanism); the new test pins row-group pruning on the stored column.
• hour_partition_in_window needed no change — §3.4 partitioning already used the same fallback.

ourios-bench — B1 eligibility keys off the effective span (§3.2 rule 7)

• The store builders' span ({min,max}_effective_time_unix_nano), the zero-timestamp guard (zero_effective_ts_rows: counts only rows with neither timestamp), and the reference corpus's hour spool all derive from the shared effective_time_unix_nano() — the bench can never disagree with what the query window filters. Observed-only corpora are now B1-eligible.

RFC0005.13 acceptance (both halves)

• Half 1 (crates/ourios-parquet/tests/effective_timestamp.rs + crates/ourios-querier/tests/rfc0005_13.rs): a time_unix_nano = 0 / observed_time_unix_nano = T record stores effective == T (read raw via the parquet crate), lands under the partition tuple derived from T, keeps the wire time_unix_nano = 0 verbatim, and a range(...) window containing T returns it (and a window over the epoch does not).
• Half 2 (rfc0005_13.rs): a pre-amendment-shaped file (the writer's batch with the column projected away, laid down via the raw ArrowWriter per the RFC0007.4 pattern) answers the same window as effective := time_unix_nano — alone and mixed with a post-amendment file in one scan. No error, no hidden rows.
• Bench: a synthetic observed-only OTLP corpus passes the B1 eligibility guard (zero_effective_ts_rows == 0, span from observed values, reference fully spooled).

CLAUDE.md §3.5 / §4-adjacent claim (reviewers: please check this one)

The schema change is additive-OPTIONAL with an explicit migration story: old files need no rewrite, and — this is the load-bearing claim — pre-amendment files keep answering time-window queries identically to before this PR, because the §3.9 rule-2 read default substitutes effective := time_unix_nano per file instead of compiling the window to false. The mixed-scan NULL case (DataFusion's schema union) is the spot where a naive implementation silently hides all historical data from every query; rfc0005_13_pre_amendment_file_windows_on_time_unix_nano pins that it does not happen. Readers tolerate the absent column (old files) and ignore the unknown column (the pre-amendment reader reading new files) per RFC0007.4, which stays green.

Hazard 4.4 (small files) untouched; hazard 4.6: the new filter stays inside the compile layer — no DataFusion type or SQL crosses the public surface.

Verification (all run locally, all green)

• cargo test --all-features — 79 suites, 613 tests, 0 failed (incl. rfc0001_time_preserved and the RFC0007 structural tests, unchanged)
• cargo fmt --all --check
• cargo clippy --all-targets --all-features -- -D warnings
• cargo doc --workspace --no-deps --all-features
• cargo bench -p ourios-bench --no-run

🤖 Generated with Claude Code

Summary by CodeRabbit

Release Notes

• New Features

  • Added effective timestamp support: query time-range filtering and data storage now derive and utilize timestamp values for improved query accuracy and data coverage.

• Improvements

  • Enhanced row-group pruning via column timestamp statistics for faster query execution.

• Tests

  • Comprehensive test coverage added for effective timestamp behavior, query window semantics, and compatibility with both current and legacy data formats.

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 48 minutes and 51 seconds. Learn how PR review limits work.

Your organization has run out of usage credits. Purchase more credits in the billing tab to continue.

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: eace9825-2e01-4b75-a7ea-a59885c41cbe

📥 Commits

Reviewing files that changed from the base of the PR and between 6420dd2 and de853de.

📒 Files selected for processing (3)

• crates/ourios-bench/benches/b1.rs
• crates/ourios-bench/benches/b2.rs
• crates/ourios-bench/src/store.rs

📝 Walkthrough

Walkthrough

This PR implements RFC 0005 §3.2 effective timestamp support across Parquet storage, benchmark tooling, and query filtering. Effective timestamps fall back to observed timestamps when wire timestamps are zero, enabling time-window queries and B1 eligibility to correctly handle observed-only telemetry records while maintaining pre-amendment backward compatibility.

Changes

RFC 0005 §3.2 Effective Timestamp Implementation

Layer / File(s)	Summary
Effective timestamp derivation helper and schema crates/ourios-parquet/src/partition.rs, crates/ourios-parquet/src/lib.rs	New public effective_time_unix_nano() function applies RFC 0005 §3.2 selection (time_unix_nano if non-zero, else observed_time_unix_nano, else epoch). PartitionKey::derive uses it. Column constant and optional schema field added.
Parquet record batch writer updates crates/ourios-parquet/src/record_batch.rs, crates/ourios-parquet/src/writer.rs, crates/ourios-parquet/src/reader.rs	RecordBatch builder computes and emits effective_time_unix_nano column per record. Writer disables dictionary encoding for the timestamp column. Test scaffolding supports reading it back.
Parquet effective timestamp validation crates/ourios-parquet/tests/effective_timestamp.rs, crates/ourios-parquet/tests/schema_pin.rs	Integration tests verify observed-only records store wire value 0 but effective column T, partition derivation matches stored effective, and schema includes the new field.
Store builder fields and pipeline updates crates/ourios-bench/src/store.rs	BuiltStore and B1Store replace raw-timestamp span fields with effective-timestamp equivalents. Shared build_store pipeline derives effective timestamps per record, wires them through callbacks, excludes zero-effective rows from span tracking and reference spooling.
Store builder tests for effective timestamps crates/ourios-bench/src/store.rs (test suite)	Existing tests updated to assert effective-span fields; new test coverage for observed-only corpora verifies B1 eligibility, reference spooling, and severity queries driven by effective timestamps.
Benchmark function updates (B1, B2) crates/ourios-bench/benches/b1.rs, crates/ourios-bench/benches/b2.rs	severity_query and first_hour_window now use effective-timestamp span fields for skip conditions and time bounds. Documentation clarifies RFC 0005 §3.2 amendment with observed-only corpus eligibility.
Query time-window filter with effective timestamp support crates/ourios-querier/src/lib.rs, crates/ourios-querier/src/compile.rs	New time_window_filter() helper generates DataFusion expression filtering on effective_time_unix_nano with OR carve-out for NULL (pre-amendment rows). apply_request_filters and compile module updated to use it. Documentation specifies time_range applies to effective timestamps.
RFC0005.13 query integration tests crates/ourios-querier/tests/rfc0005_13.rs	End-to-end tests verify observed-only records matched by effective timestamp windows (not wire zeros), pre-amendment files omitting effective_time_unix_nano behave as effective := time_unix_nano (alone and mixed), and row-group pruning by effective timestamp column statistics occurs.

Sequence Diagram

sequenceDiagram
  participant App as Application
  participant Emit as Parquet Emitter
  participant Store as Store Builder
  participant Query as Query Executor
  participant Storage as Parquet Storage
  
  App->>Emit: MinedRecord(time=0, observed=T)
  Emit->>Emit: compute effective_time_unix_nano() → T
  Emit->>Store: pass effective span
  Store->>Store: track min/max effective, exclude zero-effective
  Emit->>Storage: write time_unix_nano=0, effective_time_unix_nano=T
  
  Query->>Query: parse range(lo, hi)
  Query->>Storage: filter WHERE effective >= lo AND effective < hi
  Storage-->>Query: observed-only record (effective=T matches)
  Query-->>App: result row_count

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

• jensholdgaard/ourios#171: Both PRs modify B1/B2 bench code and crates/ourios-bench/src/store.rs around store bookkeeping; main PR switches to RFC0005 effective-timestamp fields while retrieved PR introduces B1 store/window implementation.
• jensholdgaard/ourios#44: Both PRs modify PartitionKey::derive timestamp logic in crates/ourios-parquet/src/partition.rs; main PR replaces it with effective_time_unix_nano derivation while retrieved PR adds time fallback.
• jensholdgaard/ourios#92: Retrieved PR adds the ourios-bench store-builder scaffolding (build_query_store, BuiltStore); main PR updates that same pipeline to compute/store effective timestamps by refactoring the shared build_store logic.

Poem

🐰 Timestamp Tales

Wire zeroes once told tales incomplete,
Observed timestamps now bring the heat.
Effective derivation, a fallback so sweet—
RFC 0005's §3.2, the merger so neat.
Queries now window on truth, not on wire,
B1 benchmarks ascend ever higher! ✨

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarizes the main change: implementing the effective-timestamp column and windowing fallback for RFC0005.13 across the codebase.
Description check	✅ Passed	The description is comprehensive and detailed, covering what lands, RFC acceptance criteria, verification steps, and design rationale—though it deviates from the template structure.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/effective-timestamp-rfc0005-13

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jensholdgaard]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Copilot]

Pull request overview

This PR implements RFC0005.13’s “effective timestamp” (wire time_unix_nano unless it is 0, then fall back to observed_time_unix_nano) as a stored Parquet column and updates query windowing + bench bookkeeping to use it, while preserving backward compatibility with pre-amendment files via a NULL-aware fallback.

Changes:

• ourios-parquet: Add OPTIONAL effective_time_unix_nano to the data schema and populate it via a shared effective_time_unix_nano() derivation used for both partitioning and writing.
• ourios-querier: Route both QueryRequest.time_range and DSL range(...) through a shared time_window_filter over effective_time_unix_nano, with a mixed-scan NULL fallback to time_unix_nano for pre-amendment files.
• ourios-bench: Switch span/eligibility/reference spooling from wire timestamps to effective timestamps so observed-only corpora remain benchmark-eligible.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
crates/ourios-querier/tests/rfc0005_13.rs	Adds RFC0005.13 querier-path tests for observed-only windowing, pre-amendment fallback (alone + mixed scan), and row-group pruning behavior.
crates/ourios-querier/src/lib.rs	Documents QueryRequest.time_range as effective-time bounds; introduces shared time_window_filter and uses it in the request filter path.
crates/ourios-querier/src/compile.rs	Updates DSL compilation to apply the shared effective-time window filter rather than filtering directly on time_unix_nano.
crates/ourios-parquet/tests/schema_pin.rs	Updates the pinned RFC0005.10 schema field list to include effective_time_unix_nano.
crates/ourios-parquet/tests/effective_timestamp.rs	Adds RFC0005.13 storage-path tests asserting stored effective timestamp, partition derivation consistency, and wire-time preservation.
crates/ourios-parquet/src/writer.rs	Disables dictionary encoding for the new effective-time column alongside other timestamp columns.
crates/ourios-parquet/src/record_batch.rs	Adds the effective-time builder column and populates it via the shared derivation during batch construction.
crates/ourios-parquet/src/reader.rs	Updates reader test scaffolding to account for the newly added effective-time column.
crates/ourios-parquet/src/partition.rs	Extracts/renames partition timestamp selection into public effective_time_unix_nano() and uses it in PartitionKey::derive.
crates/ourios-parquet/src/lib.rs	Adds the effective-time column constant and inserts the column into the published data_schema() in the intended position.
crates/ourios-bench/src/store.rs	Moves store span tracking, B1 eligibility guard, and reference spooling to effective timestamps; adds an observed-only eligibility test.
crates/ourios-bench/benches/b2.rs	Updates B2’s window selection to use the effective-time span.
crates/ourios-bench/benches/b1.rs	Updates B1’s query window and skip conditions to use effective-time span and the new zero-effective guard.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 13 changed files in this pull request and generated no new comments.