test(parquet): compaction re-projects across a re-typing (RFC 0042 slice 4) - #653
Conversation
…ice 4) RFC0042.8: two input files — the key promoted under the string class in one epoch, unpromoted in the other — consolidate under the current i64 declaration into an Int64 column projected from JSON truth (both epochs' values present), and the rewrite is deterministic: byte-identical across two identical runs, the RFC0036.4 property under a fixed config. The RFC banner records where the ladder stands: .1-.8 green in-repo across #649-#652 + this test; RFC0042.9 (the dogfood corpus gate) outstanding until the first agent capture under the typed set, so the status stays specified with the implementation ahead of the flip. Verified: cargo fmt --check, clippy clean, nextest 165/165 (parquet), mdbook build. Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>
|
Warning Review limit reached
Next review available in: 5 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (2)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Pull request overview
This PR completes RFC 0042 slice 4 by adding the final in-repo acceptance test for RFC0042.8 (compaction re-projecting data under the current typed promoted-attribute declaration across a re-typing), and updates the RFC document with a ladder-status note reflecting the current criterion state.
Changes:
- Add a compaction regression test that verifies re-projection under a re-typing (
string/unpromoted →i64) and asserts byte-identical determinism across runs. - Update RFC 0042 with a status note summarizing which criteria are green and what remains outstanding (RFC0042.9 dogfood corpus gate).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| docs/rfcs/0042-typed-numeric-promotion.md | Adds an RFC status note summarizing ladder progress and the remaining dogfood gate. |
| crates/ourios-parquet/src/compaction.rs | Adds an RFC0042.8 unit test validating typed re-projection during compaction and asserting deterministic output bytes. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…lasses The FinOps loop's headline query is sum(attr.cost_usd) by attr.model, but #608's promotion set carries no cost or usage key, so it errors with the promotion hint. This PR's first draft promoted the keys bare — and review caught that RFC 0022 §3.1's string-only projection would have made always-NULL columns and a silently EMPTY sum (worse than the loud error). That finding became RFC 0042; with its typed classes now green in-repo (#649-#653), the promotion lands the way it must: classes matching the AnyValue variants Claude Code actually emits, verified against captured api_request events — - { key: cost_usd, type: f64 } # doubleValue - { key: input_tokens, type: i64 } # intValue - { key: output_tokens, type: i64 } # intValue cost_usd_micros is skipped as redundant under f64 aggregation; cache token counts are additive later if a query wants them. Cost stays the source's responsibility (Claude Code computes and stamps it); Ourios stores, attributes, and aggregates. Pricing tables and FOCUS-shaped output belong to consumers of the query surface. Verified: the server boots the config through the RFC0042.6 validation path (fails only on the port the running dogfood server holds). Promotion is a write-side projection — RFC0042.9's live sum lands with the first capture after the dogfood server restarts on this config. Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org>
) * feat(dogfood): promote cost_usd + token counts under RFC 0042 typed classes The FinOps loop's headline query is sum(attr.cost_usd) by attr.model, but #608's promotion set carries no cost or usage key, so it errors with the promotion hint. This PR's first draft promoted the keys bare — and review caught that RFC 0022 §3.1's string-only projection would have made always-NULL columns and a silently EMPTY sum (worse than the loud error). That finding became RFC 0042; with its typed classes now green in-repo (#649-#653), the promotion lands the way it must: classes matching the AnyValue variants Claude Code actually emits, verified against captured api_request events — - { key: cost_usd, type: f64 } # doubleValue - { key: input_tokens, type: i64 } # intValue - { key: output_tokens, type: i64 } # intValue cost_usd_micros is skipped as redundant under f64 aggregation; cache token counts are additive later if a query wants them. Cost stays the source's responsibility (Claude Code computes and stamps it); Ourios stores, attributes, and aggregates. Pricing tables and FOCUS-shaped output belong to consumers of the query surface. Verified: the server boots the config through the RFC0042.6 validation path (fails only on the port the running dogfood server holds). Promotion is a write-side projection — RFC0042.9's live sum lands with the first capture after the dogfood server restarts on this config. Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org> * docs(dogfood): a wrong-class promotion yields a NULL aggregate, not an empty result The comment claimed sum over an always-NULL column silently returns empty; the RFC0002.18 semantics is a NULL aggregate (every input excluded, value null). Same silence, precise shape. Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org> --------- Signed-off-by: Jens Holdgaard Pedersen <jens@holdgaard.org> Signed-off-by: Jens Holdgaard Pedersen <Jens@holdgaard.org>
What (RFC 0042 slice 4 — the last in-repo criterion)
RFC0042.8: compaction re-projects under the current typed declaration, across a re-typing. Two input files — the key promoted
string-class in one epoch, unpromoted in the other — consolidate under thei64declaration into anInt64column whose cells come from JSON truth (both epochs' values present: the string-epoch file's Utf8 column was all-NULL, but its JSON carried the int, and re-projection reads truth, not the stale column). Determinism asserted the strong way: two identical runs produce byte-identical output — the RFC0036.4 property under a fixed config.This is the convergence story §3.3 promised: ordinary compaction migrates history toward the current declaration as a side effect, no rewrite campaign needed.
Ladder state (recorded in the RFC banner)
Status stays
specifieduntil .9 returns real spend — the flip should ride evidence, not implementation completeness.Invariants
Test + doc only. RFC0036.4 byte-identity is strengthened into the typed world, not weakened.
Verification
cargo fmt --check· clippy clean · nextest 165/165 (parquet) ·mdbook buildclean.