feat(parquet): scaffold ourios-parquet with RFC 0005 schemas (PR-D) - #42
Merged
Conversation
Opens Phase 2 by landing the empty crate plus the schema
declarations RFC 0005 §3.2 (data file) and §3.7 (audit event)
pin. No writer / reader / audit-stream code yet — those are
follow-on PRs per the §3.10 crate-shape plan.
What's in this PR:
- crates/ourios-parquet added to the workspace members list.
- data_schema() and audit_schema() return Arrow SchemaRefs
matching RFC 0005's tables column-for-column (logical type,
physical type via Arrow DataType, REQUIRED/OPTIONAL via
is_nullable).
- columns::* and audit_columns::* public constants for
greppability so production code addressing columns by name
uses a single source-of-truth identifier per the §3.2
"readers MUST address columns by name" rule.
- RFC0005.10 schema-pin test: hand-written expected field list
in tests/schema_pin.rs, asserted against data_schema() /
audit_schema(). Adding a column to lib.rs without updating
the fixture fails the test — the "schema-as-spec" pin RFC
0005 §5 / §6 names. Mirrors the RFC0004.3 tunable-set pin
precedent.
Encoding choices the Arrow schema commits to:
- Timestamps: Timestamp(Nanosecond, Some("UTC")) — Parquet
TIMESTAMP(NANOS, isAdjustedToUTC=true).
- attributes / resource_attributes: Utf8 (Parquet STRING)
carrying canonical JSON per §3.3.
- trace_id / span_id: FixedSizeBinary(16) / (8) with no
logical type — opaque bytes, not RFC 4122 UUIDs.
- params: List<Struct<type_tag: Int32, value: Binary>>,
REQUIRED (empty list when body_kind = Structured).
- separators: List<Binary>, REQUIRED.
- slots_expanded: List<Struct<slot_index: Int32, types_added:
List<Int32>>>, REQUIRED — matches RFC 0001 §6.4's
Vec<SlotExpansion> shape.
- body: Binary (no logical type) — non-UTF-8 raw bytes allowed
per the §3.6 cardinality invariant rationale.
Verified locally:
- cargo build -p ourios-parquet — clean
- cargo test --all-features -p ourios-parquet — 2 passed
- cargo fmt --all --check — clean
- cargo clippy --all-targets --all-features -- -D warnings — clean
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Scaffolds the new ourios-parquet workspace crate and pins the RFC 0005 Parquet/Arrow schema surface in executable form, so subsequent writer/reader work can build on an immutable, greppable contract.
Changes:
- Added
crates/ourios-parquetwithdata_schema()/audit_schema()returning ArrowSchemaRefs aligned with RFC 0005. - Introduced public
columns::*andaudit_columns::*constants for column-name addressing by name. - Added RFC0005.10 “schema-as-spec” pin tests that assert schema field order/types/nullability against a handwritten fixture.
Reviewed changes
Copilot reviewed 4 out of 5 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| crates/ourios-parquet/src/lib.rs | Defines the data-file and audit-event Arrow schemas plus column-name constants. |
| crates/ourios-parquet/tests/schema_pin.rs | Adds schema pin tests asserting exact field lists for RFC0005.10. |
| crates/ourios-parquet/Cargo.toml | Introduces the new crate and its Arrow dependency, with workspace lints enabled. |
| Cargo.toml | Adds crates/ourios-parquet to the workspace members list. |
| Cargo.lock | Locks new Arrow-related transitive dependencies for reproducible builds. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
jensholdgaard
added a commit
that referenced
this pull request
May 21, 2026
#45) * feat(parquet): reader with §3.9 contract + RFC0005.1/2/3/4/9/11 (PR-F) Lands the Reader half of RFC 0005 §3.10's crate-shape plan. Audit stream is still pending (PR-G). - `crates/ourios-core/src/audit.rs` — adds `ParamType::Unknown(i32)` for the RFC 0005 §3.9 catch-all variant. The reader produces it for type_tag ordinals 8..; the writer round-trips the carried ordinal back to disk via the updated `param_type_ordinal` match arm. No existing exhaustive matches on `ParamType` (audited via grep), so the new tuple variant doesn't break any callers. - `crates/ourios-parquet/src/reader.rs` — `Reader::open_partition` (production query path, enforces §3.9 row-vs-path validation) + `Reader::open_file` (diagnostic, no validation) + `read_all`. Implements §3.9's three normative contract clauses end-to-end: unknown columns silently ignored, missing OPTIONAL columns surface as None, missing baseline REQUIRED columns are a hard error naming the column. Row-vs-path validation reuses `PartitionKey::derive` from PR-E2 so writer and reader use the identical §3.4 fallback algorithm. `MinedRecord` reconstruction goes column-by-column with helper functions per Arrow type (string / u64 / u32 / u8 / f32 / bool / timestamp / fixed-bytes / list / struct-list). Body is UTF-8-lossy-decoded for the Option<String> ↔ Bytes column gap noted in PR-E1. Tests (14 new, all green): - `tests/round_trip.rs` — RFC0005.1: writes a populated record through Writer, reads back through Reader, asserts full struct equality. Sub-test pins the body raw-bytes round-trip. - `tests/reader_compat.rs` — hand-built Parquet files with: - RFC0005.2: omitted OPTIONAL column → reader returns None - RFC0005.3: extra unknown column → reader silently ignores - RFC0005.4: omitted REQUIRED column → reader hard-errors naming the column - RFC0005.9: type_tag = 99 → reader returns ParamType::Unknown(99) (round-trips through Writer → file → Reader) - `tests/row_vs_path_validation.rs` — RFC0005.11: - tenant_id mismatch → PartitionMismatch error - hour mismatch → PartitionMismatch error - §3.4 fallback (time=0, observed!=0) → validates cleanly when supplied partition matches the observed-time bucket - open_file mode skips validation entirely Verified locally: - cargo build --all-features — clean - cargo test --all-features — 200+ tests passing - cargo fmt --all --check — clean - cargo clippy --all-targets --all-features -- -D warnings — clean Phase 2 progress (per docs/roadmap.md): - PR-D ✅ scaffold (#42) - PR-E1 ✅ MinedRecord extension (#43) - PR-E2 ✅ Writer (#44) - PR-F ✅ Reader (this PR) - PR-G ⏳ AuditWriter / AuditReader Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * fixup! feat(parquet): reader with §3.9 contract + RFC0005.1/2/3/4/9/11 (PR-F) * fixup! feat(parquet): reader with §3.9 contract + RFC0005.1/2/3/4/9/11 (PR-F) * fixup! feat(parquet): reader with §3.9 contract + RFC0005.1/2/3/4/9/11 (PR-F) Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Opens Phase 2 of
docs/roadmap.md. Lands the emptyourios-parquetcrate plus the data-file and audit-event Arrow schemas that RFC 0005 §3.2 and §3.7 pin. NoWriter/Reader/ audit-stream code yet — those are follow-on PRs per the §3.10 crate-shape plan.crates/ourios-parquetadded to the workspace.data_schema()andaudit_schema()return ArrowSchemaRefs matching RFC 0005 column-for-column (logical type, physical type, REQUIRED/OPTIONAL).columns::*andaudit_columns::*public constants for greppable column names — production code addresses columns through these per §3.2's "readers MUST address columns by name" rule.tests/schema_pin.rs: hand-written expected field list, asserted against the production schema. Adding a column tolib.rswithout updating the fixture fails the test — the "schema-as-spec" pin (mirrors the RFC0004.3 tunable-set pin precedent).Invariant coverage
This PR is the executable form of RFC0005.10 (the schema-as-spec pin). No other §5 scenarios are exercised yet — they unblock as the writer / reader / audit-stream PRs land.
Test plan
cargo build -p ourios-parquet— cleancargo test --all-features -p ourios-parquet— 2 passed (rfc0005_10_data_schema_matches_pinned_field_list,rfc0005_10_audit_schema_matches_pinned_field_list)cargo fmt --all --check— cleancargo clippy --all-targets --all-features -- -D warnings— cleanFollow-up PRs
Writer— opens a file at a partition path, appends rows, rotates row groups at the §3.5 threshold.Reader— §3.9 forward/backward-compat contract + row-vs-path validation.AuditWriter/AuditReader— parallel file series per §3.7.🤖 Generated with Claude Code