feat(parquet): rfc0013 green — buffer-and-put encode/decode + store round-trip

[jensholdgaard]

What

RFC 0013 green slice 2: the buffer-and-put I/O primitives (the maintainer-chosen object-store-native model) — encode MinedRecords to in-memory Parquet bytes, decode from bytes, and a round-trip through Store. Foundation for migrating the writer/reader off filesystem paths; RFC stays red until the §5 stubs pass (consumer migration, later slices).

• encode_records_to_parquet(records, zstd_level) -> Vec<u8> (writer.rs) — same schema/codec/§3.6 policy as the file Writer, but targets a Vec<u8> via ArrowWriter<Vec<u8>> (row-group sizing still applies in-buffer).
• Reader::open_bytes(bytes::Bytes) (reader.rs) — decode from in-memory bytes (the Store.get read path); same RFC 0005 §3.9 baseline-column check as open_file, factored into a shared require_baseline_columns helper.
• tests/buffer_and_put.rs — 500 records → encode → Store.put → Store.get → open_bytes → read_all, asserting full row recovery (local backend).
• bytes direct dep (the parquet reader's in-memory ChunkReader source; already in the arrow/parquet tree).

Verified locally

cargo fmt, cargo clippy -p ourios-parquet --all-targets (clean), cargo test -p ourios-parquet (the new round-trip passes; full suite 70+ green; the 8 §5 stubs + the sizing stub stay ignored), cargo deny check (ok — bytes adds no new graph entries).

Next green slices

3. Migrate the ingester write path to encode_records_to_parquet + Store.put (the async edge) — write side onto object storage.
4. Migrate the querier read path to Store.get + Reader::open_bytes → RFC0013.2 / .8.
5. Manifest conditional PUT → RFC0013.4; S3 backend (aws feature) + MinIO testcontainers → RFC0013.1/.3/.5/.7; WAL-stays-local wiring → RFC0013.6.

Invariants

Adds the buffer-and-put primitives; the file-based Writer/Reader are unchanged (the open_file refactor is behaviour-preserving — shared helper). No consumer migrated yet. Serves §3.6; WAL stays local (§3.4).

🤖 Generated with Claude Code

Summary by CodeRabbit

• New Features

  • Added in-memory Parquet file encoding capability, enabling direct serialization of record collections to compressed Parquet bytes format for flexible storage and transmission.

• Improvements

  • Strengthened schema validation by ensuring baseline column requirements are consistently verified across all reader initialization paths, providing early error detection for incompatible schemas.

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 49 minutes and 35 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: c33d90c2-0df4-406c-9941-3749c46116d9

📥 Commits

Reviewing files that changed from the base of the PR and between 970c6f3 and af60f6a.

📒 Files selected for processing (1)

• crates/ourios-parquet/src/writer.rs

📝 Walkthrough

Walkthrough

Adds encode_records_to_parquet, a new public function that serializes a MinedRecord slice to in-memory Parquet bytes using ZSTD compression and row-group flushing. Refactors the RFC 0005 §3.9 baseline column check into a shared require_baseline_columns helper applied to both open_file and open_bytes. A new integration test validates the encode → Store::put → Store::get → Reader::open_bytes round-trip.

Changes

In-memory Parquet encode, reader validation, and round-trip test

Layer / File(s)	Summary
encode_records_to_parquet and public API surface crates/ourios-parquet/Cargo.toml, crates/ourios-parquet/src/writer.rs, crates/ourios-parquet/src/lib.rs	Adds bytes = "1" dependency, implements encode_records_to_parquet using ArrowWriter<Vec<u8>> with the same row-group flushing behavior as the file writer, and re-exports it from the crate root alongside existing writer exports.
require_baseline_columns helper and open_bytes validation crates/ourios-parquet/src/reader.rs	Factors out the baseline REQUIRED column check into require_baseline_columns, calls it from both open_file and open_bytes, so the in-memory bytes path now returns ReaderError::MissingRequiredColumn on schema violations identical to the file path.
Buffer-and-put round-trip integration test crates/ourios-parquet/tests/buffer_and_put.rs	New Tokio test generates 500 MinedRecords, encodes to Parquet bytes with DEFAULT_ZSTD_LEVEL, stores and retrieves via Store::local, decodes with Reader::open_bytes, and asserts row count and byte-for-byte record equality.

Sequence Diagram(s)

sequenceDiagram
  participant Test
  participant encode_records_to_parquet
  participant Store
  participant Reader

  rect rgba(70, 130, 180, 0.5)
    note over Test,encode_records_to_parquet: Encode
    Test->>encode_records_to_parquet: records[0..500], DEFAULT_ZSTD_LEVEL
    encode_records_to_parquet-->>Test: Vec<u8> (Parquet bytes)
  end

  rect rgba(60, 179, 113, 0.5)
    note over Test,Store: Store round-trip
    Test->>Store: put(key, bytes)
    Test->>Store: get(key)
    Store-->>Test: Bytes
  end

  rect rgba(210, 105, 30, 0.5)
    note over Test,Reader: Decode & assert
    Test->>Reader: open_bytes(bytes)
    Reader-->>Test: decoded records
    Test->>Test: assert row count == 500 and records match
  end

Loading

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• jensholdgaard/ourios#45: Implements ReaderError::MissingRequiredColumn for RFC 0005 §3.9 baseline column enforcement in reader.rs, which this PR refactors into the shared require_baseline_columns helper.
• jensholdgaard/ourios#84: Introduces DEFAULT_ZSTD_LEVEL and the writer_properties / validated zstd_level plumbing that encode_records_to_parquet builds on directly.
• jensholdgaard/ourios#231: Provides the Store async put/get API exercised by the new buffer_and_put round-trip integration test.

Poem

🐇 Hoppity-hop, bytes fly through the air,
Parquet is packed with the greatest of care,
A helper now checks every column in place,
Round-trips through the store at a bunny-quick pace,
Five hundred records, all matched row for row —
The warren of data puts on quite a show! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title clearly references RFC 0013 and the buffer-and-put encode/decode round-trip mechanism, directly matching the PR's core contribution.
Description check	✅ Passed	Description is comprehensive with clear sections covering what was added, RFC reference, verification steps, and next steps; all required template sections are present and well-populated.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rfc0013-green-buffer-and-put

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Implements RFC 0013 “buffer-and-put” primitives in ourios-parquet: encoding MinedRecords into in-memory Parquet bytes for object-store PUT, and decoding from in-memory bytes for object-store GET, plus a local-backend round-trip test. This is a foundation step toward migrating ingester/querier I/O off filesystem paths and onto Store.

Changes:

• Add encode_records_to_parquet(&[MinedRecord], zstd_level) -> Vec<u8> for in-memory Parquet encoding.
• Add Reader::open_bytes(bytes::Bytes) and factor the RFC 0005 §3.9 baseline REQUIRED-column validation into a shared helper.
• Add an end-to-end buffer-and-put round-trip test using Store::local, plus a direct bytes dependency.

Reviewed changes

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
crates/ourios-parquet/tests/buffer_and_put.rs	Adds an async round-trip test: encode → Store.put → Store.get → Reader::open_bytes → decode.
crates/ourios-parquet/src/writer.rs	Adds in-memory Parquet encoding helper encode_records_to_parquet.
crates/ourios-parquet/src/reader.rs	Adds open_bytes and factors baseline REQUIRED-column enforcement into require_baseline_columns.
crates/ourios-parquet/src/lib.rs	Re-exports encode_records_to_parquet.
crates/ourios-parquet/Cargo.toml	Adds direct bytes dependency to support Reader::open_bytes.
Cargo.lock	Records the added direct dependency usage.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated 1 comment.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/ourios-parquet/src/writer.rs`:
- Around line 606-637: Add a `#[cfg(test)]` module colocated with the
`encode_records_to_parquet` function in this file that includes both unit tests
and property-based tests. The unit tests should verify basic encoding behavior
(e.g., that records are correctly encoded to valid Parquet bytes and can be
decoded). Include at least one `proptest` case that verifies the row-group
sizing invariant mentioned in the comments — specifically that when in-progress
buffer size exceeds ROW_GROUP_FLUSH_BYTES, row groups are properly sealed and
flushed, preventing oversized row groups from being produced even with large
inputs. Additionally, add tests for edge cases like empty record batches and
verify round-trip invariants (encoding then decoding records preserves data
integrity).

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 0059404e-af1a-4acf-be15-db355b3c6653

📥 Commits

Reviewing files that changed from the base of the PR and between f7a2e13 and 970c6f3.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (5)

• crates/ourios-parquet/Cargo.toml
• crates/ourios-parquet/src/lib.rs
• crates/ourios-parquet/src/reader.rs
• crates/ourios-parquet/src/writer.rs
• crates/ourios-parquet/tests/buffer_and_put.rs

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.

[Copilot]

Pull request overview

Copilot reviewed 5 out of 6 changed files in this pull request and generated no new comments.