docs(rfc-0014): ingest write path — record sink & flush policy (drafted)

[jensholdgaard]

RFC 0014 (drafted) — the missing ingest write path. Production wires NoOpRecordSink, so mined records are dropped today; this RFC specifies how they become Parquet objects on the store.

The gap

Every ingest layer is built and tested (OTLP → WAL → miner; Parquet writer/reader; compaction; the RFC 0013 Store seam + buffer-and-put Writer) — but the miner emits into a RecordSink whose only production impl is NoOpRecordSink. No RFC specifies when mined records flush to a Parquet file. That flush policy is a §4 (small-file) / §3.4 (WAL-durability) / §3.7 (multi-tenancy) decision → RFC-gated per CLAUDE.md §5.1.

The design (per maintainer direction)

• Hybrid flush policy: per-partition size target (toward RFC 0005 §3.5's file band) or max age, and a force-flush on WAL segment rotation (RFC 0008) — aligns the published-Parquet horizon with a WAL boundary.
• Hard buffered-bytes ceiling with early-flush + backpressure — bounded memory, never an OOM liability.
• Reuses RFC 0008's batch/rotation machinery (one cadence, not a parallel timer).
• Records reach the sink post-WAL-durable, so an un-flushed buffer is always recoverable by replay (§3.4).

Scope (narrow)

Just the flush policy + the RecordSink. Follow-on (tracked as open questions): server constructs/injects a Store (RFC 0004 local-vs-S3), and compaction's manifest publish adopts Manifest::publish_cas on S3 — together those green RFC0013.6.

Status

drafted: §§1–4, 7, 8 filled; §5 acceptance (RFC0014.1–.6) + §6 testing sketched for the specified pass. Mermaid flow diagram included (mdbook build verified). Doc-only PR.

🤖 Generated with Claude Code

Summary by CodeRabbit

• Documentation
  • Added RFC 0014 documentation outlining the ingest write path architecture, including buffering and flush mechanisms for data persistence.

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 47 minutes and 59 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 65c382fe-ee1c-421c-8063-37ad4d5ae432

📥 Commits

Reviewing files that changed from the base of the PR and between 49a2089 and 96a7f5a.

📒 Files selected for processing (1)

• docs/rfcs/0014-ingest-write-path.md

📝 Walkthrough

Walkthrough

Adds RFC 0014, a 282-line design document specifying a ParquetRecordSink for the ingest write path. The RFC covers per-partition buffering, three flush triggers (size, age, WAL rotation), a global memory ceiling with backpressure, the WAL-based durability/crash-recovery contract, rejected alternatives, acceptance criteria, testing strategy, and open questions. A corresponding entry is added to docs/SUMMARY.md.

Changes

RFC 0014 — Ingest Write Path

Layer / File(s)	Summary
RFC proposal, motivation, and sink design docs/SUMMARY.md, docs/rfcs/0014-ingest-write-path.md	RFC metadata header, status/scope note, and high-level proposal added. Motivation covers completing the OTLP-to-queryable-Parquet path, small-file hazards, WAL-before-ack durability, and tenant isolation. Sink design specifies per-partition buffers keyed by PartitionKey, a Store handle, flush configuration, the three flush triggers (size threshold, oldest-record age, WAL-rotation force-flush), and the global buffered-bytes ceiling with early-flush soft pressure and backpressure hard limit.
Durability contract, alternatives, acceptance criteria, testing, and open questions docs/rfcs/0014-ingest-write-path.md	Crash-recovery contract states acknowledged data is preserved via WAL tail replay; flush is not independently crash-durable. "Not changed" list and end-to-end Mermaid flow diagram included. Rejected alternatives enumerate omitting rotation trigger, WAL-rotation-only flush, per-batch writer open/close, and no ceiling. Acceptance criteria map to each trigger and invariant. Testing strategy lists unit, property, and crash-recovery tests. Open questions cover defaults, victim selection, integration surface, backpressure signaling, size estimation, and follow-on server-wiring.

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Possibly related PRs

• jensholdgaard/ourios#64: RFC 0014's "forced flush on WAL segment rotation" and "unflushed acknowledged data preserved via WAL replay" behaviors are directly grounded in the WAL durability and ack semantics documented in RFC 0007.

Poem

🐇 A sink for records, partitioned with care,
Flush on size, on age, or when WAL segments tear.
The ceiling holds back any runaway bytes,
And WAL tail replay keeps the data alight.
No Parquet is lost — the rabbit ensures,
Every ack'd record safely endures! 🥕

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding RFC 0014 documentation on the ingest write path's record sink and flush policy.
Description check	✅ Passed	The description clearly explains the RFC's purpose, design rationale, scope, and status, though it doesn't follow the template structure with explicit Summary, Related, and Checklist sections.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch rfc0014-ingest-write-path-drafted

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[Copilot]

Pull request overview

Adds a new drafted RFC (0014) documenting the missing ingest write path from mined records to Parquet objects, including a proposed buffering RecordSink and a hybrid flush policy that ties size/age triggers to WAL segment rotation.

Changes:

• Add RFC 0014 (“Ingest write path — record sink and flush policy”) as a new doc.
• Link RFC 0014 from docs/SUMMARY.md so it appears in the mdBook navigation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 10 comments.

File	Description
docs/SUMMARY.md	Adds RFC 0014 to the RFC index for navigation.
docs/rfcs/0014-ingest-write-path.md	Introduces drafted RFC 0014 specifying the record sink + flush policy design and testing/acceptance sketches.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/rfcs/0014-ingest-write-path.md`:
- Line 7: The created date in the RFC metadata is set to 2026-06-17, which is
one day in the future when it should reflect today's date of 2026-06-16. Update
the created field from 2026-06-17 to 2026-06-16 to align the RFC metadata with
the actual draft date and maintain chronological consistency throughout the
document.
- Around line 103-105: The RFC document uses inconsistent language when
describing the max_buffer_age threshold, alternating between "reaches" and
"exceeds" without explicitly stating whether the boundary is inclusive or
exclusive. This ambiguity can lead to off-by-one errors during implementation.
Clarify the boundary semantics by choosing one consistent term (either "reaches"
for inclusive >= semantics or "exceeds" for exclusive > semantics) and apply it
uniformly across all sections that reference max_buffer_age, including both the
section at lines 103-105 (Age trigger description) and the section at lines
213-215 (also referenced in the comment). Ensure the chosen wording aligns with
the acceptance criterion that states the flush happens on the next tick.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: fc4e92d3-0fc6-40c5-83b0-fae6c591998b

📥 Commits

Reviewing files that changed from the base of the PR and between 24db263 and 49a2089.

📒 Files selected for processing (2)

• docs/SUMMARY.md
• docs/rfcs/0014-ingest-write-path.md

[Copilot]

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.