feat(graph): rfc 0047 slice 4 — emitter + erasure (RFC0047.10–.11), RFC green

[jensholdgaard]

Summary

RFC 0047 implementation slice 4 of 4 — the graph emitter and conversation erasure (RFC0047.10–.11, §3.3 / §3.6). Follows #705, #706, #707. With this, all twelve §5 criteria pass and the RFC moves to green (validated/accepted are yours).

• parquet — compact_partition_hooked threads RowHooks through the rewrite: observe sees every input row once as decoded; drop removes rows from the consolidated output (rewriting even a single-file partition so an erasure lands; CompactionOutcome.rows_dropped). New audit kind 9 conversation_erased with four OPTIONAL erasure_* columns (RFC 0005 §3.7 table rows 6–9 added — 6–8 were already in code but missing from the table; schema pin + round-trip updated).
• core — OpenFgaClient::read_by_object (paginated Read, bounded); TenantObjects::conversation_fits and an encoded-length bound in TenantObjects::new (Copilot's feat(auth): rfc 0047 slice 2 — planner two-step + visibility (RFC0047.4–.8) #706 follow-up: percent-encoding can grow the tenant segment past 256 B). MCP_TOOL_NAMES shared vocabulary.
• ingester — graph_emitter::GraphEmitter: derives the §3.3 tuples (conversation:T/<id>#parent@tenant:T, #participant@user:<user.hash|enduser.pseudo.id> + tenant:T#scoped_reader@user:…, #actor@agent:<gen_ai.agent.id> + binding, and the per-tenant tool:T/<name>#parent objects so operators grant caller only); values that cannot be object ids are skipped; writes in ≤100-tuple on_duplicate=ignore batches, counted on ourios.graph.tuples{ourios.graph.tuple.operation} (registry + weaver regen). Fed from both hooks: the compaction sweep (SweepHooks observe on every row it decodes) and the receiver's PublishCoordinator after each durable batch (off the flush path). sweep_once is the daemon's per-tick body (blocking pass → async graph phase → blocking audit/marker phase), used by Compactor::run and the tests.
• erasure (§3.6) — the RFC left the request channel open; the choice here is a durable marker object in the store (erasure/tenant_id=<enc>/conversation=<enc>, request_erasure / pending_erasures): the object store is the source of truth (§3.6), no new network surface or credential, and an operator can write it with tooling they have. The sweep rewrites every partition of the tenant with the conversation's rows dropped, advances the marker to the tuples phase, then — after the blocking pass — reads and deletes the object's tuples, writes the conversation_erased audit event after the sweep's compaction events, and removes the marker. Interrupted sweeps retry only the tuple deletion; an unreachable graph leaves the marker; without a bound conversation object a marker is a recorded sweep error, never silently dropped.
• server — the emitter is built at startup (when a conversation object is bound) and wired into the receiver's coordinator and the compactor; a compactor-only pod gets it too.
• tests — fake-backed unit tests (derivation table; sweep emits ≤100-tuple chunks and a second sweep sends nothing; erasure drops rows → deletes tuples → audit event last → marker gone, other conversations untouched); RFC0047.10/.11 on a real openfga/openfga container, including the end-to-end proof that emitted tuples bind a participant on the served binary (no operator-written conversation tuples anywhere) and that a tenant-wide reader no longer sees the erased rows. The openfga-resolver CI job runs all three container tests. Green locally.
• Docs: authentication guide (feeding the graph, erasing a conversation); RFC 0047 §3.3/§3.6 slice-4 decisions, §9 follow-ons (backfill sweep, operator-facing erasure surface), status → green; RFC 0005 §3.7 amendment row.

Decision to flag (RFC-level, chosen here)

The erasure request channel (store marker, above) was not specified by the RFC. Alternatives were an admin HTTP endpoint (new surface + auth) or a programmatic-only API (not operator-usable). If you'd rather have a CLI verb / MCP tool over the marker, that's the §9 follow-on.

Invariants / hazards touched

• §3.5 Parquet schema — the audit schema gains four OPTIONAL columns (additive; readers tolerate absent columns; RFC 0005 §3.7 amendment + schema pin updated). Data schema unchanged.
• §3.6 object storage is truth — the erasure request lives in the store; tuples follow rows.
• §3.7 multi-tenancy — the emitter is tenant-scoped through the partition key; object naming is the single TenantObjects rule.
• §3.4 WAL-before-ack / hot path — nothing is added to the ack path; the flush-cadence hook runs after the batch is durable, fire-and-forget.
• hazard docs: apply RFC maturity-model amendments #4 — erasure rewrites are per-partition through the existing compaction writer (row-group sizing unchanged).

Related

RFC: docs/rfcs/0047-rebac-resolver-and-graph-visibility.md (spec #704; slices #705, #706, #707).

Checklist

• cargo fmt clean
• cargo clippy clean (no new warnings)
• Tests added/updated
• Docs updated
• RFC linked

🤖 Generated with Claude Code

https://claude.ai/code/session_01JZXtbyWoQY19ZGtNecDfgv

[coderabbitai]

Warning

Review limit reached

@jensholdgaard, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 23 minutes

Limit details: You’ve used all 1 included review currently available under your plan.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e21991c5-7ea5-42f3-bdbc-538f2d8d3d6f

📥 Commits

Reviewing files that changed from the base of the PR and between 5dadcc2 and 86dc67a.

📒 Files selected for processing (28)

• .github/workflows/ci.yml
• crates/ourios-core/src/alias.rs
• crates/ourios-core/src/audit.rs
• crates/ourios-core/src/auth/openfga/client.rs
• crates/ourios-core/src/auth/openfga/mod.rs
• crates/ourios-ingester/src/compactor.rs
• crates/ourios-ingester/src/graph_emitter.rs
• crates/ourios-ingester/src/lib.rs
• crates/ourios-ingester/src/metrics.rs
• crates/ourios-ingester/src/publish.rs
• crates/ourios-parquet/src/audit_reader.rs
• crates/ourios-parquet/src/audit_record_batch.rs
• crates/ourios-parquet/src/compaction.rs
• crates/ourios-parquet/src/lib.rs
• crates/ourios-parquet/tests/it/audit_round_trip.rs
• crates/ourios-parquet/tests/it/schema_pin.rs
• crates/ourios-semconv/src/lib.rs
• crates/ourios-server/src/main.rs
• crates/ourios-server/src/receiver.rs
• crates/ourios-server/tests/it/main.rs
• crates/ourios-server/tests/it/rfc0047_9_tool_gate.rs
• crates/ourios-server/tests/it/rfc0047_emitter.rs
• crates/ourios-server/tests/it/rfc0047_visibility.rs
• docs/guides/authentication.md
• docs/rfcs/0005-parquet-storage.md
• docs/rfcs/0047-rebac-resolver-and-graph-visibility.md
• semconv/registry/attributes.yaml
• semconv/registry/metrics.yaml

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Implements RFC 0047 slice 4 by adding a data-fed OpenFGA graph emitter (flush cadence + compaction sweep) and durable conversation erasure (store marker → row rewrite → tuple delete → audit), plus the supporting semconv, audit schema, and CI coverage to move RFC 0047 to green.

Changes:

• Add GraphEmitter and wire it into the receiver publish cadence and compaction sweeps; add ourios.graph.tuples{ourios.graph.tuple.operation} metric.
• Add erasure request markers in object storage and a sweep flow that rewrites partitions to drop rows, then deletes OpenFGA tuples, then emits a conversation_erased audit event (new kind 9 + optional erasure_* columns).
• Add unit + real-container integration tests (and CI job coverage) for RFC0047.10–.11.

Reviewed changes

Copilot reviewed 28 out of 28 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
semconv/registry/metrics.yaml	Adds ourios.graph.tuples counter definition.
semconv/registry/attributes.yaml	Adds ourios.graph.tuple.operation attribute enum.
docs/rfcs/0047-rebac-resolver-and-graph-visibility.md	Marks RFC 0047 as green and records slice-4 decisions.
docs/rfcs/0005-parquet-storage.md	Updates audit-kind mapping table with kind 9 conversation_erased.
docs/guides/authentication.md	Documents graph feeding and erasure marker workflow for operators.
crates/ourios-server/tests/it/rfc0047_visibility.rs	Exposes helper fns to reuse in new emitter/erasure E2E tests.
crates/ourios-server/tests/it/rfc0047_emitter.rs	Adds real-OpenFGA container E2E test for emitter + erasure ordering.
crates/ourios-server/tests/it/rfc0047_9_tool_gate.rs	Updates scenario description to reflect tenant-wide bypass behavior.
crates/ourios-server/tests/it/main.rs	Registers the new RFC0047 emitter/erasure integration test module.
crates/ourios-server/src/receiver.rs	Threads optional graph emitter into the publish coordinator.
crates/ourios-server/src/main.rs	Builds graph emitter at startup (when configured) and wires it into receiver + compactor.
crates/ourios-semconv/src/lib.rs	Adds constants for the new metric and attribute key.
crates/ourios-parquet/tests/it/schema_pin.rs	Pins audit schema to include new optional erasure_* columns.
crates/ourios-parquet/tests/it/audit_round_trip.rs	Adds round-trip test for conversation_erased audit events.
crates/ourios-parquet/src/lib.rs	Exposes RowHooks, compact_partition_hooked, hour_partitions; adds audit column constants.
crates/ourios-parquet/src/compaction.rs	Implements RowHooks (observe + drop) and rows_dropped reporting; exports hour_partitions.
crates/ourios-parquet/src/audit_record_batch.rs	Writes new conversation_erased payload columns into audit batches.
crates/ourios-parquet/src/audit_reader.rs	Reads/decodes new conversation_erased payload from erasure_* columns (tolerant of absent columns).
crates/ourios-ingester/src/publish.rs	Derives and emits graph tuples asynchronously after durable publish (flush-cadence bridge).
crates/ourios-ingester/src/metrics.rs	Updates sweep-report test fixture for new report fields.
crates/ourios-ingester/src/lib.rs	Exposes graph_emitter module behind openfga feature.
crates/ourios-ingester/src/graph_emitter.rs	Adds GraphEmitter (derive tuples, emit in chunks, erase by read+delete) + unit tests.
crates/ourios-ingester/src/compactor.rs	Adds erasure marker workflow, sweep hooks, async graph phase, and audit emission for erasure completion.
crates/ourios-core/src/auth/openfga/mod.rs	Adds MCP_TOOL_NAMES and strengthens tenant/object-id length constraints.
crates/ourios-core/src/auth/openfga/client.rs	Adds bounded, paginated read_by_object for erasure tuple deletion.
crates/ourios-core/src/audit.rs	Adds ConversationErased audit payload + kind/type constants (kind 9).
crates/ourios-core/src/alias.rs	Ensures alias-map audit replay ignores the new audit payload kind.
.github/workflows/ci.yml	Extends openfga-resolver job to run RFC0047.10–.11 container test.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated 1 comment.

Suppressed comments (2)

crates/ourios-ingester/src/compactor.rs:842

• If a sweep crashes after advancing a marker to the tuples phase (rows already dropped) but before graph_phase completes, the next sweep will see ErasurePhase::Tuples and emit a conversation_erased audit event with partitions_rewritten = 0 / rows_dropped = 0 (because the rows phase is skipped). This makes the audit payload incorrect for the interrupted-sweep case described in the RFC/PR description.

                        payload: AuditPayload::ConversationErased {
                            conversation_id: request.conversation_id.clone(),
                            partitions_rewritten: outcome.partitions_rewritten,
                            rows_dropped: outcome.rows_dropped,
                            tuples_deleted: to_u64(deleted),

crates/ourios-core/src/auth/openfga/client.rs:461

• Typo in error context: this formats "read read: {e}", which is confusing in logs/errors. It should describe the actual failing step (reading response bytes).

                .map_err(|e| OpenFgaError::Unavailable(format!("read read: {e}")))?;

[Copilot]

Pull request overview

Copilot reviewed 28 out of 28 changed files in this pull request and generated no new comments.

Suppressed comments (2)

crates/ourios-ingester/src/compactor.rs:522

• erase_pending performs potentially large Parquet rewrites, but on a committed rewrite it only updates the per-erasure outcome. The sweep-level accounting (SweepReport.bytes_read, compacted_files → bytes_written, partitions_compacted / files_compacted / rows_compacted, and compaction_events) is not updated for these rewrites, so CompactionMetrics::record_sweep under-reports IO and the audit trail won’t show which files were rewritten as part of an erasure.

Consider recording erasure-triggered rewrites in the sweep report (and emitting compaction audit events for them), or introduce separate erasure IO/rewrites metrics if you intentionally want to exclude them from compaction metrics.

                        if o.committed.is_some() {
                            outcome.partitions_rewritten += 1;
                            outcome.rows_dropped += o.rows_dropped;
                        }

crates/ourios-core/src/auth/openfga/client.rs:462

• The error string "read read: {e}" looks like a copy/paste typo and will be confusing in logs/telemetry. It should describe the failing step (reading the response bytes).

            let bytes = response
                .bytes()
                .await
                .map_err(|e| OpenFgaError::Unavailable(format!("read read: {e}")))?;
            let page: ReadResponse = serde_json::from_slice(&bytes)