Repository navigation
fix(partition): drain DEFAULT before creating dated partitions - #1435
Conversation
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to drain the DEFAULT partition of the genie_runtime_events table into specific dated partitions, resolving a bug where stray rows in the default partition blocked the creation of new partitions. The feedback identifies several critical issues: the DETACH PARTITION call lacks idempotency and may fail if a previous execution was interrupted, the use of session_replication_role requires superuser privileges which could lead to failures in restricted or managed database environments, and the hardcoded column list in the INSERT statement creates a maintenance risk that could be mitigated by using SELECT *.
|
|
||
| -- Detach so the re-insert below routes to dated partitions instead of | ||
| -- looping back into DEFAULT. Non-CONCURRENTLY form is transaction-safe. | ||
| EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default'; |
There was a problem hiding this comment.
This DETACH PARTITION call will fail if the function is re-run after a partial failure (e.g., if a previous execution was interrupted after detaching but before re-attaching). This would prevent the service from starting because the DEFAULT partition would remain detached and subsequent attempts to detach it would throw an error. It is safer to check if the table is currently a partition before attempting to detach it.
-- Detach if it is currently a partition. If already detached (e.g. from a
-- previous failed run), we proceed to drain the orphaned table.
IF EXISTS (
SELECT 1 FROM pg_inherits i
JOIN pg_class c ON c.oid = i.inhrelid
JOIN pg_class p ON p.oid = i.inhparent
WHERE c.relname = 'genie_runtime_events_default'
AND p.relname = 'genie_runtime_events'
) THEN
EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default';
END IF;| -- on their original insert; re-firing would broadcast duplicates to every | ||
| -- LISTEN'er. | ||
| prev_role := current_setting('session_replication_role'); | ||
| PERFORM set_config('session_replication_role', 'replica', true); |
There was a problem hiding this comment.
Setting session_replication_role to replica requires superuser privileges in PostgreSQL. This will cause the function to fail in environments where the application connects as a restricted user or on managed database services (like AWS RDS) that do not allow this setting. Since this function is called during genie serve start, it could prevent the service from starting. Consider if trigger suppression is strictly necessary, or if there is an alternative way to handle it (e.g., a session variable checked by the trigger).
| INSERT INTO genie_runtime_events ( | ||
| id, repo_path, subject, kind, source, agent, team, direction, peer, | ||
| text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id, | ||
| severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at | ||
| ) | ||
| OVERRIDING SYSTEM VALUE | ||
| SELECT id, repo_path, subject, kind, source, agent, team, direction, peer, | ||
| text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id, | ||
| severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at | ||
| FROM genie_runtime_events_default |
There was a problem hiding this comment.
The INSERT statement uses a hardcoded list of columns. This creates a maintenance burden: any future migration that adds a column to genie_runtime_events must also update this function, or data will be lost during the drain process. Since genie_runtime_events_default is guaranteed to have the same schema as its parent (even when detached), you can use SELECT * with OVERRIDING SYSTEM VALUE to make this more robust and maintainable.
INSERT INTO genie_runtime_events
OVERRIDING SYSTEM VALUE
SELECT * FROM genie_runtime_events_defaultThere was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 1ee90d68f0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| INSERT INTO genie_runtime_events ( | ||
| id, repo_path, subject, kind, source, agent, team, direction, peer, | ||
| text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id, | ||
| severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at | ||
| ) |
There was a problem hiding this comment.
Preserve tenant_id when draining default-partition rows
The reinsert statement drops tenant_id by not selecting/inserting it, even though migration 041 added tenant isolation on genie_runtime_events with tenant_id defaulting to 'default'. When DEFAULT contains rows for non-default tenants, this drain rewrites them into the default tenant, which corrupts tenant scoping and can leak data across tenants. Add tenant_id to both column lists so drained rows keep their original tenant.
Useful? React with 👍 / 👎.
| prev_role := current_setting('session_replication_role'); | ||
| PERFORM set_config('session_replication_role', 'replica', true); |
There was a problem hiding this comment.
Remove superuser-only GUC change from maintenance path
This uses session_replication_role='replica', which is not generally available to ordinary application roles. genie_runtime_events_maintain_partitions is granted to events_admin (created as NOINHERIT/non-superuser in 041), so in RBAC deployments the drain can fail with permission denied exactly when DEFAULT has rows, rebreaking startup-time partition maintenance. The drain should avoid this superuser-only toggle (or use a controlled SECURITY DEFINER path).
Useful? React with 👍 / 👎.
genie_runtime_events_maintain_partitions never drained the DEFAULT partition. Rows that landed in genie_runtime_events_default (e.g. inserts after UTC midnight before the next maintenance call) blocked creation of new dated partitions: PG validates that no DEFAULT row would belong to the new partition and raises SQLSTATE 23514. Every subsequent `genie serve start` then failed during the partition precondition. Migration 038 promised a "scheduler daemon nightly maintain_partitions" that converted DEFAULT rows into named partitions; that code never existed. Fix: - New helper genie_runtime_events_drain_default() detaches DEFAULT, ensures dated partitions exist for every day represented in its rows, re-inserts with notify-trigger suppression (session_replication_role=replica) so duplicate notifies don't fan out, truncates the now-empty detached default, and re-attaches it as DEFAULT. - maintain_partitions calls drain_default first, then creates today..today+forward_days, then drops past retention as before. - Migration runs an immediate one-shot drain so existing installs unstick on the first post-upgrade `genie serve start`. Verified live against the affected pgserve: drain ran at migration time, default partition is now empty, and maintain_partitions(2,30) returns drained_from_default=0 on subsequent calls. Regression test asserts a backdated row stuck in DEFAULT moves into its dated partition and DEFAULT empties. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
1ee90d6 to
e14de31
Compare
Summary
genie_runtime_events_maintain_partitionsnever drained the DEFAULT partition. Rows that landed ingenie_runtime_events_default(e.g. inserts after UTC midnight before the next maintenance call) blocked creation of new dated partitions because PG validates that no DEFAULT row would belong to the new partition (SQLSTATE 23514). Every subsequentgenie serve startthen failed at the partition precondition.drain_default()helper, replacesmaintain_partitionsto drain first, and runs a one-shot drain at migration time so existing stuck installs unstick on the first post-upgradegenie serve start.Root cause
Reproduced on a live install:
Trigger sequence: UTC clock crosses midnight while genie is running → inserts route to DEFAULT (next dated partition not yet created) → next
servecallsmaintain_partitions(2,30)→CREATE TABLE … PARTITION OF parent FOR VALUES FROM ('YYYY-MM-DD 00:00:00+00')→ PG scans DEFAULT, finds rows belonging to the new range, errors.What changed
src/db/migrations/051_runtime_events_partition_drain.sql(new):genie_runtime_events_drain_default()detaches DEFAULT, ensures dated partitions exist for every day represented in its rows, re-inserts withsession_replication_role=replica(suppresses notify-trigger fan-out — those rows already fired their notifies on original insert), truncates, re-attaches DEFAULT empty.maintain_partitionscalls drain first. Mirrors 041'sevents_admingrant for the new helper.src/db/migrations/observability-migrations.test.ts(+47): regression test inserts a backdated row (2099-12-31) into DEFAULT, runsmaintain_partitions, asserts row moved to its dated partition and DEFAULT is empty.Verification
Live-verified against the affected pgserve before opening this PR:
genie_runtime_events_defaultrow count post-drain: 0.SELECT genie_runtime_events_maintain_partitions(2,30)::jsonbreturns{"created_or_present":3,"dropped":0,"drained_from_default":0,"next_rotation_at":"2026-04-29T00:00:00+00:00"}.genie servepartition precondition now passes on the same machine that was producing 23514 minutes earlier.Test plan
bun test src/db/migrations/observability-migrations.test.ts(regression test exercises drain viamaintain_partitions).bun run check(typecheck + lint + dead-code + test).genie_runtime_events_default(any date outside the rolling window), runmaintain_partitions(2,30), observedrained_from_default≥ 1 in the JSON return and 0 rows in DEFAULT after.src/term-commands/serve/ensure-ready.ts:148-156only destructurescreated_or_present,dropped,next_rotation_at— the newdrained_from_defaultkey is additive.Notes
maintain_partitionsis only invoked fromensureServeReadyat serve startup. Any host that crosses UTC midnight while running will accumulate DEFAULT rows until the next start. A follow-up to schedule maintenance more frequently (hourly) is worth tracking separately — out of scope here, but the bug now self-heals on next start instead of permanently bricking serve.🤖 Generated with Claude Code