Skip to content

fix(partition): drain DEFAULT before creating dated partitions - #1435

Merged
namastex888 merged 1 commit into
devfrom
fix/partition-default-drain
Apr 28, 2026
Merged

namastex888 merged 1 commit into
devfrom
fix/partition-default-drain

Conversation

@namastex888

Copy link
Copy Markdown
Contributor

Summary

  • genie_runtime_events_maintain_partitions never drained the DEFAULT partition. Rows that landed in genie_runtime_events_default (e.g. inserts after UTC midnight before the next maintenance call) blocked creation of new dated partitions because PG validates that no DEFAULT row would belong to the new partition (SQLSTATE 23514). Every subsequent genie serve start then failed at the partition precondition.
  • Migration 038's docstring promised a "scheduler daemon nightly maintain_partitions" that converted DEFAULT rows into named partitions; that code never existed.
  • New migration 051 adds a drain_default() helper, replaces maintain_partitions to drain first, and runs a one-shot drain at migration time so existing stuck installs unstick on the first post-upgrade genie serve start.

Root cause

Reproduced on a live install:

PostgresError: updated partition constraint for default partition
"genie_runtime_events_default" would be violated by some row
  code: 23514, table_name: "genie_runtime_events_default"
  routine: "check_default_partition_contents"

Trigger sequence: UTC clock crosses midnight while genie is running → inserts route to DEFAULT (next dated partition not yet created) → next serve calls maintain_partitions(2,30) → CREATE TABLE … PARTITION OF parent FOR VALUES FROM ('YYYY-MM-DD 00:00:00+00') → PG scans DEFAULT, finds rows belonging to the new range, errors.

What changed

  • src/db/migrations/051_runtime_events_partition_drain.sql (new): genie_runtime_events_drain_default() detaches DEFAULT, ensures dated partitions exist for every day represented in its rows, re-inserts with session_replication_role=replica (suppresses notify-trigger fan-out — those rows already fired their notifies on original insert), truncates, re-attaches DEFAULT empty. maintain_partitions calls drain first. Mirrors 041's events_admin grant for the new helper.
  • src/db/migrations/observability-migrations.test.ts (+47): regression test inserts a backdated row (2099-12-31) into DEFAULT, runs maintain_partitions, asserts row moved to its dated partition and DEFAULT is empty.

Verification

Live-verified against the affected pgserve before opening this PR:

  • Migration applied at 2026-04-28 01:53 UTC.
  • genie_runtime_events_default row count post-drain: 0.
  • SELECT genie_runtime_events_maintain_partitions(2,30)::jsonb returns {"created_or_present":3,"dropped":0,"drained_from_default":0,"next_rotation_at":"2026-04-29T00:00:00+00:00"}.
  • genie serve partition precondition now passes on the same machine that was producing 23514 minutes earlier.

Test plan

  • bun test src/db/migrations/observability-migrations.test.ts (regression test exercises drain via maintain_partitions).
  • bun run check (typecheck + lint + dead-code + test).
  • Manual: insert a row into genie_runtime_events_default (any date outside the rolling window), run maintain_partitions(2,30), observe drained_from_default ≥ 1 in the JSON return and 0 rows in DEFAULT after.
  • Backwards compat: existing JSON-result consumer at src/term-commands/serve/ensure-ready.ts:148-156 only destructures created_or_present, dropped, next_rotation_at — the new drained_from_default key is additive.

Notes

  • maintain_partitions is only invoked from ensureServeReady at serve startup. Any host that crosses UTC midnight while running will accumulate DEFAULT rows until the next start. A follow-up to schedule maintenance more frequently (hourly) is worth tracking separately — out of scope here, but the bug now self-heals on next start instead of permanently bricking serve.

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Apr 28, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 23abedb5-0323-44d8-b77e-268682f28980

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/partition-default-drain

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to drain the DEFAULT partition of the genie_runtime_events table into specific dated partitions, resolving a bug where stray rows in the default partition blocked the creation of new partitions. The feedback identifies several critical issues: the DETACH PARTITION call lacks idempotency and may fail if a previous execution was interrupted, the use of session_replication_role requires superuser privileges which could lead to failures in restricted or managed database environments, and the hardcoded column list in the INSERT statement creates a maintenance risk that could be mitigated by using SELECT *.


-- Detach so the re-insert below routes to dated partitions instead of
-- looping back into DEFAULT. Non-CONCURRENTLY form is transaction-safe.
EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This DETACH PARTITION call will fail if the function is re-run after a partial failure (e.g., if a previous execution was interrupted after detaching but before re-attaching). This would prevent the service from starting because the DEFAULT partition would remain detached and subsequent attempts to detach it would throw an error. It is safer to check if the table is currently a partition before attempting to detach it.

  -- Detach if it is currently a partition. If already detached (e.g. from a
  -- previous failed run), we proceed to drain the orphaned table.
  IF EXISTS (
    SELECT 1 FROM pg_inherits i
    JOIN pg_class c ON c.oid = i.inhrelid
    JOIN pg_class p ON p.oid = i.inhparent
    WHERE c.relname = 'genie_runtime_events_default'
      AND p.relname = 'genie_runtime_events'
  ) THEN
    EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default';
  END IF;

-- on their original insert; re-firing would broadcast duplicates to every
-- LISTEN'er.
prev_role := current_setting('session_replication_role');
PERFORM set_config('session_replication_role', 'replica', true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Setting session_replication_role to replica requires superuser privileges in PostgreSQL. This will cause the function to fail in environments where the application connects as a restricted user or on managed database services (like AWS RDS) that do not allow this setting. Since this function is called during genie serve start, it could prevent the service from starting. Consider if trigger suppression is strictly necessary, or if there is an alternative way to handle it (e.g., a session variable checked by the trigger).

Comment on lines +68 to +77
INSERT INTO genie_runtime_events (
id, repo_path, subject, kind, source, agent, team, direction, peer,
text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id,
severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at
)
OVERRIDING SYSTEM VALUE
SELECT id, repo_path, subject, kind, source, agent, team, direction, peer,
text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id,
severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at
FROM genie_runtime_events_default

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The INSERT statement uses a hardcoded list of columns. This creates a maintenance burden: any future migration that adds a column to genie_runtime_events must also update this function, or data will be lost during the drain process. Since genie_runtime_events_default is guaranteed to have the same schema as its parent (even when detached), you can use SELECT * with OVERRIDING SYSTEM VALUE to make this more robust and maintainable.

    INSERT INTO genie_runtime_events
    OVERRIDING SYSTEM VALUE
    SELECT * FROM genie_runtime_events_default

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ee90d68f0

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +68 to +72
INSERT INTO genie_runtime_events (
id, repo_path, subject, kind, source, agent, team, direction, peer,
text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id,
severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve tenant_id when draining default-partition rows

The reinsert statement drops tenant_id by not selecting/inserting it, even though migration 041 added tenant isolation on genie_runtime_events with tenant_id defaulting to 'default'. When DEFAULT contains rows for non-default tenants, this drain rewrites them into the default tenant, which corrupts tenant scoping and can leak data across tenants. Add tenant_id to both column lists so drained rows keep their original tenant.

Useful? React with 👍 / 👎.

Comment on lines +64 to +65
prev_role := current_setting('session_replication_role');
PERFORM set_config('session_replication_role', 'replica', true);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove superuser-only GUC change from maintenance path

This uses session_replication_role='replica', which is not generally available to ordinary application roles. genie_runtime_events_maintain_partitions is granted to events_admin (created as NOINHERIT/non-superuser in 041), so in RBAC deployments the drain can fail with permission denied exactly when DEFAULT has rows, rebreaking startup-time partition maintenance. The drain should avoid this superuser-only toggle (or use a controlled SECURITY DEFINER path).

Useful? React with 👍 / 👎.

genie_runtime_events_maintain_partitions never drained the DEFAULT
partition. Rows that landed in genie_runtime_events_default (e.g.
inserts after UTC midnight before the next maintenance call) blocked
creation of new dated partitions: PG validates that no DEFAULT row
would belong to the new partition and raises SQLSTATE 23514. Every
subsequent `genie serve start` then failed during the partition
precondition. Migration 038 promised a "scheduler daemon nightly
maintain_partitions" that converted DEFAULT rows into named
partitions; that code never existed.

Fix:
- New helper genie_runtime_events_drain_default() detaches DEFAULT,
  ensures dated partitions exist for every day represented in its
  rows, re-inserts with notify-trigger suppression
  (session_replication_role=replica) so duplicate notifies don't
  fan out, truncates the now-empty detached default, and re-attaches
  it as DEFAULT.
- maintain_partitions calls drain_default first, then creates
  today..today+forward_days, then drops past retention as before.
- Migration runs an immediate one-shot drain so existing installs
  unstick on the first post-upgrade `genie serve start`.

Verified live against the affected pgserve: drain ran at migration
time, default partition is now empty, and maintain_partitions(2,30)
returns drained_from_default=0 on subsequent calls. Regression test
asserts a backdated row stuck in DEFAULT moves into its dated
partition and DEFAULT empties.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@namastex888
namastex888 force-pushed the fix/partition-default-drain branch from 1ee90d6 to e14de31 Compare April 28, 2026 05:01
@namastex888
namastex888 merged commit c1e31f8 into dev Apr 28, 2026
11 checks passed
@automagik-genie
automagik-genie deleted the fix/partition-default-drain branch September 25, 2026 04:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant