Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
138 changes: 138 additions & 0 deletions src/db/migrations/055_runtime_events_partition_drain.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,138 @@
-- 055_runtime_events_partition_drain.sql
-- Bug: genie_runtime_events_maintain_partitions never drained the DEFAULT
-- partition. Rows that landed in genie_runtime_events_default (e.g. inserts
-- after UTC midnight before the next maintenance call ran) permanently
-- blocked creation of new dated partitions, because PG validates that no
-- existing default-partition row would belong to the new partition. Symptom:
-- SQLSTATE 23514 "updated partition constraint for default partition
-- genie_runtime_events_default would be violated by some row" on every
-- subsequent `genie serve start`. Migration 038's docstring claimed the
-- nightly scheduler converted DEFAULT rows into named partitions; no such
-- code ever existed.
--
-- Fix:
-- 1. New helper genie_runtime_events_drain_default() detaches DEFAULT,
-- ensures dated partitions exist for every day represented by its rows,
-- re-inserts the rows (parent-routing them to the correct dated
-- partition) with notify-trigger suppression, truncates the now-empty
-- detached default, and re-attaches it as DEFAULT.
-- 2. genie_runtime_events_maintain_partitions calls drain_default() before
-- creating today..today+forward_days, so a stuck DEFAULT can no longer
-- poison subsequent partition creation.
-- 3. One-shot drain at migration time so existing installs unstick on the
-- first post-upgrade `genie serve start`.

CREATE OR REPLACE FUNCTION genie_runtime_events_drain_default()
RETURNS INTEGER AS $$
DECLARE
drained INTEGER := 0;
d_rec RECORD;
prev_role TEXT;
BEGIN
IF NOT EXISTS (
SELECT 1
FROM pg_class c
JOIN pg_namespace n ON n.oid = c.relnamespace
WHERE c.relname = 'genie_runtime_events_default'
AND n.nspname = current_schema()
) THEN
RETURN 0;
END IF;

IF NOT EXISTS (SELECT 1 FROM genie_runtime_events_default LIMIT 1) THEN
RETURN 0;
END IF;

-- Detach so the re-insert below routes to dated partitions instead of
-- looping back into DEFAULT. Non-CONCURRENTLY form is transaction-safe.
EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default';

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

This DETACH PARTITION call will fail if the function is re-run after a partial failure (e.g., if a previous execution was interrupted after detaching but before re-attaching). This would prevent the service from starting because the DEFAULT partition would remain detached and subsequent attempts to detach it would throw an error. It is safer to check if the table is currently a partition before attempting to detach it.

  -- Detach if it is currently a partition. If already detached (e.g. from a
  -- previous failed run), we proceed to drain the orphaned table.
  IF EXISTS (
    SELECT 1 FROM pg_inherits i
    JOIN pg_class c ON c.oid = i.inhrelid
    JOIN pg_class p ON p.oid = i.inhparent
    WHERE c.relname = 'genie_runtime_events_default'
      AND p.relname = 'genie_runtime_events'
  ) THEN
    EXECUTE 'ALTER TABLE genie_runtime_events DETACH PARTITION genie_runtime_events_default';
  END IF;


-- Ensure a dated partition exists for every day represented in the
-- detached default. create_partition is CREATE TABLE IF NOT EXISTS, so
-- pre-existing partitions are fine.
FOR d_rec IN
SELECT DISTINCT date_trunc('day', created_at)::DATE AS d
FROM genie_runtime_events_default
LOOP
PERFORM genie_runtime_events_create_partition(d_rec.d);
END LOOP;

-- Suppress AFTER INSERT triggers (notify_runtime_event_split, audit chain
-- triggers, etc.) for the drain. These rows already fired their notifies
-- on their original insert; re-firing would broadcast duplicates to every
-- LISTEN'er.
prev_role := current_setting('session_replication_role');
PERFORM set_config('session_replication_role', 'replica', true);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Setting session_replication_role to replica requires superuser privileges in PostgreSQL. This will cause the function to fail in environments where the application connects as a restricted user or on managed database services (like AWS RDS) that do not allow this setting. Since this function is called during genie serve start, it could prevent the service from starting. Consider if trigger suppression is strictly necessary, or if there is an alternative way to handle it (e.g., a session variable checked by the trigger).

Comment on lines +64 to +65

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Remove superuser-only GUC change from maintenance path

This uses session_replication_role='replica', which is not generally available to ordinary application roles. genie_runtime_events_maintain_partitions is granted to events_admin (created as NOINHERIT/non-superuser in 041), so in RBAC deployments the drain can fail with permission denied exactly when DEFAULT has rows, rebreaking startup-time partition maintenance. The drain should avoid this superuser-only toggle (or use a controlled SECURITY DEFINER path).

Useful? React with 👍 / 👎.


EXECUTE $sql$
INSERT INTO genie_runtime_events (
id, repo_path, subject, kind, source, agent, team, direction, peer,
text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id,
severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at
)
Comment on lines +68 to +72

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Preserve tenant_id when draining default-partition rows

The reinsert statement drops tenant_id by not selecting/inserting it, even though migration 041 added tenant isolation on genie_runtime_events with tenant_id defaulting to 'default'. When DEFAULT contains rows for non-default tenants, this drain rewrites them into the default tenant, which corrupts tenant scoping and can leak data across tenants. Add tenant_id to both column lists so drained rows keep their original tenant.

Useful? React with 👍 / 👎.

OVERRIDING SYSTEM VALUE
SELECT id, repo_path, subject, kind, source, agent, team, direction, peer,
text, data, thread_id, trace_id, parent_event_id, span_id, parent_span_id,
severity, schema_version, duration_ms, dedup_key, source_subsystem, created_at
FROM genie_runtime_events_default
Comment on lines +68 to +77

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The INSERT statement uses a hardcoded list of columns. This creates a maintenance burden: any future migration that adds a column to genie_runtime_events must also update this function, or data will be lost during the drain process. Since genie_runtime_events_default is guaranteed to have the same schema as its parent (even when detached), you can use SELECT * with OVERRIDING SYSTEM VALUE to make this more robust and maintainable.

    INSERT INTO genie_runtime_events
    OVERRIDING SYSTEM VALUE
    SELECT * FROM genie_runtime_events_default

$sql$;

GET DIAGNOSTICS drained = ROW_COUNT;

PERFORM set_config('session_replication_role', prev_role, true);

EXECUTE 'TRUNCATE genie_runtime_events_default';
EXECUTE 'ALTER TABLE genie_runtime_events ATTACH PARTITION genie_runtime_events_default DEFAULT';

RETURN drained;
END;
$$ LANGUAGE plpgsql;

CREATE OR REPLACE FUNCTION genie_runtime_events_maintain_partitions(
forward_days INTEGER DEFAULT 2,
retention_days INTEGER DEFAULT 30
)
RETURNS JSONB AS $$
DECLARE
drained INTEGER;
created INTEGER := 0;
dropped INTEGER := 0;
i INTEGER;
BEGIN
-- Drain DEFAULT first so any rows that accumulated there (UTC-midnight
-- rollover between maintenance calls) get routed to their proper dated
-- partitions before we try to CREATE the new ones.
drained := genie_runtime_events_drain_default();

FOR i IN 0..forward_days LOOP
PERFORM genie_runtime_events_create_partition((CURRENT_DATE + i)::DATE);
created := created + 1;
END LOOP;
SELECT genie_runtime_events_drop_old_partitions(retention_days) INTO dropped;
RETURN jsonb_build_object(
'created_or_present', created,
'dropped', dropped,
'drained_from_default', drained,
'next_rotation_at', (CURRENT_DATE + 1)::TIMESTAMPTZ
);
END;
$$ LANGUAGE plpgsql;

-- Mirror the role grants from migration 041 for the new helper. Conditional
-- so this works on installs that ran 041 before events_admin existed.
DO $$
BEGIN
IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'events_admin') THEN
EXECUTE 'GRANT EXECUTE ON FUNCTION genie_runtime_events_drain_default() TO events_admin';
END IF;
END$$;

DO $$
DECLARE
drained INTEGER;
BEGIN
drained := genie_runtime_events_drain_default();
IF drained > 0 THEN
RAISE NOTICE 'genie_runtime_events_default: drained % stuck row(s) into dated partitions', drained;
END IF;
END$$;
47 changes: 47 additions & 0 deletions src/db/migrations/observability-migrations.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -196,6 +196,53 @@ describe.skipIf(!DB_AVAILABLE)('Group 1 observability migrations', () => {
expect(deleteErr?.message).toMatch(/append-only/);
});

test('maintain_partitions drains rows stuck in DEFAULT (regression: SQLSTATE 23514)', async () => {
const sql = await getConnection();

// Pick a far-future date that no rolling-window partition covers.
// create_partition uses session-TZ midnight, so the date string suffices.
const stuckDate = '2099-12-31';

// Pre-clean any leftovers from a previous test run.
await sql`DELETE FROM genie_runtime_events WHERE kind = 'partition.drain.test'`;

// Insert a row whose created_at falls outside every dated partition. It
// must route to genie_runtime_events_default.
await sql`
INSERT INTO genie_runtime_events (repo_path, kind, source, agent, text, created_at)
VALUES ('test', 'partition.drain.test', 'test', 'test', 'stuck', ${`${stuckDate} 12:00:00+00`}::TIMESTAMPTZ)
`;

const beforeDefault = await sql<{ n: number }[]>`
SELECT count(*)::INT AS n FROM genie_runtime_events_default
WHERE kind = 'partition.drain.test'
`;
expect(beforeDefault[0].n).toBe(1);

const result = await sql<
Array<{ r: { created_or_present: number; drained_from_default: number } }>
>`SELECT genie_runtime_events_maintain_partitions(2, 30)::jsonb AS r`;
expect(result[0].r.drained_from_default).toBeGreaterThanOrEqual(1);

const afterDefault = await sql<{ n: number }[]>`
SELECT count(*)::INT AS n FROM genie_runtime_events_default
WHERE kind = 'partition.drain.test'
`;
expect(afterDefault[0].n).toBe(0);

const inDated = await sql<{ relname: string }[]>`
SELECT tableoid::regclass::TEXT AS relname
FROM genie_runtime_events
WHERE kind = 'partition.drain.test'
`;
expect(inDated.length).toBe(1);
expect(inDated[0].relname).toMatch(/genie_runtime_events_p20991231$/);

// Cleanup so the dated partition created above doesn't haunt the rolling
// window or the retention sweep on subsequent runs.
await sql`DELETE FROM genie_runtime_events WHERE kind = 'partition.drain.test'`;
});

test('Group 1 migrations are idempotent on re-apply', async () => {
// Re-run each migration body against the already-migrated schema and
// confirm the guarded statements are all no-ops.
Expand Down
Loading