fix(db): auto-clean conversation_turn_nodes and orphaned conversations

[pacocartones]

Summary

• conversation_turn_nodes and agentic_conversations (migrations 155/156, feat(dashboard): agentic conversation tracking — v4, decoupled + storage-architecture concern resolved #10263) had no retention path at all: src/lib/db/cleanup.ts never referenced either table, there is no cascade FK, and nothing else deletes from them. The reporter's numbers in fix(backend): conversation_turn_nodes / agentic_conversations have no retention — unbounded storage.sqlite growth #12453 (1.15M node rows, ~775 MB of a 1.1 GB storage.sqlite after four days, ~190 MB/day) are the same failure class as feat(backend): auto-cleanup/upsert for telemetry tables to bound storage.sqlite #6848 / fix: startup cleanup ignores dashboard data-retention setting (always deletes logs older than 7d) #4354 / fix(db): pre-migration backups are never pruned — db_backups grew to 204 GB / 49k files #10421.
• The nodes are identity-only: the transcript view resolves each turn's display content from the call_logs row that last_correlation_id points at (156_conversation_turn_nodes.sql header). Once cleanupCallLogs purges that row the node can never render again, so both tables now follow the existing retention.callLogs window (src/types/databaseSettings.ts) instead of getting a knob of their own, as the issue proposes. No new dashboard setting, no new environment variable, and docs/reference/ENVIRONMENT.md is untouched because there is nothing new to document there.
• Two new functions registered at the end of runAutoCleanup: cleanupConversationTurnNodes() deletes nodes with last_seen_at before the cutoff through the shared deleteFromTableBefore helper (src/lib/db/cleanup/usagePurge.ts, so a database that predates migration 156 is a no-op rather than an error); cleanupAgenticConversations() then sweeps roots past the same cutoff that have no remaining node, with WHERE last_seen_at < ? AND NOT EXISTS (SELECT 1 FROM conversation_turn_nodes n WHERE n.conversation_id = agentic_conversations.id). The last_seen_at guard uses the index from migration 155 and keeps a root that createConversation wrote moments ago, before the same request inserted its nodes.
• Semantics are the ones the issue describes: deleting an old node only affects reconnect anchors, and a conversation resumed after the window mints a new id, which is already the documented anchor-miss behavior of resolveConversationId. Deliberately out of scope: the "Clear all" / resetUsageHistory paths (a separate, user-triggered contract), and a last_seen_at index on conversation_turn_nodes. The node DELETE is a table scan today because migration 156 defined no index on that column; it runs once per cleanup cycle, and I can send the index as a follow-up migration if the maintainer wants it.

Related Issues

• Closes fix(backend): conversation_turn_nodes / agentic_conversations have no retention — unbounded storage.sqlite growth #12453
• Related to feat(dashboard): agentic conversation tracking — v4, decoupled + storage-architecture concern resolved #10263
• Related to feat(backend): auto-cleanup/upsert for telemetry tables to bound storage.sqlite #6848

Validation

• Change type: DB (auto-cleanup lifecycle)
• Focused tests and category gates from the golden path: tests/unit/db-cleanup-conversation-nodes-12453.test.ts + telemetry-auto-cleanup-6848 + db-cleanup + db-cleanup-xp-audit-log 23/23, npm run check:db-rules OK, node scripts/check/check-complexity-ratchets.mjs --base-ref origin/release/v3.8.51 OK (0 violations), npm run check:changelog-integrity OK, npm run typecheck:core 0 errors
• npm run lint
• Reconciled with the current active release base release/v3.8.51; focused checks rerun afterward
• Production-code changes include a new or updated automated test in this PR
• SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Tests Added Or Updated

• tests/unit/db-cleanup-conversation-nodes-12453.test.ts (new, 4 cases, real SQLite adapter in a mkdtemp DATA_DIR, same pattern as telemetry-auto-cleanup-6848.test.ts): nodes older than retention.callLogs deleted and recent ones kept; stale orphan root swept while a stale-but-anchored root and a fresh root without nodes both stay; a chain whose nodes expire is swept in the same pass, and never before its nodes are gone; runAutoCleanup reports conversationTurnNodes and agenticConversations with the expected counts. Red on the base (4/4 fail: cleanupConversationTurnNodes is not a function, cleanupAgenticConversations is not a function, conversationTurnNodes missing from results), green with the change (4/4).

Coverage Notes

• src/lib/db/cleanup.ts: cleanupConversationTurnNodes, cleanupAgenticConversations and their registration in runAutoCleanup are exercised end-to-end by the new test file; the existing tests/unit/db-cleanup.test.ts and telemetry-auto-cleanup-6848.test.ts stay green.
• No touched file lost coverage.

Reviewer Notes

• Purely additive in cleanup.ts: two new functions inserted before runAutoCleanup and two new entries at the end of its results map. feat(teams): billing cost centers, soft budgets and dashboard #10409 also edits this file, but in different hunks (cleanupMcpAudit, cleanupA2aEvents, resetUsageHistory), so the two should merge cleanly in either order. feat(dashboard): parent-link, genuine-continuation badge, and modal perf fixes #12448 edits src/lib/db/agenticConversations.ts, which this PR does not touch.
• Behavior change for operators: with the default retention.callLogs (30 days), conversations idle for longer than that lose their reconnect anchor and start a new conversation id on resume. Operators who set a short call-log retention (the reporter runs 3 days) get the same window for anchors, which is the trade-off the issue asks for.
• The node DELETE scans conversation_turn_nodes (no last_seen_at index in migration 156). On the reporter's 1.15M-row table that is one pass per cleanup cycle; if that is a concern I will add the index as a follow-up migration rather than widen this PR.

[insoln]

Review: schema, semantics, pipeline, SQL, tests — verified locally

Ran tests/unit/db-cleanup-conversation-nodes-12453.test.ts (4/4 pass, ~3 s), npm run lint (clean), and reproduced the reported scale from #12453 in a scratch DB with migration 156's exact schema (1,155,364 rows, 758.9 MB). The core fix is correct: both migration-comment claims check out (155 does index agentic_conversations.last_seen_at; 156 has no last_seen_at index), every writer of both tables' last_seen_at uses toISOString() so the string < comparisons are sound, the NOT EXISTS probe is a covering-index seek (idx_turn_nodes_conversation, measured 0.9 s for 36k conversations), and all readers (reconnect walk, tree route, conversation list) already degrade gracefully to the documented anchor-miss path. Scheduler wiring is fine: startCleanupScheduler() runs runAutoCleanup() at startup+6h with autoCleanupEnabled defaulting to true, and the VACUUM in the scheduler does reclaim the disk (measured 758.9 → 71.8 MB).

Blocking

1. Unbatched 1M-row DELETE freezes the whole server for ~36 s (measured).
cleanupConversationTurnNodes → deleteFromTableBefore (src/lib/db/cleanup/usagePurge.ts:50-52) is one synchronous DELETE ... WHERE last_seen_at < ? with SCAN conversation_turn_nodes as the query plan. On a replica of the reporter's table (1,155,364 rows / 758.9 MB, WAL, pragmas as in src/lib/db/core.ts): 1,039,828 rows deleted in 36.0 s with the Node event loop stalled the entire 36.0 s (a 10 ms timer fired zero times during the delete). This runs 30 s after startup, so a first deploy onto the reporter's already-bloated DB is 36 s of total outage — every API request, SSE chunk, and /health probe hangs. core.ts's own comment (:1276) documents that better-sqlite3 writes park the event loop. Suggested fix: chunk it, e.g. loop DELETE FROM conversation_turn_nodes WHERE rowid IN (SELECT rowid FROM conversation_turn_nodes WHERE last_seen_at < ? LIMIT 10000) with setImmediate yields between chunks, summing changes. (A last_seen_at index doesn't help — it's a delete-most-of-table scan either way; batching is the right lever. Note the follow-up VACUUM is another synchronous stall, but that's pre-existing scheduler behavior.)

Should-fix

2. "Clear all" doesn't clear these tables (issue item 3). RESET_TARGETS (src/lib/db/cleanup.ts:695-737) lists 12 tables; conversation_turn_nodes and agentic_conversations are absent. A user with the reported bloat who clicks Reset usage data — All wipes call_logs (the nodes' display-content anchor) but keeps the 775 MB of nodes — doubly dead data that then waits up to 30 days for the retention sweep. The PR body declares this out of scope; I'd argue the "Clear all" path is exactly where it belongs since that's the documented escape hatch for the reported state. Happy to be overruled if a follow-up is planned.

3. CALL_LOG_RETENTION_DAYS does not control this window — the reporter's setting has no effect here. The new functions read getUserDatabaseSettings().retention.callLogs (dashboard DB setting, default 30 days; src/types/databaseSettings.ts:120). The env var is consumed only by the compliance path (src/lib/compliance/index.ts:496) and never reaches getUserDatabaseSettings (zero process.env reads in databaseSettings.ts). #12453's reporter runs CALL_LOG_RETENTION_DAYS=3 and will still get a 30-day node window (~5–6 GB steady-state at their ~190 MB/day) unless they also change the dashboard setting. Pre-existing coupling for call_logs itself, but this PR makes it the only knob for two more tables without documenting it — worth at least a Reviewer Notes line or a docs touch in docs/reference/ENVIRONMENT.md next to the CALL_LOG_RETENTION_DAYS row (issue item 4).

Nits

• cleanupAgenticConversations' doc comment credits the NOT EXISTS guard with protecting a freshly-created root, but a root written moments ago has last_seen_at = now and can't match last_seen_at < cutoff anyway — the last_seen_at predicate alone excludes it. The guard is fine; the stated reason isn't quite the real one.
• The tableExists early-return in cleanupAgenticConversations (cleanup.ts:489-491) is unique new logic with no test — a cheap DROP TABLE subtest would pin it.
• Test nit: OLD = -(R+10), RECENT = -1 silently assumes retention ≥ 2 days.

Everything else — typing of runAutoCleanup results (untyped record, totals include the new keys), scheduler defaults, dbstat-based size display picking the tables up automatically, test isolation (mkdtempSync override beats the harness default) — verified fine.

Verdict: the retention semantics are correct and well-tested; the event-loop freeze in finding 1 is the one thing I'd want addressed before this merges onto a production DB of the reported size.