feat(kg): automatic self-curation — durable coverage grows + duplicates auto-merge

[Weegy]

Summary

Follow-up to #317 (durable knowledge tier). #317 made curated knowledge surface reliably; this PR makes the durable-knowledge lifecycle self-curating and automatic on any deployment — no per-instance backfill or cleanup scripts. Two outcomes:

1. Knowledge the agent learns once stops being re-learned. Learned schema/conventions auto-promote to the durable tier and are rendered + trusted in full, so the agent stops re-running discovery tools (e.g. dynamics_describe) for structure it already knows.
2. Duplicates stop accumulating. High-confidence duplicate MemorableKnowledge is merged automatically (durable nodes protected) instead of only being flagged for an operator.

Background — the reported problem

On the live test instance (3rd session on the same topic) the operator saw:

• the agent re-discovering the Dynamics course schema every session (a full dynamics_describe/dynamics_query sweep) even though the schema was already in the knowledge graph;
• the KG marking the same facts NEW on every turn (the same schema stored 4+ times), plus agent monologue ("Ich schaue kurz in den Memory…") captured as a MemorableKnowledge;
• stale/irrelevant recall in the "earlier sessions" panel.

The ask was explicitly general, not data-specific: "on a live deployment this must happen automatically."

Root causes

#	Cause	Effect
A	Durable insight summaries were truncated to the fuzzy 300-char cap in both toRecalledInsight and renderRecallBlocks	agent saw a stub of the curated schema → re-ran discovery
B	Ingest hygiene (narration filter) was applied only to durable promotion, not to fuzzy MK creation	agent monologue stored as MK and re-surfaced every session
C	excerptExtractor + significanceScorer had no notion of schema/data-model/convention knowledge	learned schema classified as generic insight → never auto-promoted to durable → coverage never grew
D	The merge-candidate detector only flagged near-duplicates (MergeCandidate records) for an operator; it never resolved them	duplicates accumulate forever on an unattended deployment
E	buildOrchestrator forwarded autoPromote/autoPromoteThreshold but dropped autoPromoteDurableMinSignificance/Kinds	dynamic/registry agents (the default fallback) promoted only to the fuzzy tier — durable auto-promotion fired only for the static chatAgent@1

(E is the same class as the earlier "missing modelRouting config in registry-managed per-Agent orchestrators" fix.)

What this PR changes

1. Durable full-render + trust prompt (A)

• RecalledInsight gains a durable marker; durable insights render at 2000 chars (fuzzy stays 300) in both toRecalledInsight and renderRecallBlocks.
• Orchestrator system prompt: when curated schema/reference knowledge is in the recalled context, trust it and do not re-run discovery tools for those entities — go straight to data queries.

2. Ingest hygiene for all auto-harvest (B)

• passesIngestHygiene (narration regex) now gates every auto-harvested MK in promoteTurnIfSignificant, not just durable promotion. Agent narration is dropped entirely (reason: 'hygiene-skip'). Length is not a gate for fuzzy (short facts like "Preis 1200€" are kept); the durable tier keeps its own length floor.

3. Schema-aware classification → auto-durable (C)

• excerptExtractor and significanceScorer now recognise data-model / schema / domain conventions (entities, field names, entity-sets, joins) as kind=reference + high significance (≥0.85). Such turns auto-promote to durable (T3), so once the agent learns e.g. the booking/participant schema it becomes durable and is not re-discovered. Time-bound data snapshots ("29 courses next week") stay mid-significance insights.

4. Automatic merge of high-confidence duplicates (D)

• The merge detector (already fires on every MK create) now resolves pairs at or above autoMergeThreshold itself instead of only flagging. Aggressive default 0.90 (kg_auto_merge_threshold) so paraphrased re-statements merge too; the flag floor is lowered to the threshold so sub-0.95 near-dups actually surface. Safety: a manuallyAuthored (durable) node is never deleted; when exactly one side is durable it wins; both-durable is left for an operator; otherwise the older node wins. Config-gated (kg_auto_merge_enabled, default on) + startup telemetry.

5. Durable config reaches dynamic/registry agents (E)

• buildOrchestrator now threads autoPromoteDurableMinSignificance/autoPromoteDurableKinds into the per-agent Orchestrator (the value was already provided in OrchestratorDeps; it was being dropped). So every agent — including registry-built ones — self-curates the durable tier, not just chatAgent@1.

Config flags (all default to the new behaviour, all reversible)

Flag / env	Default	Effect
kg_auto_merge_enabled / KG_AUTO_MERGE_ENABLED	on	auto-resolve high-confidence dup pairs
kg_auto_merge_threshold	0.90	cosine floor for auto-merge (and the flag floor)
kg_durable_autopromote / KG_DURABLE_AUTOPROMOTE	on@0.85	T3 durable auto-promotion (from #317; now reaches dynamic agents)

Tests

12 files changed, +381/-20. New/extended:

• test/autoMerge.test.ts — durable-protected, durable-as-source, both-durable-skip, older-wins, flag-only-without-threshold.
• test/promoteTurnIfSignificant.test.ts — narration → hygiene-skip; short factual turn still stored.
• test/durableRecall.test.ts — durable insight renders full-length (not truncated) + durable flag set.

All merge / promotion / durable / recall / buildOrchestrator suites green; full workspace build + ESLint clean (node 22.22.3).

Verified live (omadia-test, on the dynamic fallback agent)

[promotion] PROMOTED … significance=0.95 kind=reference durable=YES        # schema → durable (C+E)
[merge-detector] flagged …55e5937a vs …c2370e63 cosine=0.928
[merge-detector] auto-merged cosine=0.928 kept=…55e5937a (retired the duplicate)   # auto-merge (D), durable kept

• durable MK count grew automatically as schema was learned (3 → 5);
• redundant dynamics_describe calls roughly halved (11 → 6) for a course query once the durable schema rendered in full and the trust-prompt applied;
• a course re-ask triggered auto-merge of a near-duplicate (cosine 0.928), durable protected.

Out of scope / follow-ups

• Plan recall is still lexical term-overlap (no durable/relevance tier) — the "stale plan surfaced" symptom is not addressed here; logical next slice.
• Pre-existing duplicate nodes on an already-polluted DB are auto-merged only when re-touched (auto-merge triggers on new MK creation); a one-time pass of the existing operator-triggered bulk merge-detect service can clear legacy cruft. Fresh deployments never accumulate it.
• content_hash exact-dedup remains unset (low value — the semantic auto-merge supersedes it for the observed paraphrase duplicates).