perf(api): derive entity edges from unit_entities instead of materializing them - #1766
Merged
Merged
Conversation
Drift picked up by the generate-docs-skill pre-commit hook — keeps skills/hindsight-docs/ in sync with the upstream hindsight-docs/ sources.
…izing them Stop writing link_type='entity' rows to memory_links and derive entity edges on demand in the /graph endpoint (from the unit_entities self-join recall already uses) and in /stats (by replicating the historical writer cap). Why: on the recall-perf-medium bench bank (10k units), entity rows were 53% of all memory_links — 345k rows, ~190 MB of table+index — and recall never read them (entity expansion in link_expansion_retrieval.py uses unit_entities, not memory_links). Retain was running a synchronous pairwise loop per shared entity to write rows nothing read; per-unit entity degree was uncapped (max 326 outgoing on a single unit), and overall per-unit total degree averaged 130 with a p99 of 462. Changes: - Drop Phase 3 entity-link build/insert from retain orchestrator. Keep entity_resolver.flush_pending_stats() so entity_cooccurrences (which feeds /entities/graph) still updates. - Delete build_entity_links_from_resolved, insert_entity_links_batch, MAX_LINKS_PER_ENTITY, EntityLink, Phase3Context, and the now-dead fetch_entity_unit_fanout op (PG + Oracle). - /graph: filter memory_links query to link_type <> 'entity'; broaden the existing observation-inferred entity-pair loop to cover all visible units; cap at 10 units per entity to bound hot entities. - /stats: split link_breakdown into a memory_links query (non-entity) and a unit_entities-based derivation for entity, sized to the historical writer cap so link_counts.entity stays in the same magnitude. - Migration e9b2c7d1f3a4: drop idx_memory_links_entity_covering and chunk-delete existing entity rows (PG + Oracle paths). - Tests: rewrite test_entity_links_creation and test_all_link_types_together to assert via /graph + /stats; assert no entity rows in memory_links. API response shapes (graph edges, stats link_counts/links_breakdown) are unchanged at the boundary, so SDKs and the control plane do not need to be regenerated.
The previous derivation kept only the first 10 units per entity before pairing, so any unit beyond #10 for a hot entity had zero entity edges in /graph — even though it shared the entity with many visible units. Switch to a sliding window: each unit links to its next N neighbors in the per-entity list. Every unit that shares an entity with another visible unit gets edges (its successors directly, predecessors via their pairs), and total edges stay bounded at ~N * cap per entity instead of N². Adds a regression test that retains 15 facts mentioning the same person and asserts every retained unit appears in at least one entity edge in /graph.
…on main #1762 landed e1b2c3d4f5a6_drop_unused_indexes between this PR opening and CI run, which also drops idx_memory_links_entity_covering. Our migration's down_revision still pointed at the prior head, leaving Alembic with two heads and tripping test_alembic_dag.test_single_head. Re-parent to e1b2c3d4f5a6 to unify the head. The DROP INDEX IF EXISTS line becomes a defensive no-op (since #1762 already dropped it), but is retained in case this migration runs against a snapshot taken before #1762.
nicoloboschi
force-pushed
the
perf-derive-entity-links-from-unit-entities
branch
from
May 26, 2026 16:22
2e32ce3 to
205901a
Compare
nicoloboschi
added a commit
that referenced
this pull request
May 27, 2026
- Merge entity-edge-derivation (#1766), unused-index drops (#1762), and async link recompute into a single "Graph Storage & Maintenance" section that leads with the ~50% storage reduction. - Merge "Targeted Consolidation by Scope" and "Consolidation Quality Rewrite" into one "Consolidation Improvements" section; drop prompt internals. - Rewrite the multilingual section at a higher level (concepts, not env vars) and link out to /developer/multilingual.
nicoloboschi
added a commit
that referenced
this pull request
May 27, 2026
* docs: add 0.7.0 changelog and release blog post Documents the 0.7.0 release: ParadeDB pg_search BM25 backend (Citus-compatible), PGroonga + configurable BM25 language for multilingual/CJK search, async link recompute that fixes outgoing-link staleness after deletes, Control Plane i18n in 8 locales, targeted consolidation by observation scope, an observation-consolidation prompt rewrite, a clear-mental-model endpoint, ZeroEntropy + Codex OAuth embeddings, and a long tail of bug fixes. Also fixes release.sh to refresh the root package-lock.json after workspace version bumps. Without this, npm ci in CI fails because the lock pins the previous workspace versions and the publish + docs-deploy jobs break (which is what happened to the initial v0.7.0 tag). * docs(blog): tighten 0.7.0 release post - Merge entity-edge-derivation (#1766), unused-index drops (#1762), and async link recompute into a single "Graph Storage & Maintenance" section that leads with the ~50% storage reduction. - Merge "Targeted Consolidation by Scope" and "Consolidation Quality Rewrite" into one "Consolidation Improvements" section; drop prompt internals. - Rewrite the multilingual section at a higher level (concepts, not env vars) and link out to /developer/multilingual. * docs(blog): rewrite 0.7.0 release post in announcement tone Rewrite each section in the same voice as prior major-release posts (0.5.0, 0.6.0): lead with what the user gets and why it matters, drop implementation internals (queue tables, FK cascades, JSON predicates, AST walkers), keep concrete config knobs and code examples where they help, and link out to docs for deep dives. * docs(blog): move ParadeDB section to last; reorder intro to match * docs(blog): demote Clear Mental Model from feature section to Other Notable Changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
link_type='entity'rows tomemory_links. Derive them on demand in/graph(from theunit_entitiesself-join recall already uses) and in/stats(replicating the historical writer cap solink_counts.entitystays in the same magnitude). API response shapes are unchanged — SDKs and the control plane do not need to be regenerated.idx_memory_links_entity_coveringand chunk-deletes existing entity rows (PG + Oracle paths).build_entity_links_from_resolved,insert_entity_links_batch,MAX_LINKS_PER_ENTITY,EntityLink,Phase3Context,fetch_entity_unit_fanout. Retain orchestrator keepsflush_pending_stats()soentity_cooccurrences(which feeds/entities/graph) still updates.Why
On the
recall-perf-mediumbench bank (10k units), entity rows were 53% of allmemory_links— 345k rows, ~190 MB of table+index — and recall never read them. Entity expansion inlink_expansion_retrieval.pyusesunit_entities, notmemory_links. Retain was running a synchronous pairwise loop per shared entity to write rows nothing read; per-unit entity degree was uncapped (one MU had 326 outgoing entity links), and overall per-unit total degree averaged 130 with a p99 of 462.Empirical per-unit link distribution on
recall-perf-mediumbefore this change:Total degree (in+out) per unit: avg 130, p99 462, max 741.
Test plan
./scripts/hooks/lint.sh(clean)uv run pytest tests/test_retain.py tests/test_delta_retain.py tests/test_consolidation.py tests/test_link_utils.py tests/test_memory_links_deferred_fk.py tests/test_migration_shape.py— 137 passtest_entity_links_creationandtest_all_link_types_togetherassert via/graph+/statsand verify no entity rows are written tomemory_links/graphand/statsresponses match the pre-migration baseline (modulo the documented stats approximation)bank-stats-view.tsx) still renders an entity slice after migrate + retain🤖 Generated with Claude Code