chore(db): drop 9 unused/redundant indexes - #1762
Merged
Merged
Conversation
nicoloboschi
changed the title
…ndexes Code audit identified 9 indexes on memory_links, entities, documents, and unit_entities that are either dead (no code path exercises them) or fully covered by composite indexes the planner already prefers. See the migration docstring for the per-index rationale. Also fixes two stale comments that referenced indexes which no longer match the code paths: - link_expansion_retrieval.py claimed entity expansion uses idx_memory_links_entity_covering, but the CTE traverses unit_entities, not memory_links — that's why the covering index has no code path exercising it. - memory_engine.py referenced idx_memory_links_bank_link_type, which was never created on PostgreSQL (only the bank_id column exists). The skills/hindsight-docs/ regen is a drive-by from the pre-commit hook catching up with embeddings-provider docs that landed on main earlier.
nicoloboschi
force-pushed
the
drop-unused-prod-indexes
branch
from
May 26, 2026 13:50
a63bb66 to
2321a85
Compare
nicoloboschi
added a commit
that referenced
this pull request
May 26, 2026
…on main #1762 landed e1b2c3d4f5a6_drop_unused_indexes between this PR opening and CI run, which also drops idx_memory_links_entity_covering. Our migration's down_revision still pointed at the prior head, leaving Alembic with two heads and tripping test_alembic_dag.test_single_head. Re-parent to e1b2c3d4f5a6 to unify the head. The DROP INDEX IF EXISTS line becomes a defensive no-op (since #1762 already dropped it), but is retained in case this migration runs against a snapshot taken before #1762.
nicoloboschi
added a commit
that referenced
this pull request
May 26, 2026
…izing them (#1766) * chore: regenerate docs skill (sync Tigris S3 config notes) Drift picked up by the generate-docs-skill pre-commit hook — keeps skills/hindsight-docs/ in sync with the upstream hindsight-docs/ sources. * perf(api): derive entity edges from unit_entities instead of materializing them Stop writing link_type='entity' rows to memory_links and derive entity edges on demand in the /graph endpoint (from the unit_entities self-join recall already uses) and in /stats (by replicating the historical writer cap). Why: on the recall-perf-medium bench bank (10k units), entity rows were 53% of all memory_links — 345k rows, ~190 MB of table+index — and recall never read them (entity expansion in link_expansion_retrieval.py uses unit_entities, not memory_links). Retain was running a synchronous pairwise loop per shared entity to write rows nothing read; per-unit entity degree was uncapped (max 326 outgoing on a single unit), and overall per-unit total degree averaged 130 with a p99 of 462. Changes: - Drop Phase 3 entity-link build/insert from retain orchestrator. Keep entity_resolver.flush_pending_stats() so entity_cooccurrences (which feeds /entities/graph) still updates. - Delete build_entity_links_from_resolved, insert_entity_links_batch, MAX_LINKS_PER_ENTITY, EntityLink, Phase3Context, and the now-dead fetch_entity_unit_fanout op (PG + Oracle). - /graph: filter memory_links query to link_type <> 'entity'; broaden the existing observation-inferred entity-pair loop to cover all visible units; cap at 10 units per entity to bound hot entities. - /stats: split link_breakdown into a memory_links query (non-entity) and a unit_entities-based derivation for entity, sized to the historical writer cap so link_counts.entity stays in the same magnitude. - Migration e9b2c7d1f3a4: drop idx_memory_links_entity_covering and chunk-delete existing entity rows (PG + Oracle paths). - Tests: rewrite test_entity_links_creation and test_all_link_types_together to assert via /graph + /stats; assert no entity rows in memory_links. API response shapes (graph edges, stats link_counts/links_breakdown) are unchanged at the boundary, so SDKs and the control plane do not need to be regenerated. * fix(graph): cap entity edges per unit, not per entity list The previous derivation kept only the first 10 units per entity before pairing, so any unit beyond #10 for a hot entity had zero entity edges in /graph — even though it shared the entity with many visible units. Switch to a sliding window: each unit links to its next N neighbors in the per-entity list. Every unit that shares an entity with another visible unit gets edges (its successors directly, predecessors via their pairs), and total edges stay bounded at ~N * cap per entity instead of N². Adds a regression test that retains 15 facts mentioning the same person and asserts every retained unit appears in at least one entity edge in /graph. * fix(migration): re-parent entity-link drop after e1b2c3d4f5a6 landed on main #1762 landed e1b2c3d4f5a6_drop_unused_indexes between this PR opening and CI run, which also drops idx_memory_links_entity_covering. Our migration's down_revision still pointed at the prior head, leaving Alembic with two heads and tripping test_alembic_dag.test_single_head. Re-parent to e1b2c3d4f5a6 to unify the head. The DROP INDEX IF EXISTS line becomes a defensive no-op (since #1762 already dropped it), but is retained in case this migration runs against a snapshot taken before #1762.
nicoloboschi
added a commit
that referenced
this pull request
May 27, 2026
- Merge entity-edge-derivation (#1766), unused-index drops (#1762), and async link recompute into a single "Graph Storage & Maintenance" section that leads with the ~50% storage reduction. - Merge "Targeted Consolidation by Scope" and "Consolidation Quality Rewrite" into one "Consolidation Improvements" section; drop prompt internals. - Rewrite the multilingual section at a higher level (concepts, not env vars) and link out to /developer/multilingual.
nicoloboschi
added a commit
that referenced
this pull request
May 27, 2026
* docs: add 0.7.0 changelog and release blog post Documents the 0.7.0 release: ParadeDB pg_search BM25 backend (Citus-compatible), PGroonga + configurable BM25 language for multilingual/CJK search, async link recompute that fixes outgoing-link staleness after deletes, Control Plane i18n in 8 locales, targeted consolidation by observation scope, an observation-consolidation prompt rewrite, a clear-mental-model endpoint, ZeroEntropy + Codex OAuth embeddings, and a long tail of bug fixes. Also fixes release.sh to refresh the root package-lock.json after workspace version bumps. Without this, npm ci in CI fails because the lock pins the previous workspace versions and the publish + docs-deploy jobs break (which is what happened to the initial v0.7.0 tag). * docs(blog): tighten 0.7.0 release post - Merge entity-edge-derivation (#1766), unused-index drops (#1762), and async link recompute into a single "Graph Storage & Maintenance" section that leads with the ~50% storage reduction. - Merge "Targeted Consolidation by Scope" and "Consolidation Quality Rewrite" into one "Consolidation Improvements" section; drop prompt internals. - Rewrite the multilingual section at a higher level (concepts, not env vars) and link out to /developer/multilingual. * docs(blog): rewrite 0.7.0 release post in announcement tone Rewrite each section in the same voice as prior major-release posts (0.5.0, 0.6.0): lead with what the user gets and why it matters, drop implementation internals (queue tables, FK cascades, JSON predicates, AST walkers), keep concrete config knobs and code examples where they help, and link out to docs for deep dives. * docs(blog): move ParadeDB section to last; reorder intro to match * docs(blog): demote Clear Mental Model from feature section to Other Notable Changes
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.