feat(memory): validity windows on memory_units (valid_to + /invalidate) — revives #1395

[slayoffer]

Summary

Revives the abandoned PR #1395 by @nikolay-bratanov, which implemented validity windows on memory_units (valid_to TIMESTAMPTZ NULL column + partial index + POST /memories/{memory_id}/invalidate endpoint + recall filter + tests). Resolves the two mechanical blockers diagnosed by @yonefive71 in the original PR review:

1. Alembic revision id collision — the original migration claimed a2b3c4d5e6f7, which was subsequently reused by add_text_signals_column on main. Renumbered to a fresh hex id, chained off the current upstream head.
2. Test import typo — from hindsight_api.engine.sql.postgresql import PostgresDialect → ... import PostgreSQLDialect as PostgresDialect. One-line aliasing, no test-body changes.

The substantive design question from issue #1391 ("who's responsible to invalidate facts?") is addressed in the design memo below — short answer: split by signal source (in-stream vs out-of-band), not by responsible actor. The consolidator + a better-shaped observations_mission handles in-stream contradictions; /invalidate is the missing primitive for out-of-band cases (user retracts, agent self-correction, post-hoc dedup of pre-existing duplicate clusters). They are complementary, not competing.

Marked as draft to give the maintainer space to react to the design framing before this goes to ready-for-review.

Original code is preserved verbatim across the 6 touched files; only the migration filename + revision frontmatter + one test import line differ from PR #1395.

Commits

1. feat(memory): validity windows on memory_units (cherry-pick of feat(memory): validity windows on memory_units (valid_to + /invalidate endpoint) #1395) — squashed apply of the original diff, Co-authored-by: Nikolay Bratanov.
2. alembic: renumber valid_to migration to resolve revision id collision — fresh hex id + chain to current upstream head.
3. tests: alias PostgreSQLDialect as PostgresDialect for test_invalidate_memory — credit @yonefive71's diff from the review.

---

Design context — fact obsolescence in Hindsight

TL;DR

1. The maintainer's question on PR feat(memory): validity windows on memory_units (valid_to + /invalidate endpoint) #1395 ("who's responsible to invalidate facts?") has a concrete answer: both, but for different signal classes. The consolidator handles implicit invalidation (new fact contradicts an old fact in the same retain stream); the agent/user needs an explicit /invalidate for everything else (user retracts, agent self-correction, post-hoc dedup, out-of-band staleness like a server being decommissioned). These are different problems with different signal sources — neither subsumes the other.
2. We tested the implicit half on a real production deployment (2026-05-31) and confirmed it can be improved dramatically with zero schema changes, just by rewriting the per-bank observations_mission. On a sandbox bank seeded with 12 raw facts (3 contradiction cases + 4 protection cases), the default mission produced 5 unmerged observations from just the BMW/Toyota contradiction pair; a mission rewrite produced 1 cleanly merged observation with both states + transition date, while preserving every legitimate historical fact and multi-facet split. Numbers and outputs below.
3. This does not eliminate the need for Feature: validity windows on memory_units (valid_to + /invalidate endpoint) #1391 / PR feat(memory): validity windows on memory_units (valid_to + /invalidate endpoint) #1395. The mission improves consolidation when the signal is in-stream; it cannot help when the signal is out-of-band (user clicks "this is wrong," agent detects a stale operational fact, dedup script wants to retire 1,280 byte-identical clusters as in @yonefive71's data). Both layers are needed.
4. Propose: land this PR for the schema-level out-of-band primitive, and separately document the mission-rewrite pattern as the first-line improvement in docs/concepts/. A short "in-stream vs out-of-band invalidation" docs section keeps the framing clear so future users reach for the right tool.

1. The framing question — who invalidates?

@nicoloboschi on #1391:

who's responsible to invalidate facts?

• if it's the agent, this will not work consistently.
• if it's Hindsight, this is already done via consolidation - could be improved but it has to be a different design

Both halves of the dichotomy are partially true but each one alone is insufficient. The proposed answer is to split the problem by signal source, not by responsible actor:

Signal source	Example	Best mechanism	Why
In-stream contradiction — new fact arrives that directly conflicts with a recent fact about the same entity/facet	"Alice likes BMW" → 3 months later "Alice likes Toyota"	Consolidator with stronger observations_mission	The signal is structurally present at consolidation time. The LLM already sees both facts in the same batch (or via recall budget). A better-shaped prompt is sufficient. Measured below in §3.
Out-of-band invalidation — fact is wrong, stale, or duplicate but no contradicting fact is being retained	User clicks "this memory is incorrect"; agent's tool inventory changes; dedup script finds 22 byte-identical copies of one canon sentence (@yonefive71's case)	Explicit POST /memories/{id}/invalidate endpoint (this PR)	No new fact triggers consolidation. No LLM has the signal. The decision lives outside Hindsight's pipeline; Hindsight only needs to honour it.
Agent self-correction during recall — agent reads conflicting memories at recall time and updates its own understanding	Agent gets BMW + Toyota both at recall, decides Toyota is current	Recall-time disambiguation (existing) + optional agent-driven /invalidate if it has high confidence	Same as out-of-band — the consolidator never sees the agent's reasoning.

The current implementation only addresses row 1, and it does so conservatively. Rows 2 and 3 are not addressed at all today. This PR is the missing primitive for rows 2 and 3. It is not in competition with the consolidator; it complements it.

The "if it's the agent, this will not work consistently" concern is real, but it is a policy concern (when should the agent be trusted to invalidate?), not a mechanism concern. Even an explicit endpoint can be gated — IAM-style, with admin-only by default and an opt-in allow_agent_invalidate per-bank flag. The mechanism needs to exist regardless of the policy.

2. State of the art — three patterns, one good fit

We surveyed how other LLM memory systems handle this:

System	Mechanism	History preserved?	Notes
Graphiti / Zep (Sarmah et al. 2025)	Bi-temporal edges (valid_at / invalid_at / created_at / expired_at)	Yes — explicit, queryable	The most rigorous design. New facts trigger LLM-based fact-resolution that can invalidate prior edges.
mem0 v2	ADD / UPDATE / DELETE classification per new fact	Yes — immutable ADD log	Abandoned in v3 (April 2026) — the classifier was 97.8% noise per the team's own audit. Now append-only.
mem0 v3 (current)	Append-only + retrieval ranking	Yes — append-only	Same architectural family as Hindsight today.
Letta / MemGPT	Agent rewrites its own core memory blocks	No — overwrite	History only preserved if the agent chooses to log it to archival.
Hindsight today	Append-only raw memory_units + LLM-synthesised observations + soft recency boost in reranking	Yes — append-only	Same family as mem0 v3.

The Graphiti bi-temporal model maps most naturally onto Hindsight's existing architecture: memory_units are already immutable append-only rows; observations are already synthesised summaries with a history JSONB audit column. Adding a valid_to TIMESTAMPTZ NULL column to memory_units (this PR) is the smallest possible step toward bi-temporal semantics — it adds one bit of state (active/invalidated) without committing to a full temporal-graph refactor.

mem0's v2→v3 retreat is instructive. The reason their explicit UPDATE/DELETE classifier failed was that they ran it as a separate per-fact LLM call without sufficient context, leading to high noise. Hindsight's consolidator avoids this trap because UPDATE/DELETE decisions already happen inside the existing consolidation LLM call, which already loads related observations via recall — the signal is there, only the prompt shape is weak. This is empirically validated below.

3. Empirical evidence — mission rewrite alone closes 80%+ of the in-stream gap

Tested on a fresh sandbox bank (obsolescence-test) on a production Hindsight deployment (API v0.6.2) on 2026-05-31. Same engine, same default LLM (gpt-5-mini), no code changes — only per-bank observations_mission differed.

Test setup

12 raw facts retained via POST /v1/default/banks/{bank_id}/memories, with backdated timestamps so recency reasoning had real time gaps to work with:

#	Fact	Backdated to	Category
1	Alice told me she really likes BMW cars... best brand for driving experience and engineering quality.	2026-02-15	contradiction (BMW→Toyota)
2	Alice told me she loves Toyota cars now. She prefers them over BMW because of reliability and lower maintenance costs.	2026-05-31	contradiction (BMW→Toyota)
3	Alice has a dog named Rex. Rex is a German Shepherd, 5 years old.	2026-01-10	dated event (pet died, history)
4	Rex passed away on March 15, 2026. Alice is heartbroken.	2026-03-16	dated event (pet died, history)
5	Alice adopted a new dog named Luna, a Golden Retriever, in April 2026.	2026-04-20	multi-facet (new pet, different entity)
6	Alice owns a Tesla Model 3 since 2024. She drives it daily.	2026-02-01	multi-facet (ownership)
7	Alice also owns a sailing boat called Wind Dancer, kept at the marina.	2026-02-05	multi-facet (ownership)
8	Bob works at Google as a software engineer.	2026-01-20	contradiction (employer change)
9	Bob left Google and joined Microsoft as a senior engineer in May 2026.	2026-05-15	contradiction (employer change)

Baseline — default observations_mission

The default mission is "Track every detail: names, numbers, dates, places, and relationships. Prefer specifics over abstractions, never generalise." (see hindsight-api-slim/hindsight_api/engine/consolidation/prompts.py:_DEFAULT_MISSION).

After retain + explicit consolidation, observations for the BMW/Toyota pair alone:

1. "Alice prefers Toyota cars over BMW cars because she considers Toyota cars to be more reliable and to have lower maintenance costs."
2. "Alice now loves Toyota cars."
3. "Alice believes BMW is the best brand for engineering quality."
4. "Alice believes BMW is the best brand for driving experience."
5. "Alice likes BMW cars."

5 unmerged observations from 2 raw facts. None marked superseded. Recall for "what cars does Alice like" returned the BMW raw fact at position 1 despite being 3.5 months older — the embedding-similarity advantage swamped the ~6.8% recency penalty (engine/search/reranking.py:101-129). This is the failure mode reported by @yonefive71 (1,280 duplicate cluster, 17,569 total) and @1ain0516 (multi-agent bank with English/Chinese duplicates).

After — rewritten observations_mission

Track every detail with these contradiction rules. When a new fact replaces a
prior fact about the same entity's preference, opinion, location, employment,
ownership, or state, UPDATE the matching observation to the new state and
include the date of the change in the observation text (e.g. "Alice preferred
BMW; switched to Toyota on 2026-05-31"). Recency wins for these mutable
attributes. PRESERVE HISTORY only when the prior fact records a discrete,
dated event (sold X, X died, moved to Y) — those are history, not
contradictions. Never create two observations for the same facet of the
same entity.

After bank clear-observations + bank consolidate with the new mission, the BMW/Toyota pair collapses to a single observation:

"Alice preferred BMW cars but switched to Toyota on 2026-05-31; she now loves Toyota and prefers it over BMW because she considers Toyota more reliable and lower maintenance."

Adding facts 3–9 and re-consolidating, the full bank produced 8 observations (vs the ~15–20 a baseline default would produce extrapolating from the 5-for-2 baseline ratio):

#	Observation text	Verdict
1	Alice preferred BMW cars but switched to Toyota on 2026-05-31…	✅ merged contradiction with date
2	Bob left Google and joined Microsoft as a senior engineer in May 2026.	✅ merged contradiction with date
3	Alice had a dog named Rex, a German Shepherd who was 5 years old; Rex died on 2026-03-15.	✅ preserved history (dated event)
4	Alice adopted a new dog named Luna, a Golden Retriever, on 2026-04-20.	✅ separate (different entity)
5	Alice owns a Tesla Model 3 and has owned it since 2024.	✅ preserved (ownership)
6	Alice owns a sailing boat called Wind Dancer that is kept at the marina.	✅ preserved (separate entity)
7	Alice drives her Tesla Model 3 daily.	⚠️ near-duplicate of #5 (could be merged)
8	Alice is heartbroken.	⚠️ orphan observation from Rex death context

Recall for "where does Bob work" returns row 2 only — single clean answer. Recall for "what cars does Alice prefer" returns row 1 first — the merged superseding observation. Recall for "what pets has Alice had" returns rows 4 then 3 — Luna first (recent), Rex with death info second (history). All three queries give the agent immediately usable, deduplicated, recency-correct answers.

What this empirically confirms

• The conservative DELETE rule in _PROCESSING_RULES (rule 7: "PRESERVE HISTORY: …. Be very conservative with deletes") is not the binding constraint. The mission, which is injected above the rules (build_batch_consolidation_prompt), takes priority when the LLM has to choose. A mission that explicitly says "UPDATE for preferences/state, PRESERVE for dated events" successfully overrides the default conservative bias.
• The protection cases (rules 1, 4 in _PROCESSING_RULES — facet matching, multi-entity protection) still work under the aggressive mission. We saw no regressions on Tesla+boat (multi-facet), Rex (dated event), or Luna (different-entity-same-facet).
• The 2 minor issues (Tesla driving vs ownership split; orphan "Alice is heartbroken") are mission-tunable, not architectural. They show the upper bound of mission-rewrite is not 100% — more refinement helps marginally.

What this does NOT solve

The raw memory_units rows still contain the BMW fact verbatim. If a downstream consumer queries by fact_type=world or fact_type=experience, they still get the conflict. Only observation-layer queries are clean. This is exactly the gap this PR is designed to address — when you want raw recall to skip a fact, the only available mechanism today is DELETE (audit gone) or "wait and hope consolidation handles it." Mission rewrite does not give you valid_to.

It also does nothing for the @yonefive71 case (1,280 exact-duplicate memory_units already in the live bank from past sessions). Mission only affects future consolidation; it cannot retire pre-existing duplicates. A /invalidate endpoint can — and per @yonefive71's analysis, the trivial GROUP BY text HAVING count(*) > 1 sweep would reclaim ~825 rows on day one.

4. Proposed path forward

1. Land this PR under the framing in §1: this is the out-of-band invalidation primitive, complementary to (not in competition with) the consolidator.
2. Add docs at docs/concepts/ distinguishing in-stream vs out-of-band invalidation. Reference the mission-rewrite pattern as the first-line in-stream improvement. Reference /invalidate as the explicit primitive for everything else. Happy to do this in a follow-up PR.
3. Optional policy gate: add allow_agent_invalidate bool to bank config, defaulting false (only API-key holders can invalidate). This addresses the maintainer's "if it's the agent, this will not work consistently" concern by making agent-driven invalidation an explicit opt-in. The mechanism still exists for the cases where it's needed (admin scripts, user-explicit retracts). Can be a follow-up PR or included here if you prefer.
4. RFC: subject-predicate-object triples layer with temporal validity #1392 (SPO triples): deferred. The mission improvements + valid_to together cover most of the pain. Triples are a larger architectural conversation that should not block this work.

5. Acknowledgements

• @nikolay-bratanov for the original RFC + PR work.
• @yonefive71 for the production-bank data, the migration diagnosis, and the import fix.
• @ai-ag2026 for the curation-API comment on Edit or delete incorrect memories? #1094 that helped frame the soft-vs-hard delete distinction.
• @noctuid, @zviratko, @1ain0516, @suchanek for production datapoints across Edit or delete incorrect memories? #1094.
• @nicoloboschi for the framing question that forced the design clarification.

---

Test plan

• pytest hindsight-api-slim/tests/test_invalidate_memory.py passes after the PostgreSQLDialect alias fix (5 tests).
• Migration applies cleanly against the current upstream head (no multiple heads after revision-id renumber).
• CI on this PR passes (will track separately).
• Integration test on a real Postgres asserting an invalidated row drops out of recall — happy to add in a follow-up; original PR author offered the same.
• Doc page added under docs/concepts/ distinguishing in-stream vs out-of-band invalidation — follow-up PR after design is approved.

🤖 Authored by Claude (Anthropic) in collaboration with @nikolay-bratanov, @yonefive71, and the other thread contributors named above.

[UnderLotus]

Really appreciate this PR — both the direction and the thorough write-up. I'm a production Hermes Agent user running Hindsight as the sole memory backend, and we've been actively trying to figure out memory pruning over the past few days. This is exactly the missing piece.

Our bank has ~54K nodes / ~1K documents after a few weeks of use. We went through the same discovery process you described:

• Tried DELETE /memories/{id}/observations first — quickly realized it only clears observations and triggers re-consolidation, doesn't actually remove the memory.
• Looked into DELETE /documents/{id}, but in practice almost none of our documents are entirely noise — most contain a mix of valuable facts and transient Kanban snapshots.
• Found that the official stance is to delete source documents rather than individual memories, but that doesn't fit our pruning use case at all.

The valid_to / invalidate approach is the right primitive for row 2 in your taxonomy. We specifically have batches of memories that we can identify with high confidence as superseded or duplicate (e.g. via text similarity clustering on /memories/list output), but have no safe way to mark them as such today. The append-only + temporal validity model preserves history while letting recall cleanly exclude stale facts — this aligns perfectly with Hindsight's architectural philosophy.

One practical question: for batch dedup scenarios (say a script identifies 200 near-duplicate memories), would the intended workflow be to call /invalidate for each one individually, or is there room for a batch endpoint down the line? Either way the primitive itself is what matters first.

Hope the maintainers can take a look at this — the in-stream consolidation improvements alone are valuable, but the invalidation endpoint is what makes pruning actually viable for production users.

[zviratko]

I had my agent delete memories mentioning the stuff that got injected constantly, my laptop then spent 1 day recomputing the database (LLM).

While ago I reinstalled hermes and hindsight (as self hosted external), webui is useful, but I just hit the same problem as before - the agent refuses to use some tools, because it know from memory those don't work, even though I've fixed them, or tries to inject non-relevant memory facts everywhere, though this might be a hermes bug (or maybe my LLMs are too stupid).
In the end, it could be as simple as to add a simple blacklist filter of facts or combinations we could temporarily add to filter what is returned to the agent.

[nicoloboschi]

superseded by #1976