Fix JSON content, connection resilience, RAG Phase 2

[cmeans]

Summary

• JSON content field fix: remember/update_entry accept JSON objects/arrays — re-serialized to string when Pydantic deserializes before str validator
• Connection resilience: Auto-healing _conn property with 30s health check debounce. Dead connections reconnect transparently after Postgres restarts.
• Background embedding: Thread pool (2 workers) replaces synchronous embedding — writes no longer block on Ollama HTTP calls
• backfill_embeddings tool (28 total): Embeds pre-existing entries + re-embeds stale ones (text changed since embedding)
• hint param on get_knowledge: Semantic re-ranking of tag-filtered results by similarity to a natural language phrase
• Stale embedding detection: get_stale_embeddings finds entries whose content changed after their embedding was generated
• 304 tests (+10 new)

QA

Prerequisites

• pip install -e ".[dev]"
• Deploy to test instance on alternate port (AWARENESS_PORT=8421)
• Ollama running with nomic-embed-text model

Manual tests (via MCP tools)

1. • JSON dict content accepted

   remember(source="test", tags=["test"], description="Config", content={"key": "value"}, content_type="application/json")
   

   Expected: status: "ok" — content stored as JSON string

2. • Connection survives Postgres restart

   docker compose restart postgres
   # Wait 30+ seconds
   get_knowledge(tags=["test"])
   

   Expected: returns results after transparent reconnect

3. • backfill_embeddings creates embeddings for existing entries

   backfill_embeddings(limit=50)
   

   Expected: new_embeddings > 0, remaining = 0

4. • semantic_search finds backfilled entries

   semantic_search(query="retirement planning")
   

   Expected: returns entries that existed before Ollama was configured

5. • hint re-ranks get_knowledge results

   get_knowledge(tags=["finance"], hint="retirement savings", mode="list")
   

   Expected: results include similarity scores, retirement-related entries ranked first

6. • backfill detects stale embeddings

   update_entry(entry_id="...", description="completely new description")
   backfill_embeddings(limit=50)
   

   Expected: refreshed_embeddings >= 1

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[cmeans]

QA Review — PR #43: Fix JSON content, connection resilience, RAG Phase 2

Automated tests

• 315/315 pass (PR says 304 — subsequent commits added more)
• ruff: clean
• mypy: clean

Manual tests (6/6 pass)

All executed via isolated MCP instance with Ollama enabled.

#	Test	Result
1	JSON dict content accepted	✅ remember with content={"key":"value"} — stored as JSON string. Verified in DB.
2	Connection resilience	⚠️ Code-reviewed only (Postgres restart would affect other services). Logic is sound — health check property with 30s debounce, rollback-before-ping pattern, reconnect on OperationalError.
3	backfill_embeddings	✅ Returns new: 0, remaining: 0 — background thread pool already embedded entries. Verified in DB: all 3 entries have embeddings.
4	semantic_search finds entries	✅ "retirement planning" → 401k (0.56) > Roth IRA (0.37) > Config (0.24). Correct semantic ordering.
5	hint re-ranks get_knowledge	✅ tags=["finance"], hint="retirement savings" — results include similarity scores, 401k ranked above Roth IRA.
6	Stale embedding detection	✅ After update_entry with new description, background thread re-embedded within milliseconds. backfill_embeddings shows 0 stale (already refreshed). Verified in DB: text_hash and created updated.

Code review

Background embedding (key improvement): ThreadPoolExecutor(max_workers=2) replaces synchronous _generate_embedding. _do_embed runs in the pool with its own DB connection (via psycopg.connect(store.dsn)) to avoid racing with the main thread's shared connection. Entry data is serialized to primitives before submission (no mutable sharing). Good.

Connection resilience: _conn is now a @property that health-checks every 30s. Pattern: check closed → rollback (clear INERROR state) → SELECT 1 → rollback. On failure: close old, create new. The rollback-before-ping is important — without it, a connection in INERROR state would fail the health check every time. Good.

JSON content fix: if not isinstance(content, str): content = json.dumps(content) in both remember and update_entry. Handles the Pydantic deserialization edge case where JSON strings arrive as dicts. Simple, correct.

backfill_embeddings tool: Two phases — missing entries via get_entries_without_embeddings, then stale entries via get_stale_embeddings. Stale detection compares text_hash(compose_embedding_text(entry)) against stored emb_text_hash. Clean.

hint parameter: Re-ranks by doing a semantic_search with the hint text, then sorting the original entries by their similarity scores. Entries without embeddings sort to the end (-1.0). similarity field added to both list and full output when hint is active. Good.

get_related tool: Bidirectional traversal — forward via data.related_ids list, reverse via JSONB containment data->'related_ids' @> %s::jsonb. Deduplicates. Not in the QA manual tests but code-reviewed; the convention relies on callers storing related_ids in data.

Docs: Comprehensive README restructure — new config tables (Server, Embedding, Docker Compose), new sections (Semantic search, 29 tools), updated Current status. CLAUDE.md updated with embedding pipeline and connection resilience docs. CHANGELOG entries. Good.

Findings

1. Test count discrepancy (nit)

PR says 304, CHANGELOG says 310, actual is 315. Update to match.

2. get_related not in QA manual tests (observation)

The PR adds get_related tool (28 → 29 tools) but it's not listed in the QA section. PR summary also doesn't mention it. It was code-reviewed and the logic is straightforward (forward + reverse JSONB lookup, dedup), but it wasn't manually tested through MCP.

3. hint over-fetches from semantic_search (minor)

The hint implementation calls semantic_search with limit=len(entries) + 10 but no filters — it searches ALL entries, not just the ones that matched the tag filter. For large stores, this could return similarity scores for unrelated entries. Currently harmless (the re-sort only applies scores to entries already in the result set), but the over-fetch is unnecessary work. Could pass the same source/tags filters through.

Verdict

No substantive issues. Finding #1 is a nit, #2 is an observation (get_related is simple enough for code review), #3 is a minor optimization opportunity. The three headline features (JSON fix, connection resilience, background embedding) are all well-implemented.

Ready for QA Approved label.

[cmeans]

Review response

Finding #1: Test count ✅ Fixed

Updated CHANGELOG and README to 315.

Finding #2: get_related not in QA ✅ Acknowledged

Simple bidirectional JSONB lookup — code review sufficient. Will include in QA for future PRs that touch it.

Finding #3: hint over-fetches ✅ Fixed

semantic_search call in hint re-ranking now passes through all active filters (source, tags, entry_type, since, until) instead of searching all entries. Reduces unnecessary work on large stores.