feat(claims): Provable Memory layer, default-off (v2 spine) - #163
Conversation
Qodo reviews are paused for this user.Troubleshooting steps vary by plan Learn more → On a Teams plan? Using GitHub Enterprise Server, GitLab Self-Managed, or Bitbucket Data Center? |
📝 WalkthroughWalkthroughThis PR introduces a complete claims layer (Provable Memory) feature for extracting, storing, verifying, and gating search results based on claim verification status. The implementation spans extraction, storage, verification, search integration, and CLI tooling, with a pre-registered E-009 experiment design. ChangesProvable Memory Claims Layer
Sequence Diagram(s)sequenceDiagram
participant Ingest
participant Archive
participant ClaimStore
participant MemoryIndex
Ingest->>Archive: record(text)
Archive-->>Ingest: span_id
Ingest->>Ingest: claims_from_text(text, span_id)
Ingest->>MemoryIndex: add vector record<br/>(archive_span_id=span_id)
Ingest->>ClaimStore: add_claim(text, spans)<br/>(status=unverified)
sequenceDiagram
participant Search
participant MemoryIndex
participant ClaimStore
participant Gate
Search->>MemoryIndex: retrieve(query)
MemoryIndex-->>Search: hits with confidence
loop For each hit
Search->>ClaimStore: status_for_spans(archive_span_id)
ClaimStore-->>Search: claim_status
end
Search->>Gate: apply_claims_gate(hits, prefer_verified)
Gate->>Gate: filter: drop unsupported
Gate->>Gate: boost: +score for supported
Gate->>Gate: sort by score DESC
Gate-->>Search: reordered hits
Search-->>Search: strip transient fields
Estimated code review effort🎯 4 (Complex) | ⏱️ ~60 minutes Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| while True: | ||
| pending = await store.pull_unverified(limit=batch) | ||
| fresh = [c for c in pending if c["id"] not in seen] | ||
| if not fresh: | ||
| break | ||
| for claim in fresh: | ||
| seen.add(claim["id"]) | ||
| spans = await fetch_spans(claim["archive_span_ids"]) | ||
| status, model = verifier.verify(claim["text"], spans) | ||
| if status == "unverified": | ||
| # fail-closed: stays unverified, retried in a future pass. | ||
| continue | ||
| await store.set_status(claim["id"], status, verifier_model=model, now=now) | ||
| done += 1 |
There was a problem hiding this comment.
⚠️ Bug: verify_pass skips claims beyond a fully-failing first batch
verify_pass (taosmd/claims/verify_pass.py:28-41) paginates via store.pull_unverified(limit=batch), which always returns the first batch claims still in status 'unverified' ordered by id (store.py:77-81). When a claim's verifier returns 'unverified' it is left unverified (fail-closed) but added to the seen set. On the next loop iteration pull_unverified returns the SAME first batch rows (they are still 'unverified' and lowest id), fresh becomes empty after filtering by seen, and the loop breaks.
Consequence: if a whole batch (e.g. 100) of claims fails or is genuinely unsupported-but-returned-unverified, every unverified claim with a higher id is never pulled in that pass — and since seen is reset on each new verify invocation, those claims remain permanently unreachable as long as the leading batch keeps returning 'unverified'. The store can quietly accumulate unverifiable claims that block all claims behind them.
Fix: advance pagination by id cursor instead of relying on status changing to move the window.
Page by an id cursor so failed/unverified claims no longer block the rest of the pass.:
# store.py
async def pull_unverified(self, limit: int = 100, after: int = 0) -> list[dict]:
rows = self._conn.execute(
"SELECT * FROM claims WHERE status = 'unverified' AND id > ?"
" ORDER BY id LIMIT ?", (after, limit)
).fetchall()
return [self._row(r) for r in rows]
# verify_pass.py
async def verify_pass(store, verifier, fetch_spans, batch=100, now=None) -> int:
done = 0
after = 0
while True:
pending = await store.pull_unverified(limit=batch, after=after)
if not pending:
break
for claim in pending:
after = max(after, claim["id"])
spans = await fetch_spans(claim["archive_span_ids"])
status, model = verifier.verify(claim["text"], spans)
if status == "unverified":
continue
await store.set_status(claim["id"], status, verifier_model=model, now=now)
done += 1
return done
Was this helpful? React with 👍 / 👎
| async def status_for_spans(self, span_ids: list[int]) -> str | None: | ||
| """Worst status among claims backed by any of these archive spans. | ||
|
|
||
| Used by the recall gate to judge a hit by the claims its source spans | ||
| back. Worst-wins so one unsupported claim demotes the hit. None when no | ||
| claim references these spans (a raw, non-claim memory).""" | ||
| if not span_ids: | ||
| return None | ||
| rows = self._conn.execute("SELECT archive_span_ids, status FROM claims").fetchall() | ||
| want = set(span_ids) | ||
| order = {s: i for i, s in enumerate( | ||
| ("supported", "unverified", "partial", "contradicted", "unsupported"))} | ||
| worst = None | ||
| for r in rows: | ||
| if want & set(json.loads(r["archive_span_ids"])): |
There was a problem hiding this comment.
⚠️ Performance: status_for_spans does a full claims table scan per hit
status_for_spans (taosmd/claims/store.py:83-100) runs SELECT archive_span_ids, status FROM claims — a full table scan — and JSON-decodes every row on every call. _attach_and_gate_claims (taosmd/api.py:336-357) invokes it once per hit when the gate is on, so a single search() with the gate enabled costs O(hits × total_claims) row reads plus JSON parsing. As the claim store grows this becomes a hot-path cost on every gated recall (exactly the path E-009 exercises before the default is flipped).
Suggested fix: scan the claims table once per search and build a span→worst-status map (or filter rows in SQL by the union of needed span ids), rather than re-scanning per hit. The archive_span_ids JSON column also prevents index use; consider a normalized claim_spans(span_id, claim_id) table for indexed lookups if the store is expected to be large.
Build the span→status map with a single table scan instead of one scan per hit.:
# api.py _attach_and_gate_claims: scan once, resolve all hits against one map
from taosmd.claims.gate import apply_claims_gate # noqa: PLC0415
span_map = await claim_store.worst_status_map() # {span_id: worst_status}
for h in hits:
span = (h.get("metadata") or {}).get("archive_span_id")
h["claim_status"] = span_map.get(span) if isinstance(span, int) else None
h["score"] = h.get("confidence", 0.0)
# (add ClaimStore.worst_status_map() that does the single scan + worst-wins fold)
Was this helpful? React with 👍 / 👎
Code Review
|
| Auto-apply | Compact |
|
|
Was this helpful? React with 👍 / 👎 | Gitar
Code Review SummaryStatus: 9 Issues Found | Recommendation: Address before merge Overview
Issue Details (click to expand)CRITICAL
WARNING
SUGGESTION
Other Observations (not in diff)No additional issues found outside diff lines. Inline comment posting was attempted but rejected by the non-interactive environment permission layer. Files Reviewed (18 files)
Fix these issues in Kilo Cloud Reviewed by nex-n2-pro:free · 955,110 tokens |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 10
🧹 Nitpick comments (2)
tests/claims/test_verify_pass.py (1)
17-45: ⚡ Quick winAdd a regression case where pending unverified claims exceed one batch.
Current tests use datasets smaller than the batch size (or single-claim fail-closed). Add a case with
N > batchwhere verifier always returns"unverified"and assert all claims are attempted once. This will lock in the intended pass-progress semantics.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/claims/test_verify_pass.py` around lines 17 - 45, Add a new test in tests/claims/test_verify_pass.py that creates N claims where N > batch (use _store and asyncio.run with s.add_claim in a loop), use a verifier class whose verify method always returns ("unverified", "msg"), call verify_pass(s, verifier, _spans_text, batch=batch_size, now=...) and assert the returned n equals N (all claims attempted once) and each claim's status (via asyncio.run(s.get(cid))) remains "unverified"; reference the existing verify_pass function and _store helper and pick a concrete batch_size (e.g., 5) with N > batch to reproduce the regression.tests/claims/test_gate.py (1)
8-35: ⚡ Quick winAdd an invalid-mode regression test.
Once mode validation is enforced, add a test asserting invalid modes raise
ValueErrorso silent fallback cannot regress.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/claims/test_gate.py` around lines 8 - 35, Add a regression test that ensures invalid modes raise ValueError: create a new test function (e.g. test_invalid_mode_raises_value_error) that calls apply_claims_gate(hits, mode="invalid_mode") and asserts a ValueError is raised (use pytest.raises) to prevent silent fallback; place it alongside the existing tests for apply_claims_gate.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@taosmd/api.py`:
- Around line 134-136: In _ensure_stores, make ClaimStore initialization
fail-open: wrap the creation and await of ClaimStore (the ClaimStore(...) call
and await claims.init()) in a try/except that catches Exception, logs a
warning/error with the exception, and continues without re-raising (optionally
setting claims = None or marking the store as unavailable) so core ingest/search
won't fail when the gating is off; ensure callers handle a missing/None claims
store gracefully.
- Around line 231-238: The claims extraction/writes inside the ingest path (the
block using claims_from_text, stores["claims"].add_claim, span_id checks and the
archived variable) must be isolated so failures there do not propagate and
prevent incrementing archived or surface as a partial-success error; wrap the
entire claims extraction + per-claim add loop in a try/except that catches
Exception, logs the failure (including the exception) and continues without
re-raising, leaving archived += 1 executed as before; keep the existing guard
that the block only runs when "claims" in stores and span_id is a non-negative
int and do not change the surrounding control flow.
In `@taosmd/claims/gate.py`:
- Around line 21-25: The function apply_claims_gate silently treats any
non-"strict" mode as lenient, so validate the mode explicitly: accept only
"off", "strict", or "lenient" (or whatever allowed set you decide), raise a
ValueError for unknown modes, and only compute drop when mode is "strict" or
"lenient"; keep the existing logic that returns hits unchanged for "off" and
that sets drop = _DROP_STRICT for "strict" and drop = _DROP_LENIENT for
"lenient", then build kept = [h for h in hits if (h.get("claim_status") not in
drop)] and return kept.
In `@taosmd/claims/store.py`:
- Around line 91-99: status_for_spans() currently does a full-table scan and
JSON-decodes every row; refactor it to query an indexed join table instead:
add/use a normalized table (e.g., claim_spans with columns claim_id, span_id and
indexed span_id) and change the logic in status_for_spans to SELECT status FROM
claims JOIN claim_spans ON claims.id=claim_spans.claim_id WHERE
claim_spans.span_id IN (...) (or batch the IN with params), then compute the
worst status using the existing order mapping without per-row json.loads; this
removes the full-table scan and per-call JSON parsing and limits work to
matching spans only.
In `@taosmd/claims/verifier.py`:
- Around line 22-24: parse_verdict currently returns 'supported' because the
loop (v in ("CONTRADICTED","UNSUPPORTED","PARTIAL","SUPPORTED")) matches the
bare word SUPPORTED even when preceded by negation (e.g., "not supported");
update the regex used when v == "SUPPORTED" to exclude common negation patterns
before the word. Concretely, in parse_verdict where you iterate over v and use
re.search(rf"\b{v}\b", t), special-case SUPPORTED to use a negative-lookbehind
or explicit pattern like
r"(?<!\b(?:not|no|n't|never|cannot|can't)\b\s*)\bSUPPORTED\b" (with
re.IGNORECASE) so phrases containing negation do not return 'supported'; keep
the existing checks for "CONTRADICTED", "UNSUPPORTED", and "PARTIAL" unchanged.
In `@taosmd/claims/verify_pass.py`:
- Around line 29-32: The current fixed-window repull using
store.pull_unverified(limit=batch) plus filtering by seen causes starvation;
change the loop to use cursor-based pagination by tracking last_seen_id and
calling a paginated pull (e.g., store.pull_unverified_after(last_seen_id,
limit=batch) or add a parameter to store.pull_unverified to do WHERE id >
last_seen_id ORDER BY id LIMIT ?), update last_seen_id to the max id returned
each fetch, and remove the fresh filtering that stops the loop early (or only
skip already-seen ids while continuing the cursor). Ensure the new query returns
rows ordered by id so every unverified claim is attempted once per pass even if
earlier rows remain unverified.
- Around line 35-40: Per-claim exceptions in the verification loop currently
abort the whole pass; wrap the per-claim work (the calls to fetch_spans,
verifier.verify, and store.set_status for each claim) in a try/except so any
exception is caught, logged, and the loop continues (leaving that claim
unverified) rather than re-raising; specifically, enclose the sequence spanning
fetch_spans(claim["archive_span_ids"]), verifier.verify(claim["text"], spans),
and the subsequent await store.set_status(...) in a try block and in except log
the exception and continue to the next claim to preserve fail-closed behavior.
In `@taosmd/cli.py`:
- Around line 1488-1491: The --batch argument currently uses
verify_p.add_argument("--batch", type=int, ...) and accepts zero or negative
values; add a validator function (e.g., def positive_int(value): ...) that
converts to int, checks value > 0, and raises argparse.ArgumentTypeError on
failure, then change verify_p.add_argument to use type=positive_int and keep the
same default/help; this ensures --batch only accepts positive integers and
produces a clear argparse error message when invalid.
- Around line 998-999: The verify command in taosmd/cli.py imports httpx at
runtime (see the import added for the verify handler) but httpx is not declared
or installed; either add "httpx" to project.dependencies in pyproject.toml (or
create a dedicated extra) so it’s installed, or modify the verify handler to
catch ModuleNotFoundError around the httpx import and raise a clear error
message instructing the user to pip install httpx (and update
taosmd/auto_setup.py if you choose to auto-install it so the setup logic
installs the new dependency).
In `@tests/claims/test_store.py`:
- Around line 35-39: Replace the fragile "assert False, 'expected ValueError'"
in the test for s.set_status with an explicit test failure call (e.g.,
pytest.fail("expected ValueError")) so the test triggers a clear failure instead
of using assert False; update the test in tests/claims/test_store.py around the
s.set_status(cid, "bogus", verifier_model="m", now=1.0) try/except block to call
pytest.fail in the try branch and keep the except ValueError: pass as the
expected path.
---
Nitpick comments:
In `@tests/claims/test_gate.py`:
- Around line 8-35: Add a regression test that ensures invalid modes raise
ValueError: create a new test function (e.g.
test_invalid_mode_raises_value_error) that calls apply_claims_gate(hits,
mode="invalid_mode") and asserts a ValueError is raised (use pytest.raises) to
prevent silent fallback; place it alongside the existing tests for
apply_claims_gate.
In `@tests/claims/test_verify_pass.py`:
- Around line 17-45: Add a new test in tests/claims/test_verify_pass.py that
creates N claims where N > batch (use _store and asyncio.run with s.add_claim in
a loop), use a verifier class whose verify method always returns ("unverified",
"msg"), call verify_pass(s, verifier, _spans_text, batch=batch_size, now=...)
and assert the returned n equals N (all claims attempted once) and each claim's
status (via asyncio.run(s.get(cid))) remains "unverified"; reference the
existing verify_pass function and _store helper and pick a concrete batch_size
(e.g., 5) with N > batch to reproduce the regression.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: f3717adb-e53f-44ca-b8f2-9090a0ab5b13
📒 Files selected for processing (18)
CHANGELOG.mddocs/research-report.mdtaosmd/api.pytaosmd/claims/__init__.pytaosmd/claims/extract.pytaosmd/claims/gate.pytaosmd/claims/store.pytaosmd/claims/verifier.pytaosmd/claims/verify_pass.pytaosmd/cli.pytests/claims/__init__.pytests/claims/test_cli_claims.pytests/claims/test_extract.pytests/claims/test_gate.pytests/claims/test_retrieve_gate.pytests/claims/test_store.pytests/claims/test_verifier.pytests/claims/test_verify_pass.py
| from taosmd.claims.store import ClaimStore # noqa: PLC0415 | ||
| claims = ClaimStore(db_path=str(path / "claims.db")) | ||
| await claims.init() |
There was a problem hiding this comment.
Keep claims store initialization fail-open to preserve default-off behavior.
ClaimStore init is now a hard dependency of _ensure_stores; if it fails, core ingest/search fail even when gating is off.
Proposed fix
- from taosmd.claims.store import ClaimStore # noqa: PLC0415
- claims = ClaimStore(db_path=str(path / "claims.db"))
- await claims.init()
+ claims = None
+ try:
+ from taosmd.claims.store import ClaimStore # noqa: PLC0415
+ claims = ClaimStore(db_path=str(path / "claims.db"))
+ await claims.init()
+ except Exception as exc: # pragma: no cover - defensive
+ logger.warning("taosmd: claims store disabled (init failed): %s", exc)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/api.py` around lines 134 - 136, In _ensure_stores, make ClaimStore
initialization fail-open: wrap the creation and await of ClaimStore (the
ClaimStore(...) call and await claims.init()) in a try/except that catches
Exception, logs a warning/error with the exception, and continues without
re-raising (optionally setting claims = None or marking the store as
unavailable) so core ingest/search won't fail when the gating is off; ensure
callers handle a missing/None claims store gracefully.
| # Claims layer (additive): extract facts as claims tagged with the same | ||
| # archive span. Stored unverified; the verify-pass checks them async. | ||
| if "claims" in stores and isinstance(span_id, int) and span_id >= 0: | ||
| from taosmd.claims.extract import claims_from_text # noqa: PLC0415 | ||
| for c in claims_from_text(text, span_id): | ||
| await stores["claims"].add_claim( | ||
| c["text"], c["archive_span_ids"], c["source_extractor"]) | ||
| archived += 1 |
There was a problem hiding this comment.
Isolate claims extraction/write failures from core ingest writes.
If claims extraction/add fails here, ingest raises after archive/vector were already written; that creates a partial-success write path and under-reports archived.
Proposed fix
- if "claims" in stores and isinstance(span_id, int) and span_id >= 0:
- from taosmd.claims.extract import claims_from_text # noqa: PLC0415
- for c in claims_from_text(text, span_id):
- await stores["claims"].add_claim(
- c["text"], c["archive_span_ids"], c["source_extractor"])
+ if stores.get("claims") is not None and isinstance(span_id, int) and span_id >= 0:
+ try:
+ from taosmd.claims.extract import claims_from_text # noqa: PLC0415
+ for c in claims_from_text(text, span_id):
+ await stores["claims"].add_claim(
+ c["text"], c["archive_span_ids"], c["source_extractor"]
+ )
+ except Exception as exc: # pragma: no cover - defensive
+ logger.warning("taosmd: claims extraction skipped for span %s: %s", span_id, exc)
archived += 1📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # Claims layer (additive): extract facts as claims tagged with the same | |
| # archive span. Stored unverified; the verify-pass checks them async. | |
| if "claims" in stores and isinstance(span_id, int) and span_id >= 0: | |
| from taosmd.claims.extract import claims_from_text # noqa: PLC0415 | |
| for c in claims_from_text(text, span_id): | |
| await stores["claims"].add_claim( | |
| c["text"], c["archive_span_ids"], c["source_extractor"]) | |
| archived += 1 | |
| # Claims layer (additive): extract facts as claims tagged with the same | |
| # archive span. Stored unverified; the verify-pass checks them async. | |
| if stores.get("claims") is not None and isinstance(span_id, int) and span_id >= 0: | |
| try: | |
| from taosmd.claims.extract import claims_from_text # noqa: PLC0415 | |
| for c in claims_from_text(text, span_id): | |
| await stores["claims"].add_claim( | |
| c["text"], c["archive_span_ids"], c["source_extractor"] | |
| ) | |
| except Exception as exc: # pragma: no cover - defensive | |
| logger.warning("taosmd: claims extraction skipped for span %s: %s", span_id, exc) | |
| archived += 1 |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/api.py` around lines 231 - 238, The claims extraction/writes inside
the ingest path (the block using claims_from_text, stores["claims"].add_claim,
span_id checks and the archived variable) must be isolated so failures there do
not propagate and prevent incrementing archived or surface as a partial-success
error; wrap the entire claims extraction + per-claim add loop in a try/except
that catches Exception, logs the failure (including the exception) and continues
without re-raising, leaving archived += 1 executed as before; keep the existing
guard that the block only runs when "claims" in stores and span_id is a
non-negative int and do not change the surrounding control flow.
| def apply_claims_gate(hits: list[dict], mode: str = "off") -> list[dict]: | ||
| if mode == "off" or not hits: | ||
| return hits | ||
| drop = _DROP_STRICT if mode == "strict" else _DROP_LENIENT | ||
| kept = [h for h in hits if (h.get("claim_status") not in drop)] |
There was a problem hiding this comment.
Validate gate mode explicitly instead of silently falling back.
Unknown mode values currently route to lenient behavior, which can silently change recall semantics on typos or bad caller input.
Proposed fix
def apply_claims_gate(hits: list[dict], mode: str = "off") -> list[dict]:
+ if mode not in {"off", "prefer_verified", "strict"}:
+ raise ValueError(f"unsupported claims gate mode: {mode!r}")
if mode == "off" or not hits:
return hits
drop = _DROP_STRICT if mode == "strict" else _DROP_LENIENT📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def apply_claims_gate(hits: list[dict], mode: str = "off") -> list[dict]: | |
| if mode == "off" or not hits: | |
| return hits | |
| drop = _DROP_STRICT if mode == "strict" else _DROP_LENIENT | |
| kept = [h for h in hits if (h.get("claim_status") not in drop)] | |
| def apply_claims_gate(hits: list[dict], mode: str = "off") -> list[dict]: | |
| if mode not in {"off", "prefer_verified", "strict"}: | |
| raise ValueError(f"unsupported claims gate mode: {mode!r}") | |
| if mode == "off" or not hits: | |
| return hits | |
| drop = _DROP_STRICT if mode == "strict" else _DROP_LENIENT | |
| kept = [h for h in hits if (h.get("claim_status") not in drop)] |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/claims/gate.py` around lines 21 - 25, The function apply_claims_gate
silently treats any non-"strict" mode as lenient, so validate the mode
explicitly: accept only "off", "strict", or "lenient" (or whatever allowed set
you decide), raise a ValueError for unknown modes, and only compute drop when
mode is "strict" or "lenient"; keep the existing logic that returns hits
unchanged for "off" and that sets drop = _DROP_STRICT for "strict" and drop =
_DROP_LENIENT for "lenient", then build kept = [h for h in hits if
(h.get("claim_status") not in drop)] and return kept.
| rows = self._conn.execute("SELECT archive_span_ids, status FROM claims").fetchall() | ||
| want = set(span_ids) | ||
| order = {s: i for i, s in enumerate( | ||
| ("supported", "unverified", "partial", "contradicted", "unsupported"))} | ||
| worst = None | ||
| for r in rows: | ||
| if want & set(json.loads(r["archive_span_ids"])): | ||
| if worst is None or order[r["status"]] > order[worst]: | ||
| worst = r["status"] |
There was a problem hiding this comment.
status_for_spans() does a full-table scan per retrieval hit.
Line 91 loads every claim row, and Line 97 JSON-decodes span arrays row-by-row on each call. Since retrieval attaches claim status per hit (taosmd/api.py Line 351), this grows toward O(hits × claims) and will become a latency bottleneck as claims grow. Consider normalizing span links into an indexed join table (e.g., claim_spans(claim_id, span_id)) so status lookup only touches matching spans.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/claims/store.py` around lines 91 - 99, status_for_spans() currently
does a full-table scan and JSON-decodes every row; refactor it to query an
indexed join table instead: add/use a normalized table (e.g., claim_spans with
columns claim_id, span_id and indexed span_id) and change the logic in
status_for_spans to SELECT status FROM claims JOIN claim_spans ON
claims.id=claim_spans.claim_id WHERE claim_spans.span_id IN (...) (or batch the
IN with params), then compute the worst status using the existing order mapping
without per-row json.loads; this removes the full-table scan and per-call JSON
parsing and limits work to matching spans only.
| for v in ("CONTRADICTED", "UNSUPPORTED", "PARTIAL", "SUPPORTED"): | ||
| if re.search(rf"\b{v}\b", t): | ||
| return v.lower() |
There was a problem hiding this comment.
parse_verdict() can misclassify negated replies as supported.
Line 23 matches bare SUPPORTED, so a reply like “not supported by source” can be interpreted as supported, which violates fail-closed behavior.
Proposed hardening
def parse_verdict(text: str) -> str:
"""Map a judge reply to a status; unparseable -> 'unverified' (fail-closed)."""
t = text.upper()
+ # Handle common negation before bare keyword matching.
+ if re.search(r"\bNOT\s+SUPPORTED\b", t):
+ return "unsupported"
for v in ("CONTRADICTED", "UNSUPPORTED", "PARTIAL", "SUPPORTED"):
if re.search(rf"\b{v}\b", t):
return v.lower()
return "unverified"🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/claims/verifier.py` around lines 22 - 24, parse_verdict currently
returns 'supported' because the loop (v in
("CONTRADICTED","UNSUPPORTED","PARTIAL","SUPPORTED")) matches the bare word
SUPPORTED even when preceded by negation (e.g., "not supported"); update the
regex used when v == "SUPPORTED" to exclude common negation patterns before the
word. Concretely, in parse_verdict where you iterate over v and use
re.search(rf"\b{v}\b", t), special-case SUPPORTED to use a negative-lookbehind
or explicit pattern like
r"(?<!\b(?:not|no|n't|never|cannot|can't)\b\s*)\bSUPPORTED\b" (with
re.IGNORECASE) so phrases containing negation do not return 'supported'; keep
the existing checks for "CONTRADICTED", "UNSUPPORTED", and "PARTIAL" unchanged.
| pending = await store.pull_unverified(limit=batch) | ||
| fresh = [c for c in pending if c["id"] not in seen] | ||
| if not fresh: | ||
| break |
There was a problem hiding this comment.
Fixed-window batching can starve claims beyond the first batch.
Lines 29-32 always repull the earliest unverified rows. If those stay unverified, fresh becomes empty and the loop exits before later IDs are ever attempted.
Use cursor-based pagination (WHERE id > last_seen_id ORDER BY id LIMIT ?) so each unverified claim is attempted once per pass, even when many remain fail-closed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/claims/verify_pass.py` around lines 29 - 32, The current fixed-window
repull using store.pull_unverified(limit=batch) plus filtering by seen causes
starvation; change the loop to use cursor-based pagination by tracking
last_seen_id and calling a paginated pull (e.g.,
store.pull_unverified_after(last_seen_id, limit=batch) or add a parameter to
store.pull_unverified to do WHERE id > last_seen_id ORDER BY id LIMIT ?), update
last_seen_id to the max id returned each fetch, and remove the fresh filtering
that stops the loop early (or only skip already-seen ids while continuing the
cursor). Ensure the new query returns rows ordered by id so every unverified
claim is attempted once per pass even if earlier rows remain unverified.
| spans = await fetch_spans(claim["archive_span_ids"]) | ||
| status, model = verifier.verify(claim["text"], spans) | ||
| if status == "unverified": | ||
| # fail-closed: stays unverified, retried in a future pass. | ||
| continue | ||
| await store.set_status(claim["id"], status, verifier_model=model, now=now) |
There was a problem hiding this comment.
Per-claim exceptions currently abort the whole verification pass.
On Lines 35-40, exceptions from fetch_spans, verifier.verify, or set_status bubble out and terminate the pass. For fail-closed behavior, these should leave the claim unverified and continue with the next claim.
Proposed fail-closed guard
for claim in fresh:
seen.add(claim["id"])
- spans = await fetch_spans(claim["archive_span_ids"])
- status, model = verifier.verify(claim["text"], spans)
+ try:
+ spans = await fetch_spans(claim["archive_span_ids"])
+ status, model = verifier.verify(claim["text"], spans)
+ except Exception: # fail-closed at per-claim granularity
+ continue
if status == "unverified":
# fail-closed: stays unverified, retried in a future pass.
continue
- await store.set_status(claim["id"], status, verifier_model=model, now=now)
- done += 1
+ try:
+ await store.set_status(claim["id"], status, verifier_model=model, now=now)
+ done += 1
+ except Exception:
+ continue🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/claims/verify_pass.py` around lines 35 - 40, Per-claim exceptions in
the verification loop currently abort the whole pass; wrap the per-claim work
(the calls to fetch_spans, verifier.verify, and store.set_status for each claim)
in a try/except so any exception is caught, logged, and the loop continues
(leaving that claim unverified) rather than re-raising; specifically, enclose
the sequence spanning fetch_spans(claim["archive_span_ids"]),
verifier.verify(claim["text"], spans), and the subsequent await
store.set_status(...) in a try block and in except log the exception and
continue to the next claim to preserve fail-closed behavior.
| import httpx # noqa: PLC0415 | ||
|
|
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Verify whether httpx is declared in packaging/dependency manifests.
fd -i 'pyproject.toml'
fd -i 'setup.py'
fd -i 'setup.cfg'
fd -i 'requirements.*\.txt'
rg -n --iglob 'pyproject.toml' --iglob 'setup.py' --iglob 'setup.cfg' --iglob 'requirements*.txt' '\bhttpx\b'Repository: jaylfc/taosmd
Length of output: 93
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# 1) Inspect the actual code around the reported lines in taosmd/cli.py
python3 - <<'PY'
import itertools
path = "taosmd/cli.py"
start = 980
end = 1030
with open(path, "r", encoding="utf-8") as f:
for i, line in enumerate(f, 1):
if start <= i <= end:
print(f"{i}:{line.rstrip()}")
if i > end:
break
PY
# 2) Find all occurrences of "httpx" in packaging/manifest files
rg -n --hidden --no-ignore -S '\bhttpx\b' pyproject.toml setup.py setup.cfg taosmd/auto_setup.py requirements*.txt 2>/dev/null || true
# 3) If pyproject.toml exists, print the dependency sections (small, targeted extraction)
if [ -f pyproject.toml ]; then
echo "---- pyproject.toml dependency sections (context around 'httpx') ----"
rg -n -S 'httpx' pyproject.toml -n
rg -n -S '(\[project\]|\[tool\.poetry\.dependencies\]|\[project\.dependencies\]|\[project\.optional-dependencies\])' pyproject.toml || true
fi
# 4) If requirements files exist, show their contents around httpx
if ls requirements*.txt >/dev/null 2>&1; then
for f in requirements*.txt; do
echo "---- $f (context around 'httpx') ----"
rg -n -C 2 -S '\bhttpx\b' "$f" || true
done
fiRepository: jaylfc/taosmd
Length of output: 1985
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate dependency manifests
ls -la pyproject.toml setup.py setup.cfg requirements*.txt 2>/dev/null || true
# Search for declared dependency on httpx in pyproject.toml (and common sections)
if [ -f pyproject.toml ]; then
echo "---- pyproject.toml lines mentioning httpx ----"
rg -n -S '\bhttpx\b' pyproject.toml || true
echo "---- pyproject.toml dependency sections headers ----"
rg -n -S '(\[project\]|\[project\.dependencies\]|\[project\.optional-dependencies\]|\[tool\.poetry\.dependencies\]|\[tool\.poetry\.group\.[^]]*\.dependencies\])' pyproject.toml || true
fi
# Search for httpx across the repo to see if it’s declared anywhere else
rg -n -S '\bhttpx\b' . || trueRepository: jaylfc/taosmd
Length of output: 8554
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show pyproject dependency blocks with line numbers
python3 - <<'PY'
import itertools
p="pyproject.toml"
with open(p,encoding="utf-8") as f:
for i,line in enumerate(f,1):
if 1 <= i <= 80:
pass
# print first 120 lines (pyproject is small: 1513 bytes)
with open(p,encoding="utf-8") as f:
for i,line in enumerate(f,1):
if i<=120:
print(f"{i}:{line.rstrip()}")
else:
break
PY
# Check if requirements*.txt exist and contain httpx
ls -1 requirements*.txt 2>/dev/null || true
for f in requirements*.txt; do
[ -f "$f" ] || continue
echo "---- $f (httpx matches) ----"
rg -n -S '\bhttpx\b' "$f" || true
done
# Inspect auto_setup.py for any dynamic install of httpx
python3 - <<'PY'
import re
path="taosmd/auto_setup.py"
with open(path,encoding="utf-8") as f:
for i,line in enumerate(f,1):
if i<=260:
if 'httpx' in line or 'install' in line or 'requirements' in line:
print(f"{i}:{line.rstrip()}")
PY
# Confirm whether any of the packaging manifests mention httpx at all
rg -n -S '\bhttpx\b' pyproject.toml setup.py setup.cfg requirements*.txt 2>/dev/null || trueRepository: jaylfc/taosmd
Length of output: 2201
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# 1) Inspect taosmd/auto_setup.py around any httpx or dependency-install logic
python3 - <<'PY'
import re
path="taosmd/auto_setup.py"
with open(path,encoding="utf-8") as f:
lines=f.readlines()
# Print only lines that reference installs/requirements/httpx, plus a small window
targets=[]
for i,line in enumerate(lines,1):
if re.search(r'httpx', line, re.I) or re.search(r'pip\s+install|requirements|dependency|install_requires', line, re.I):
targets.append(i)
for i in targets[:80]:
start=max(1,i-10); end=min(len(lines), i+10)
print(f"--- context {start}-{end} around line {i} ---")
for j in range(start,end+1):
print(f"{j}:{lines[j-1].rstrip()}")
PY
# 2) Find where auto_setup is invoked from the CLI path
rg -n -S 'auto_setup' taosmd/cli.py taosmd/*.py || true
# 3) Inspect taosmd/cli.py around verify command wiring (to see if any setup/auto-install runs)
python3 - <<'PY'
path="taosmd/cli.py"
with open(path,encoding="utf-8") as f: lines=f.readlines()
# Look for _verify_cmd usage and any call to auto_setup/main setup
for i,line in enumerate(lines,1):
if 'def _verify_cmd' in line or '_verify_cmd(' in line or 'verify' in line and 'add_parser' in line:
start=max(1,i-40); end=min(len(lines), i+40)
print(f"--- context {start}-{end} around line {i} ---")
for j in range(start,end+1):
print(f"{j}:{lines[j-1].rstrip()}")
PYRepository: jaylfc/taosmd
Length of output: 10748
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -S '\bhttpx\b' taosmd/auto_setup.py || true
rg -n -S 'pip\s+install|requirements|install_requires|dependency|ensurepip|subprocess\..*pip' taosmd/auto_setup.py || trueRepository: jaylfc/taosmd
Length of output: 39
taosmd verify imports httpx at runtime, but httpx isn’t declared in dependencies.
taosmd/cli.py imports httpx in the verify command handler, but pyproject.toml’s project.dependencies does not include httpx, and taosmd/auto_setup.py doesn’t install it—so taosmd verify can raise ModuleNotFoundError. Declare httpx in project.dependencies (or a dedicated extra) or handle ModuleNotFoundError with a clear install hint.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/cli.py` around lines 998 - 999, The verify command in taosmd/cli.py
imports httpx at runtime (see the import added for the verify handler) but httpx
is not declared or installed; either add "httpx" to project.dependencies in
pyproject.toml (or create a dedicated extra) so it’s installed, or modify the
verify handler to catch ModuleNotFoundError around the httpx import and raise a
clear error message instructing the user to pip install httpx (and update
taosmd/auto_setup.py if you choose to auto-install it so the setup logic
installs the new dependency).
| verify_p.add_argument( | ||
| "--batch", type=int, default=100, | ||
| help="Claims to pull per batch (default: 100)", | ||
| ) |
There was a problem hiding this comment.
Validate --batch as a positive integer.
Line 1489 accepts non-positive values. --batch 0 or negative values can lead to unintended query behavior and effectively disable batching semantics.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@taosmd/cli.py` around lines 1488 - 1491, The --batch argument currently uses
verify_p.add_argument("--batch", type=int, ...) and accepts zero or negative
values; add a validator function (e.g., def positive_int(value): ...) that
converts to int, checks value > 0, and raises argparse.ArgumentTypeError on
failure, then change verify_p.add_argument to use type=positive_int and keep the
same default/help; this ensures --batch only accepts positive integers and
produces a clear argparse error message when invalid.
| try: | ||
| asyncio.run(s.set_status(cid, "bogus", verifier_model="m", now=1.0)) | ||
| assert False, "expected ValueError" | ||
| except ValueError: | ||
| pass |
There was a problem hiding this comment.
Use explicit failure instead of assert False in the exception path.
Line 37 uses assert False, which triggers Ruff B011 and is less robust than an explicit exception.
Proposed fix
try:
asyncio.run(s.set_status(cid, "bogus", verifier_model="m", now=1.0))
- assert False, "expected ValueError"
+ raise AssertionError("expected ValueError")
except ValueError:
pass🧰 Tools
🪛 Ruff (0.15.15)
[warning] 37-37: Do not assert False (python -O removes these calls), raise AssertionError()
Replace assert False
(B011)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@tests/claims/test_store.py` around lines 35 - 39, Replace the fragile "assert
False, 'expected ValueError'" in the test for s.set_status with an explicit test
failure call (e.g., pytest.fail("expected ValueError")) so the test triggers a
clear failure instead of using assert False; update the test in
tests/claims/test_store.py around the s.set_status(cid, "bogus",
verifier_model="m", now=1.0) try/except block to call pytest.fail in the try
branch and keep the except ValueError: pass as the expected path.
Source: Linters/SAST tools
1 participant
What
The v2 spine after surprisal died (N-009/N-010/N-011): an additive, default-off Provable Memory layer that makes F-009 (the measured 18.8 percent extraction-hallucination rate) a live, always-on gate. Built subagent-driven from the approved private spec; 9 tasks, TDD throughout.
How it works
taosmd/claims/extract.py).verify_pass.py) shows a cross-family local judge ONLY the cited spans and marks each claim supported / partial / unsupported / contradicted. Any failure leaves it unverified, never promoted.gate.py) behindsearch(prefer_verified=...)(defaultoff); verified preferred, unsupported excluded from default recall, everything still in the zero-loss archive and queryable (reversible).store.py): zero-loss sqlite, rebuildable from the archive, exposes the live hallucination rate. Store-mode gate is a pure function.taosmd verify,taosmd claims status.Safety / scope
prefer_verifieddefaultsoff, so existing installs are byte-for-byte unchanged in behaviour. 886 tests pass (+24 claims), the claims package only importsmemory_extractor+_db(clean isolation).benchmarks/claims_gate_probe.py), the actual default flip.Review
Not self-merged: it touches the live recall path (though default-off). Spec is private (
~/tinyagentos-private/specs/2026-06-13-v2-claims-layer-design.md); the public spec ships when E-009 validates.Summary by CodeRabbit
Release Notes
New Features
prefer_verifiedparameter in search to optionally demote unverified claimstaosmd claims status(view verification statistics) andtaosmd verify(verify unverified claims)Documentation