Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ This repo also publishes GitHub Releases. This file is the repo-root release sur
stamp downgrades, deadline-interruptible summary lineage expansion, delta refs rebuilt after response-cap
eviction, spend-ledger completeness on chunked backfills, and the benchmark evidence trail (`bench/`, F20–F37).

- Added nested-default-JSON-bounded, tool-extracted `lcm_expand_query` evidence provenance so successful and degraded answers retain synthesis-context identities, occurrences, paths, and excerpts while explicitly distinguishing locator coverage from unverified replay, semantic entailment, and caller authorization.

## v0.20.0 - 2026-07-23

Release focus: Lossless-Claw parity plus the merged cross-session recall and temporal retrieval stack.
Expand Down
64 changes: 63 additions & 1 deletion docs/retrieval-tools.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ is the active context engine.
| `lcm_load_session` | Load one ordered raw-message transcript page for an explicit `session_id`. This is not search: it returns raw rows in `store_id` order, bounded by `limit`, with per-message content bounded by `max_content_chars`, and continues with `after_store_id` from `next_cursor`. Set `include_exact_ref=true` when rows will feed exact citation or computation; the default response stays byte-compatible. |
| `lcm_describe` | Inspect the current-session DAG or preview an `externalized_ref` without loading full content. |
| `lcm_expand` | Recover source messages, child summaries, or externalized payloads with pagination. Use `store_id` to fetch a single raw message regardless of session, suitable for drilling into a cross-session `lcm_grep` result. In `store_id` mode, `include_exact_ref=true` adds the exact returned slice without changing default bytes. |
| `lcm_expand_query` | Answer a question using expanded current-session LCM context while returning a bounded answer. |
| `lcm_expand_query` | Answer a question using expanded current-session LCM context while returning a bounded answer plus bounded, tool-extracted `evidence_provenance` for the context supplied to synthesis. |
| `lcm_status` | Show runtime health, context pressure, config, source lineage, and lifecycle stats. |
| `lcm_inspect` | Read-only operator inventory for current-session lineage, message/frontier metadata, fresh tail, externalized refs/readability, compaction skip/no-op reasons, and matched ignore/stateless patterns. It returns metadata only; use `lcm_load_session`/`lcm_expand` when you need content. |
| `lcm_doctor` | Run database, FTS, lifecycle, config, and context-pressure diagnostics. |
Expand Down Expand Up @@ -243,6 +243,68 @@ the same 20,000-character response ceiling used by the retrieval tools.
{"period": "date:2026-07-15", "scope": "global"}
```

### `lcm_expand_query` evidence provenance

Every successful, no-match, or structured degraded `lcm_expand_query` response
includes an additive `evidence_provenance` object. On paths that run or attempt
synthesis, the tool builds it from the exact bounded context blocks supplied to
the auxiliary model; no-match returns an explicit empty bundle. It is not
generated by that model.

The bundle includes up to 24 unique summary, raw-message, and externalized-
payload identities in deterministic first-seen context order. Repeated identities
retain `context_occurrence_count` and up to eight distinct source-path records.
Each record preserves up to eight production-shaped `{node_id, source_index}`
hops plus the producer's original `depth` and `truncated` state, rather than
silently discarding recursive occurrence information.

The nested object is capped at 10,000 characters using the same default
`json.dumps` representation used by the final tool response (`ensure_ascii=true`).
The cap applies to `evidence_provenance` only, not to the complete
`lcm_expand_query` response. Whole trailing items are removed until the nested
object fits. String metadata is capped at 256 characters; any shortening reports
the original character count and SHA-256 digest. A compact fixed-envelope
fallback preserves the limit for pathological inputs.

Each quote is capped at 500 characters. `quote_chars_before_provenance_cap` is
the length of the context representation before this 500-character provenance
cap—not necessarily the durable source's original length if retrieval already
sliced it. `quote_truncated_by_provenance_cap` describes only this provenance
cap; `context_truncated` separately describes synthesis-budget omission.

Items retain available bounded node/store/ref, session, source, role,
content-offset, occurrence/path, and direct `lcm_expand` arguments. If hydrated
content differs from its durable `transcript_content`, both representations are
recorded because both were visible in the synthesis context.

Response-level and item-level fields separate four claims:

- `locator_coverage` says whether every unique bounded synthesis-context identity
has locator arguments present (`none`, `partial`, or `complete`); it does not
claim those locators still resolve;
- `locator_replay_status` is `unverified` when locator arguments are present and
`not_available` otherwise;
- `semantic_entailment` is `not_verified` for a completed answer and
`not_applicable` when synthesis did not run or failed;
- `identifiers_are_authority` is always `false`: node/store/session identifiers
locate evidence but do not authenticate a caller or grant access.

`locator_replay_safety` is `not_guaranteed`: current scalar node/store/ref and
offset locators do not provide durable staleness, revision, or database-generation
detection. Locator presence is therefore not proof of exact replay.

This PR adds no authorization mechanism. `node_id` and `externalized_ref` retain
their current-session checks; `store_id` remains an intentional cross-session
locator. Hosts/callers must authorize every expansion before invoking it and must
not treat identifiers as capabilities.

The provenance layer does not claim that a fluent answer is correct, that it
resolved contradictory sources, or that every answer clause is supported.
Existing top-level answer, match, pagination, and degraded-response fields are
unchanged. Unknown-field-tolerant callers remain compatible; strict response
schemas and fixed-size logging/gateway consumers must be updated for the new
nested object.

When temporal rollups are enabled and `ready` rollups cover the **entire**
requested window, the response includes their ids and `ready` status in
`provenance.rollups`. If any day in the window lacks a ready rollup (missing or
Expand Down
1 change: 1 addition & 0 deletions schemas.py
Original file line number Diff line number Diff line change
Expand Up @@ -1146,6 +1146,7 @@
"query matching summaries/raw messages to expand or explicit node_ids to inspect. Uses the expansion path "
"instead of the summarization path so retrieval/synthesis can use a different model or timeout. "
"When expanding parent summary nodes, it recursively descends the DAG under the context budget to include leaf evidence where possible. "
"The response includes a nested, default-JSON-bounded, tool-extracted evidence provenance object with locator coverage, while marking locator replay and semantic entailment as unverified and identifiers as non-authoritative. This adds no authorization: node_id and externalized_ref retain current-session checks, store_id remains an intentional cross-session locator, and hosts must authorize expansion before invoking it. "
"Prefer this for questions about the active conversation after compaction; for cross-session recall, use session_search first."
),
"parameters": {
Expand Down
Loading