Skip to content

feat: add durable message retrieval-reference adapter - #489

Closed
100yenadmin wants to merge 3 commits into
stephenschoettler:mainfrom
100yenadmin:research/message-retrieval-references
Closed

100yenadmin wants to merge 3 commits into
stephenschoettler:mainfrom
100yenadmin:research/message-retrieval-references

Conversation

@100yenadmin

Copy link
Copy Markdown
Contributor

Summary

  • add one internal message retrieval-reference adapter over the V1 authority foundation in feat: add provenance-bound retrieval reference foundation #487;
  • bind each opaque reference to exact positive messages.store_id, caller-supplied authorization scope, and a SHA-256 semantic revision;
  • normalize legacy blank source/conversation fields, content, valid JSON tool_calls, and finite timestamps before hashing;
  • treat content/session/source/conversation/role/tool/timestamp changes, GC tombstones, malformed rows, and malformed registry bindings as typed stale/not-found failures;
  • exclude operational pinned and token_estimate fields from semantic staleness;
  • require host target preauthorization before any transaction/message lookup, making denied existing and missing IDs indistinguishable;
  • preserve authorization-before-registry/message disclosure and the foundation's kind/scope/authority/expiry/revocation taxonomy;
  • issue atomically under BEGIN IMMEDIATE or a caller-owned transaction;
  • resolve under one stable read snapshot and hold MessageStore's re-entrant write lock so shared-connection writers cannot interleave; raw SQLite callers retain caller-managed concurrency;
  • document the adapter boundary without changing public tools, schemas, pagination, principals, or tenants.

Why

#487 establishes a product-neutral opaque authority but intentionally does not load target rows. Before strong mutation-safe pagination (#477) can reuse that authority, one target adapter must prove exact target/revision binding, authorization-before-disclosure, transactional rollback, and mutation behavior end to end.

This PR is that single vertical. It does not change tools.py, public schemas, or numeric schema version 5.

Verification

Sandboxed root matrix with both HERMES_HOME and LCM_DATABASE_PATH isolated:

python -m pytest -q -p no:cacheprovider \
  tests/test_message_references.py \
  tests/test_retrieval_reference_contract.py \
  tests/test_db_bootstrap_fts.py \
  tests/test_schema_stamp_remediation.py \
  tests/test_packaging_install.py

Result: 147 passed (including packaging/install coverage).

Focused adapter suite: 30 passed, including:

  • round trip and exact normalized revision;
  • digest-only registry persistence;
  • authorization denial before registry/message SELECT;
  • equal no-I/O forbidden behavior for denied existing/missing issuance targets;
  • content, scope, role, tool, timestamp, GC, delete, expiry, revoke, clone, kind, and malformed-binding behavior;
  • caller rollback and callback-denial behavior;
  • WAL mutation snapshot stability;
  • same-connection MessageStore writer exclusion;
  • raw SQLite connection facade.
python -m ruff check message_references.py retrieval_references.py db_bootstrap.py \
  tests/test_message_references.py tests/test_retrieval_reference_contract.py
python -m py_compile message_references.py retrieval_references.py db_bootstrap.py \
  tests/test_message_references.py tests/test_retrieval_reference_contract.py
git diff --check
git diff --cached --check

All clean.

Stack and scope

Refs #476.

@100yenadmin

Copy link
Copy Markdown
Contributor Author

Closing in favour of #505, the maintenance roll-up — this is commit 3 of that stack.

Nothing is dropped — the commits are carried across unchanged, so the review history here stays meaningful and the work is not rewritten. The consolidation is packaging: a system-level change reads better as one ordered stack than as several PRs that have to be merged in the right sequence to make sense.

Apologies for the churn on your queue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant