docs(feature-doc-linker): spec memo for #435 - #453
Conversation
Belief↔document linker — distinct from CITES (belief→belief, models.py:27) and from BM25F anchor text (#148, doc-side incoming-edge text). Schema: new belief_documents table (sibling of belief_corroborations and belief_neighbors) keyed (belief_id, doc_uri), CASCADE on belief delete. anchor_type enum: ingest | manual | derived (last reserved). Invocation: ingest-time only at v2.0.0, hooked into the DerivationInput.source_path path (derivation.py:64,:223). Retrieval- time inference (anchor_type="derived") is reserved but deferred. Surface: opt-in with_doc_anchors=True kwarg on retrieve() projects parallel doc_anchors list onto RetrievalResult. Default OFF, byte- stable pack contract. Anchors are metadata, not body — no token- budget cost. Bench-gate: NDCG@k uplift on a labeled doc_linker fixture under tests/corpus/v2_0/, comparing same beliefs with anchors populated vs unpopulated. URI scheme: opaque TEXT. file://abs/path#Lstart-Lend for local ingest, https://... for external; validation deferred.
Reviewer's GuideAdds a detailed feature spec document for the planned document/semantic linker (#435), defining storage schema, invocation points, retrieval-surface behavior, configuration, acceptance criteria, and out-of-scope items, without changing any code. Sequence diagram for ingest-time document linkingsequenceDiagram
actor Operator
participant CLI as aelf_remember
participant Derivation as DerivationPipeline
participant Store as MemoryStore
participant DB as belief_documents_table
Operator->>CLI: run aelf remember --doc=URI
CLI->>Derivation: submit content with source_path
Derivation->>Store: link_belief_to_document(belief_id, doc_uri, anchor_type="ingest", position_hint)
Store->>DB: INSERT OR IGNORE (belief_id, doc_uri, anchor_type, position_hint, created_at)
DB-->>Store: success
Store-->>Derivation: DocAnchor
Derivation-->>CLI: confirm belief stored with doc anchor
CLI-->>Operator: output success message
Sequence diagram for retrieval with doc anchorssequenceDiagram
actor Client
participant Retrieval as RetrievalService
participant Store as MemoryStore
participant DB as belief_documents_table
Client->>Retrieval: retrieve(query, with_doc_anchors=True)
Retrieval->>Store: fetch ranked beliefs for query
Store-->>Retrieval: list of beliefs
Retrieval->>Store: get_doc_anchors(belief_ids_batch)
Store->>DB: SELECT * FROM belief_documents WHERE belief_id IN (belief_ids_batch)
DB-->>Store: rows for matching anchors
Store-->>Retrieval: grouped doc anchors per belief
Retrieval-->>Client: RetrievalResult(beliefs, doc_anchors)
Entity relationship diagram for new belief_documents tableerDiagram
beliefs {
TEXT id PK
TEXT content
REAL created_at
}
belief_documents {
TEXT belief_id FK
TEXT doc_uri
TEXT anchor_type
TEXT position_hint
REAL created_at
}
ingest_log {
INTEGER id PK
TEXT source_path
TEXT source_kind
REAL created_at
}
beliefs ||--o{ belief_documents : anchors
beliefs ||--o{ ingest_log : ingests
Class diagram for DocAnchor and store/retrieval integrationclassDiagram
class DocAnchor {
+str belief_id
+str doc_uri
+str anchor_type
+str position_hint
+float created_at
}
class MemoryStore {
+link_belief_to_document(store, belief_id, doc_uri, anchor_type, position_hint) DocAnchor
+get_doc_anchors(store, belief_id) list_DocAnchor
}
class RetrievalResult {
+list_beliefs beliefs
+list_list_DocAnchor doc_anchors
}
class RetrievalService {
+retrieve(query, with_doc_anchors) RetrievalResult
+retrieve_v2(query, with_doc_anchors) RetrievalResult
}
MemoryStore "1" o-- "many" DocAnchor : persists
RetrievalResult "1" o-- "many" DocAnchor : projects
RetrievalService ..> MemoryStore : uses
RetrievalService ..> RetrievalResult : returns
Flow diagram for doc linker in ingest and retrieval pipelinesflowchart TD
IngestLog[Ingest_log_events]
Derive[Derivation_worker]
DocLinker[Doc_linker_writer]
Beliefs[Beliefs_table]
BeliefDocs[Belief_documents_table]
Retrieve[Retrieval_service]
Client[Client_or_tooling]
IngestLog --> Derive
Derive --> Beliefs
Derive --> DocLinker
DocLinker --> BeliefDocs
Client --> Retrieve
Retrieve --> Beliefs
Retrieve --> BeliefDocs
Retrieve --> Client
File-Level Changes
Possibly linked issues
Tips and commandsInteracting with Sourcery
Customizing Your ExperienceAccess your dashboard to:
Getting Help
|
|
Warning Rate limit exceeded
To keep reviews running without waiting, you can enable usage-based add-on for your organization. This allows additional reviews beyond the hourly cap. Account admins can enable it under billing. ⌛ How to resolve this issue?After the wait time has elapsed, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout. Please see our FAQ for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 1 issue
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="docs/feature-doc-linker.md" line_range="29" />
<code_context>
+ doc_uri: str # see "Doc URI scheme" below
+ anchor_type: str # "ingest" | "manual" | "derived"
+ position_hint: str | None # e.g. "L42-L60", "#section-name"; nullable
+ created_at: float # unix timestamp
+
+# Ingest-time: invoked by onboard / commit-ingest when source_path is known.
</code_context>
<issue_to_address>
**nitpick (typo):** Consider capitalizing "Unix" in the comment for `created_at`.
This matches the conventional term "Unix timestamp" used in most documentation.
```suggestion
created_at: float # Unix timestamp
```
</issue_to_address>Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.
| doc_uri: str # see "Doc URI scheme" below | ||
| anchor_type: str # "ingest" | "manual" | "derived" | ||
| position_hint: str | None # e.g. "L42-L60", "#section-name"; nullable | ||
| created_at: float # unix timestamp |
There was a problem hiding this comment.
nitpick (typo): Consider capitalizing "Unix" in the comment for created_at.
This matches the conventional term "Unix timestamp" used in most documentation.
| created_at: float # unix timestamp | |
| created_at: float # Unix timestamp |
|
[claim:review:Kulili:2026-05-05T19:46:07Z] |
|
Reviewed at Spec-memo quality:
Three flags for impl-PR review (not blocking the spec):
Spec is mergeable as-is; the three points belong in the impl-PR conversation, not in another spec revision. Merging by FF push. |
|
[release:review:Kulili:2026-05-05T19:47:32Z] |
2 participants
Spec memo for #435 — Document / semantic linker. Closes the recovery-inventory line at
docs/ROADMAP.mdrow 163 (Doc / semantic linker | v2.0.0).What this PR is
Docs-only. New file at
docs/feature-doc-linker.md. Converts the bare issue acceptance sketch into a buildable contract: storage schema, invocation point, retrieval surface, doc URI scheme, bench-gate, and out-of-scope list.No code, no schema, no flag wiring yet. This PR moves #435 from
needs-spectobench-gated.Storage decision
New table
belief_documents(sibling ofbelief_corroborations#190 andbelief_neighbors#227):(belief_id, doc_uri, anchor_type, position_hint, created_at) PK (belief_id, doc_uri); FK belief_id ON DELETE CASCADEanchor_type ∈ {ingest, manual, derived}—derivedreserved for v2.x retrieval-time inference, no writer at v2.0.0.Invocation
Ingest-time only at v2.0.0. Hooked into
derivation.py:64, :223wheresource_pathis materialised.aelf remember --doc=URIadds a CLI surface formanualanchors. Retrieval-time inference deferred (out of scope, would require fuzzy matching on the hot path).Retrieval surface
Opt-in:
retrieve(..., with_doc_anchors=True)projects paralleldoc_anchors: list[list[DocAnchor]]ontoRetrievalResult. DefaultFalsekeeps the byte-stable pack contract. Anchors are metadata, not body — notoken_budgetcost.Reconciliation
edges.anchor_textbelief_documentstableEDGE_CITESis belief→belief; this is belief→document. No overlap.Substrate
All on
mainas of68dafc0:store.py:142-162—belief_corroborationsschema-pattern precedentstore.py:316-330— migration block where the additiveCREATE TABLElandsderivation.py:64, :223—source_pathplumbingderivation_worker.py— sibling writer patterntests/corpus/v2_0/,tests/bench_gate/— corpus + harnessNo new dependencies. One additive (forward-only) schema migration.
Test plan
github/main— clean.G).Refs
docs/ROADMAP.mdrow 163belief_corroborations), [v1.x] Precomputed-neighbor vocabulary bridge — narrow 'find similar to X' lane ahead of full G6 #227 (belief_neighbors)Summary by Sourcery
Add a feature specification document for a belief-to-document linker, defining storage, invocation, retrieval projection, and acceptance criteria for future implementation.
Documentation: