refactor(paper): source-first artifacts and semantic Paper Flow V1

[keli-wen]

✨ Feature Summary

Refactor the paper domain, storage model, and paper_flow() around three explicit layers: an exact source revision, independently versioned paper artifacts, and rebuildable search projections.

Paper Flow V1 must build a page-aware PaperChunkSet first and generate one cited PaperGlobalSummary from that chunk set second. LocalKnowledgeLibrary.put_paper() then persists the source and both artifacts and makes them searchable with OpenAI text embeddings. V1 deliberately does not build a paper tree.

This issue supersedes #112, whose expected output is still a canonical Paper / TreeKnowledge. It uses the parsing, LlamaIndex, and semantic-library foundation completed by #117 and removes the failing final-Paper structured-output path reported in #91 instead of weakening canonical IDs.

🎯 Motivation

The current model treats a paper itself as Paper(TreeKnowledge), treats its summary as PaperKnowledgeCard linked to that tree, and requires every BaseKnowledge subtype to provide embedding_text(). These choices couple paper identity, one optional structural representation, retrieval grain, and one indexing technique.

A paper can instead have several independent derived representations. A global summary is useful for document selection and broad understanding. A chunk set is useful for passage-level semantic retrieval. A future paper tree is useful for structure-aware and agentic navigation, while its individual nodes may also participate in semantic retrieval. None of those artifacts should be the identity of the source paper, and no artifact should own an embedding as canonical domain data.

The storage boundary matters because it determines whether QuantMind can safely re-split a paper, rebuild embeddings with another model, generate a new summary, construct a PageIndex-style tree later, or compare multiple artifact versions without downloading and parsing the source again.

📋 Design Contract

1. Source revision is the shared anchor

Introduce one immutable paper source revision identified by the exact fetched content hash. It owns or references:

• the original paper blob and resolved source metadata, including the exact arXiv revision;
• the parsed page-aware representation with ordered pages, text blocks, bounding boxes, and parser/version metadata;
• image, screenshot, and other asset references;
• publication, availability, fetch, and provenance facts that come from the source rather than a model.

The local library owns the lifecycle of this source revision. SQLite does not need to inline PDF or image bytes; content-addressed sidecar blobs are acceptable, but their hashes, locations, and parsed manifest must be tracked transactionally enough that stored artifacts never point to missing or ambiguous source content.

2. Artifacts are independent derived products

The same source revision may own any number of independently versioned artifacts:

• PaperChunkSet: an ordered collection of chunks created by a named splitter and exact configuration;
• PaperGlobalSummary: one global summary text with citations and generation lineage;
• future PaperTree: a hierarchy whose nodes contain navigation summaries and source or asset references.

Artifacts share the source revision but do not require one another at the schema level. Paper Flow V1 generates the summary from a selected chunk set, so the summary records that chunk set as generation lineage; this does not make chunk IDs part of the summary's identity or require that chunk set for later summary retrieval. A future tree may be built directly from the parsed source without using the V1 chunk set.

A source revision may have multiple chunk sets, summaries, or trees produced by different configurations. Re-running an unchanged producer configuration should be idempotent; changing the parser, splitter, prompt, model, or relevant configuration creates or refreshes only the affected derived artifacts.

3. Artifact members are directly addressable

Do not introduce a public SearchUnit domain type. The searchable members already exist inside each artifact:

• a global summary is addressed by its artifact ID;
• a chunk is addressed by its chunk-set artifact ID and chunk ID;
• a future tree node is addressed by its tree artifact ID and node ID.

Chunks preserve their text, ordered source spans, page ownership, and asset references. They do not duplicate image bytes. Future tree nodes preserve topology, title, navigation summary, and source or asset references without forcing the tree to depend on a chunk set.

4. Search projections are rebuildable

A search projection converts an artifact or one of its members into input for a retrieval method. It is keyed by an artifact locator plus projection kind, projection version, model, dimensions, and content hash.

Canonical artifacts do not store embedding vectors and no longer need to define embedding_text(). Text selection and other projection behavior belong to the library's indexing boundary. One artifact member may later have text, image, or multimodal projections without changing its canonical schema.

V1 creates OpenAI text-embedding projections for the global summary and every non-empty chunk. LlamaIndex continues to own the private vector retrieval and ranking mechanics, while SQLite and the content-addressed source store remain the durable source of truth. LlamaIndex types must not enter public QuantMind models or persisted canonical payloads.

Semantic hits return an artifact locator, matched projection details, score, and source evidence. A future hit on a tree node can seed tree traversal through its tree_id + node_id; tree navigation and semantic retrieval are complementary serving strategies over the same stored artifact, not competing storage models.

🔧 Paper Flow V1

The required operation order is:

1. paper_flow() resolves and fetches one exact paper version.
2. paper_flow() parses and prepares the immutable source revision.
3. paper_flow() builds one page-aware PaperChunkSet with LlamaIndex and retains source spans plus image references.
4. paper_flow() generates one cited PaperGlobalSummary from the chunk-set manifest and content.
5. paper_flow() returns a typed PaperFlowResult that contains the source revision, chunk set, and global summary.
6. LocalKnowledgeLibrary.put_paper() validates and persists the source, artifacts, and lineage, then creates or reuses OpenAI text-embedding search projections for the chunks and summary.
7. LocalKnowledgeLibrary.search() retrieves summary or chunk members through those projections.

The summary path should use the OpenAI Agents SDK without asking a model to emit canonical IDs, links, source metadata, or final storage payloads. Code owns identity and provenance. The summarization agent may adaptively read or delegate over chunk groups through bounded tools, but code must enforce limits on calls, concurrency, input tokens, output tokens, and total runtime and must validate citation and source coverage before accepting the summary.

A successful paper_flow() result means the source revision, chunk set, and cited summary are valid. A successful V1 E2E run additionally means put_paper() persisted those values and their required search projections are retrievable. Fetching a PDF, parsing pages, producing only chunks, or receiving an uncited summary is not a successful flow.

API direction

Exact names may be refined in contexts/, but the public workflow should remain this small:

result = await paper_flow(
    ArxivIdentifier(id="1706.03762v7"),
    cfg=PaperFlowCfg(model="gpt-4o-mini"),
)

library = await LocalKnowledgeLibrary.open(
    ".quantmind/library.db",
    embedding_model="text-embedding-3-small",
)
await library.put_paper(result)

summary_hits = await library.search(
    SemanticQuery(
        text="What is the paper's central contribution?",
        artifact_kinds=["paper_summary"],
        top_k=3,
    )
)
passage_hits = await library.search(
    SemanticQuery(
        text="How does multi-head attention work?",
        artifact_kinds=["paper_chunk_set"],
        top_k=5,
    )
)

put_paper() is not a trivial wrapper: it owns validation, source/artifact persistence, projection generation or reuse, and cross-record consistency. The implementation may choose an equivalent typed API if the context design shows a cleaner transaction boundary.

💾 Storage Direction

Update the local storage design to represent these responsibilities explicitly:

• source revisions and parsed source manifests;
• content-addressed raw and image asset references;
• canonical artifact aggregates;
• separately addressable artifact members such as chunks and future tree nodes;
• artifact lineage and producer configuration hashes;
• rebuildable search projections and embeddings.

The implementation may evolve the current knowledge_items, knowledge_nodes, and semantic_records tables or replace them with a migration. The schema must preserve foreign-key integrity, content hashes, idempotent writes, reopen behavior, and selective projection rebuilds. It must not silently reinterpret an existing database under the new model.

🧱 Required Refactor

• Replace Paper(TreeKnowledge) as the primary paper identity and output contract.
• Replace the mandatory PaperKnowledgeCard.paper_id -> Paper.id relationship with independent artifacts anchored to one source revision.
• Move paper embedding-text selection out of canonical Pydantic models and into search-projection code; migrate other existing knowledge types without regressing their library search behavior.
• Teach LocalKnowledgeLibrary to persist the source/artifact model and return artifact-member locators from semantic search.
• Make paper_flow() consume ParsedDocument and quantmind.rag instead of flattened Markdown.
• Keep IDs, source facts, artifact membership, lineage, and citations code-owned.
• Update contexts/design/flow/paper.md, contexts/design/library/local.md, contexts/design/rag/document.md, and the paper knowledge design so they describe Source, Artifact, and SearchProjection consistently.
• Add examples/flows/paper.py as the focused user example and evolve the bounded E2E script into the V1 verification path.
• Add unittest.TestCase-based offline tests with deterministic parser, summarizer, and embedding seams.

✅ Acceptance Criteria

Domain and persistence

• One exact source revision stores or durably references the original paper, parsed pages and blocks, assets, content hash, parser metadata, and source timing facts.
• PaperChunkSet and PaperGlobalSummary are separate versioned artifacts anchored to the same source revision.
• A summary records the chunk set used to generate it as lineage without making the two artifact schemas mutually dependent.
• Multiple chunk sets can coexist for one source revision when their splitter configuration differs.
• Chunk and summary IDs are created by code, and model output cannot override identity, provenance, source spans, or storage links.
• Canonical artifacts contain no embedding vectors.
• Search projections record enough model, version, dimension, modality, and content-hash data to rebuild only stale projections.
• Closing and reopening the library preserves the source and artifacts and rebuilds private LlamaIndex retrieval state without re-embedding unchanged projections.
• Existing supported non-paper knowledge remains storable and searchable, or an explicit migration and compatibility boundary is documented and tested.

Flow behavior

• paper_flow() builds the chunk set before starting global-summary synthesis.
• Every chunk retains exact source revision identity, page ownership or character spans, and available image or screenshot references.
• The summary cites existing chunks and pages, and code rejects unknown citation locators.
• The Agents SDK summarization path is adaptively usable but bounded by explicit runtime, tool-call, concurrency, and token limits.
• V1 returns no PaperTree and does not ask the model to produce TreeKnowledge.
• A failed summary prevents paper_flow() success, and a failed required projection prevents put_paper() from reporting E2E persistence success; retry behavior is idempotent.

Offline verification

• The existing fixed PDF fixture validates parsing, source persistence, deterministic splitting, artifact round trips, locator resolution, projection invalidation, and semantic ranking with fake embeddings.
• Tests inherit from unittest.TestCase or its async subclass.
• bash scripts/verify.sh passes without live network or OpenAI access.

Attention Is All You Need vertical slice

• The focused example and bounded smoke use exact arXiv revision 1706.03762v7.
• The fetched and parsed source has 15 physical pages and preserves page-aware text plus available asset references.
• The run produces a non-empty PaperChunkSet before producing a non-empty PaperGlobalSummary.
• The run uses OpenAI text-embedding-3-small for summary and chunk projections.
• A filtered chunk search for “How does multi-head attention work?” returns a passage from the Multi-Head Attention section within the top five results and resolves it to its source page.
• A filtered summary search for “What is the paper's central contribution?” returns the stored global-summary artifact.
• The global summary accurately covers the attention-only architecture, the removal of recurrence and convolution, the encoder-decoder or multi-head-attention design, and the reported translation or training-efficiency result.
• The global summary provides valid citations to at least three existing chunks across at least two source pages.
• The example closes and reopens the library, repeats both searches, resolves each hit to its canonical artifact member, and prints the summary, chunk count, source pages, scores, and citations for inspection.
• The live smoke is timeout-bounded and remains separate from the deterministic verification suite.

🚫 Non-goals

• Building PaperTree, PageIndex navigation, Graph RAG edges, or answer synthesis in V1.
• Image embeddings or multimodal reranking; V1 preserves image references and uses OpenAI text embeddings.
• A public vector-store, embedder, retriever, backend, or LlamaIndex type hierarchy.
• Weakening UUID or canonical identity validation to accommodate model-generated storage structures.
• Supporting image-only OCR, authenticated papers, or unresolved DOI content in this refactor.

🔗 Related Issues

• Supersedes feat(example): paper extraction to semantic knowledge search #112 because that issue assumes paper_flow -> Paper / TreeKnowledge.
• Uses the completed parsing and LlamaIndex foundation from feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval #117.
• Resolves the paper_flow() failure mode in paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91 by removing direct model generation of the final tree-shaped Paper; close paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91 only after the new V1 path lands and its regression is verified.
• Establishes the source/artifact/search-projection storage contract that feat: page-preserving TreeKnowledge build + agentic navigation #95 should use for future PaperTree construction, semantic node recall, and agentic tree navigation.

Implementation Considerations

Breaking Changes

• This refactor intentionally changes the current paper knowledge and paper_flow() contracts.
• This feature is backward compatible.

Dependencies

• Requires a new retrieval framework dependency.
• Uses the existing required LlamaIndex, OpenAI SDK, and Agents SDK dependencies.

---

Checklist

• I have searched existing issues to avoid duplicates.
• I have provided a clear and detailed description.
• I have explained the motivation and use cases.
• I have considered the implementation approach.
• I have thought about breaking changes and migration.