feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval

[keli-wen]

✨ Feature Summary

Adopt LlamaIndex as QuantMind's required document RAG foundation across PDF ingestion, chunking, indexing, and retrieval. Replace the current flattened PDF preprocessing path with a page-aware multimodal representation, and replace LocalKnowledgeLibrary's private NumPy exact-cosine ranking with LlamaIndex retrieval while preserving QuantMind's canonical knowledge, provenance, persistence, and public result types.

This is a vertical-slice issue: prove the design with the pinned Attention Is All You Need PDF and the existing offline golden PDF, then fix the pipeline at each failing boundary.

🎯 Motivation

The current paper path successfully resolves and downloads PDFs, but pdf_to_markdown() concatenates only non-empty page text. It loses page boundaries, empty pages, layout coordinates, and image context before chunking or tree construction can use them.

QuantMind also maintains its own embedding storage and NumPy exact-cosine ranker inside LocalKnowledgeLibrary. That implementation works for the current local scale, but maintaining custom ingestion, chunking, indexing, retrieval, and ranking would duplicate mature RAG infrastructure and make future pipeline composition harder.

LlamaIndex should provide the document-local and collection-wide semantic RAG data plane. quantmind.rag owns the opinionated document operations, while LocalKnowledgeLibrary privately uses LlamaIndex for collection-wide ranking. QuantMind continues to own domain contracts: exact source identity, financial time semantics, canonical Knowledge and TreeKnowledge, citations and provenance, SQLite persistence, and typed public evidence. LocalKnowledgeLibrary remains a canonical knowledge library with rebuildable retrieval capabilities; it is not defined as a vector database.

📋 Detailed Description

The target document path is:

paper input
  -> exact source bytes and metadata
  -> LiteParse
  -> quantmind.preprocess ParsedDocument with ordered pages, text blocks, bounding boxes, and page screenshots/image references
  -> quantmind.rag LlamaIndex transformations, chunking, indexing, and retrieval
  -> ParsedDocumentHit evidence with source hash, page metadata, and artifact references
  -> optional canonical Paper assembly
  -> optional LocalKnowledgeLibrary storage/search of resulting Knowledge

The architecture has six ownership boundaries:

• quantmind.preprocess owns deterministic source parsing. It ends at ParsedDocument and never imports RAG.
• quantmind.rag owns opinionated document RAG. It uses LlamaIndex for page-aware chunking and document-local retrieval without becoming a generic framework.
• LlamaIndex owns RAG mechanics. Use it for document/node transformations, chunking, vector/BM25 indexing, semantic retrieval, ranking, and supported caching.
• QuantMind owns durable domain truth. Canonical knowledge objects, source/version facts, time cutoffs, provenance, citations, SQLite storage, and public APIs remain QuantMind types and behavior.
• PageIndex remains an independent tree-retrieval path. It can later navigate a selected long document without vectors and may use separately rebuildable derived state; this issue must not force PageIndex through LlamaIndex ranking or define all future retrieval as vector search.
• OpenAI Agents SDK owns the agent runtime. This issue does not add a QuantMind agent framework or replace the existing runtime.

LlamaIndex types must remain private implementation details. They must not enter public operation signatures, canonical Pydantic knowledge models, ParsedDocumentHit, SemanticQuery, SemanticHit, or persisted canonical payloads. Integration-specific settings may expose a narrowly scoped pass-through configuration, but quantmind.rag must not create a public retriever, vector-store, provider, backend, query-engine, or plugin hierarchy.

🔧 Proposed Implementation

1. Record the design in contexts/

• Add contexts/design/preprocess/pdf.md to define the page-aware multimodal parsing contract and make preprocessing end explicitly at ParsedDocument.
• Add contexts/design/rag/document.md to define the opinionated LlamaIndex document RAG package and its non-framework boundary.
• Update contexts/design/flow/paper.md so PaperSourceDocument is built from the page-aware parsed result instead of flattened Markdown.
• Update contexts/design/library/local.md so LlamaIndex replaces the current private collection-wide semantic-ranking backend while SQLite remains the source of truth for canonical knowledge and rebuildable semantic records.
• Update contexts/design/README.md so the preprocessing and RAG designs are discoverable.
• State clearly that a future PageIndex operation may live under quantmind.rag as another opinionated document adapter. It can coexist with collection-wide LlamaIndex retrieval and does not have to be served through LocalKnowledgeLibrary.search().

2. Make the RAG dependencies required and bounded

• Add llama-index-core as a required dependency and add only the integrations used by the implementation, rather than the llama-index umbrella package.
• Use LiteParse as the default local PDF parser for this path.
• Pin a compatible LlamaIndex minor line in the lockfile and treat dependency upgrades as behavior changes that must pass the offline golden fixture and Transformer smoke test.
• Do not make the dependency optional and do not maintain a parallel custom RAG implementation as the default path.

3. Introduce a page-aware parsed document

Add an internal deterministic parsing value that preserves:

• exact source hash plus parser name and version;
• every physical page in order, including pages with no extracted text;
• 1-based PDF page numbers;
• text blocks with page ownership and bounding boxes when provided;
• page screenshots and/or stable image artifact references for later multimodal use;
• cleanup/version metadata needed to reproduce the derived text.

The parsed value is a QuantMind boundary object, not a LlamaIndex public type and not another canonical knowledge model. A compatibility helper may still produce Markdown when needed, but flattened Markdown must no longer be the primary paper preprocessing result.

4. Put opinionated LlamaIndex document RAG in quantmind.rag

• Add quantmind.rag.document as the single package owner for page-aware document chunking and retrieval; it may import quantmind.preprocess, while preprocessing and the canonical library must not import RAG.
• Convert ParsedDocument pages and blocks into private LlamaIndex documents/nodes without losing source ID, page number, block coordinates, or artifact references.
• Use LlamaIndex transformations such as SentenceSplitter; allow their supported parameters to pass through a narrow integration configuration rather than reimplementing chunking behavior.
• Provide one simple document-local retrieval path using LlamaIndex BM25 for the smoke case; collection-wide vector semantic ranking remains private to LocalKnowledgeLibrary.
• Convert retrieved nodes and scores back into ParsedDocumentHit evidence so callers receive matched text, source hash, page metadata, and artifact references rather than LlamaIndex objects.
• Retain page screenshots or image references with retrieved evidence so a later multimodal agent can inspect the relevant page. Image embeddings and answer synthesis are not required in this issue.

5. Replace the current LocalKnowledgeLibrary semantic-ranking backend

• Replace the private _internal/exact_cosine.py path currently used for LocalKnowledgeLibrary.search() semantic ranking with a LlamaIndex index/retriever.
• Keep SQLite as the canonical store for knowledge payloads and rebuildable semantic records. The LlamaIndex index is private derived state and must be rebuildable from SQLite.
• Preserve the existing LocalKnowledgeLibrary.open(), put(), get(), search(), SemanticQuery, and SemanticHit public contracts unless an explicitly documented compatibility change is required.
• Do not redefine LocalKnowledgeLibrary as a vector database or make LlamaIndex the only possible retrieval mechanism. Future PageIndex tree navigation may be exposed as a separate operation after document selection and may keep its derived tree state alongside the canonical paper.
• Preserve filtering semantics for item type, source kind, confidence, tags, tree ID, as_of_before, and available_at_before. Unknown availability must continue to be excluded when an availability cutoff is present, and filtering must occur before final ranking.
• Preserve canonical item/node resolution, source metadata, citations, idempotent put(), and reopen/rebuild behavior.
• Keep deterministic fake embedding and retriever seams for offline unit tests. Do not expose a public vector-store, embedder, retriever, or backend registry.

API Design

The public library path remains stable:

library = await LocalKnowledgeLibrary.open(
    ".quantmind/library.db",
    embedding_model="text-embedding-3-small",
)

await library.put(paper)
hits = await library.search(
    SemanticQuery(
        text="How does multi-head attention work?",
        item_types=["paper"],
        top_k=5,
    )
)

LlamaIndex documents, nodes, indexes, retrievers, and NodeWithScore values remain behind this API.

Configuration

Expose only configuration that is needed by the selected LlamaIndex integrations, with upstream-compatible names where practical. QuantMind-specific source, provenance, time, and persistence settings remain explicit QuantMind configuration. Do not add a general provider registry.

🎨 User Experience

A user can fetch or open a paper, parse it without losing page and visual context, apply LlamaIndex chunking/retrieval settings, store the resulting canonical knowledge, and receive typed ranked evidence that resolves back to the original paper and pages. The user does not need to work with LlamaIndex types unless they are extending an explicitly private integration boundary.

📊 Use Cases

1. Paper RAG: Retrieve the passages that explain multi-head attention from the Transformer paper and retain the source page for citation and optional screenshot inspection.
2. Local semantic library: Store canonical Paper and tree nodes in SQLite, then search them through LlamaIndex ranking while receiving the existing SemanticHit type.
3. Future tree construction and navigation: Reuse the same page-aware parsed document in a future quantmind.rag PageIndex operation, then navigate a selected document independently of the collection-wide LlamaIndex semantic-search path.

✅ Acceptance Criteria

Context and dependency contract

• The design index and the paper, PDF preprocessing, document RAG, and local library context pages describe the ownership boundaries above and distinguish implemented behavior from planned behavior.
• Import contracts enforce that quantmind.rag may depend on quantmind.preprocess, while preprocessing and quantmind.library do not depend on RAG.
• llama-index-core, LiteParse, and only the required LlamaIndex integrations are required project dependencies; the umbrella llama-index package is not added.
• LlamaIndex types do not appear in public QuantMind signatures, canonical persisted knowledge schemas, SemanticQuery, or SemanticHit.

PDF preprocessing and ingestion

• The existing synthetic golden PDF remains the deterministic offline fixture and verifies physical page count, page order, empty-page preservation, text anchors, section/page spans, and tree invariants.
• Parsed PDF output preserves 1-based page ownership, text blocks, available bounding boxes, and screenshot/image artifact references.
• Every LlamaIndex chunk retains enough metadata to resolve to its exact source version and original PDF page or pages.
• A focused example is added under examples/rag/; parser tests remain under tests/preprocess/, RAG tests live under tests/rag/, and tests inherit from unittest.TestCase or its async subclass.

Transformer vertical slice

• A bounded live smoke script uses the pinned 1706.03762v7 Transformer PDF and is registered in the existing E2E workflow and docs/README.md catalog.
• The live smoke verifies the 15-page source, creates page-aware chunks, and retrieves a relevant multi-head-attention passage with its page metadata without requiring answer synthesis.
• Live-network or upstream parser availability does not block the deterministic verification suite.

LocalKnowledgeLibrary migration

• The existing semantic-search path in LocalKnowledgeLibrary.search() ranks through a LlamaIndex retriever and no longer imports or executes the private NumPy exact-cosine ranker.
• The library design explicitly permits a separate future PageIndex tree-navigation operation and does not define the library as vector-only.
• Existing SQLite databases remain readable, or a migration and compatibility behavior are documented and tested.
• Existing query filters, financial-time behavior, typed hit fields, canonical node resolution, citation preservation, and deterministic ordering for equal scores remain covered by tests.
• Reopening the library rebuilds private retrieval state from SQLite without re-embedding unchanged records.
• Re-putting unchanged knowledge remains idempotent and changed retrieval text refreshes only the affected semantic records.
• A focused example under examples/library/ demonstrates the unchanged public semantic-search path.

Verification

• bash scripts/verify.sh passes without network access.
• The applicable bounded PDF/RAG smoke command passes separately.

🚫 Non-goals

• PageIndex tree construction or navigation tracked by feat: page-preserving TreeKnowledge build + agentic navigation #95; this issue only establishes the bounded quantmind.rag package where that future operation may live.
• Fixing the separate final-Paper structured-output failure tracked by paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91.
• Cloud LlamaParse, OCR for image-only scans, or authenticated/paywalled PDF acquisition.
• Image embeddings, multimodal reranking, final-answer generation, or chatbot UI.
• Replacing the OpenAI Agents SDK or adding a QuantMind agent runtime.
• Exposing a public LlamaIndex backend, vector-store, retriever, embedder, or plugin registry.
• Completing video and audio parsing from [Feautre] parser: MultiModal Parser Impl #59.

🔗 Related Issues

• Implements the PDF portion of [Feautre] parser: MultiModal Parser Impl #59; it does not close the remaining video/audio scope.
• Replaces the private ranking implementation delivered by feat(library): local semantic knowledge library MVP #111 while preserving its public library contract.
• Blocks the paper-to-semantic-search E2E in feat(example): paper extraction to semantic knowledge search #112.
• Provides the preprocessing and retrieval foundation used to evaluate PageIndex work in feat: page-preserving TreeKnowledge build + agentic navigation #95 later.
• Relates to the separate paper_flow() assembly bug in paper_flow() fails on its own README example: dict-typed nodes field breaks strict structured output, UUID id fields reject the model's natural output #91.

Implementation Considerations

Breaking Changes

• This feature would introduce breaking changes.
• This feature is intended to preserve the existing public API and stored canonical knowledge contract.

Dependencies

• Requires new mandatory dependencies.
• Uses existing dependencies only.

---

Checklist

• I have searched existing issues to avoid duplicates.
• I have provided a clear and detailed description.
• I have explained the motivation and use cases.
• I have considered the implementation approach.
• I have thought about potential breaking changes.