feat(rag): adopt LlamaIndex for document ingestion and semantic retrieval

.agents/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.agents/skills/quantmind-dev/references/develop-components.md

@@ -35,6 +35,7 @@ apply throughout.
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
+| `quantmind/rag/` | `preprocess` only |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -60,6 +61,16 @@ apply throughout.
 - Surface the common path at the package root (`from quantmind.preprocess
   import fetch_arxiv`), keep explicit submodule paths working.
 
+### `quantmind/rag/` — opinionated document RAG
+
+- Use LlamaIndex for chunking, indexing, retrieval, and ranking; add only the
+  source/page/provenance conversion that QuantMind owns.
+- Import deterministic inputs from `quantmind.preprocess`; preprocessing must
+  never import RAG.
+- Keep LlamaIndex types private. Return frozen QuantMind evidence values.
+- Do not add a public retriever, vector-store, provider, backend registry, or
+  generic query-engine hierarchy.
+
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
 - Public operations are `async def` functions, not classes; state passes

.claude/skills/quantmind-dev/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: quantmind-dev
-description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
+description: Contributor workflow for the QuantMind codebase. Covers commit format, pull request format, and component development across quantmind/ modules (knowledge, configs, preprocess, rag, flows, mind, utils) with tests, examples, and verification. Use when committing, opening a PR, or implementing/refactoring QuantMind code.
 ---
 
 # QuantMind Dev

.claude/skills/quantmind-dev/references/develop-components.md

@@ -35,6 +35,7 @@ apply throughout.
 | `quantmind/knowledge/` | nothing (leaf) |
 | `quantmind/configs/` | `knowledge` only |
 | `quantmind/preprocess/` | `utils` only |
+| `quantmind/rag/` | `preprocess` only |
 | `quantmind/flows/`, `quantmind/magic.py` | apex — may import all of the above |
 
 ### `quantmind/knowledge/` — data standard
@@ -60,6 +61,16 @@ apply throughout.
 - Surface the common path at the package root (`from quantmind.preprocess
   import fetch_arxiv`), keep explicit submodule paths working.
 
+### `quantmind/rag/` — opinionated document RAG
+
+- Use LlamaIndex for chunking, indexing, retrieval, and ranking; add only the
+  source/page/provenance conversion that QuantMind owns.
+- Import deterministic inputs from `quantmind.preprocess`; preprocessing must
+  never import RAG.
+- Keep LlamaIndex types private. Return frozen QuantMind evidence values.
+- Do not add a public retriever, vector-store, provider, backend registry, or
+  generic query-engine hierarchy.
+
 ### `quantmind/flows/` and `quantmind/magic.py` — apex layer
 
 - Public operations are `async def` functions, not classes; state passes

.github/workflows/e2e.yml

@@ -16,19 +16,61 @@ on:
       - 'quantmind/preprocess/fetch/_types.py'
       - 'quantmind/preprocess/fetch/http.py'
       - 'quantmind/preprocess/fetch/rss.py'
+      - 'quantmind/preprocess/fetch/arxiv.py'
+      - 'quantmind/preprocess/format/pdf.py'
+      - 'quantmind/rag/**'
+      - 'scripts/verify_pdf_rag_e2e.py'
+      - 'pyproject.toml'
   schedule:
     - cron: "17 3 * * *"
   workflow_dispatch:
 
 permissions:
   contents: read
+  pull-requests: read
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
+  changes:
+    runs-on: ubuntu-latest
+    outputs:
+      news: ${{ steps.filter.outputs.news }}
+      pdf_rag: ${{ steps.filter.outputs.pdf_rag }}
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Detect component changes
+        id: filter
+        uses: dorny/paths-filter@v3
+        with:
+          filters: |
+            news:
+              - '.github/workflows/e2e.yml'
+              - 'scripts/verify_news_e2e.py'
+              - 'quantmind/preprocess/_news_types.py'
+              - 'quantmind/preprocess/clean.py'
+              - 'quantmind/preprocess/news.py'
+              - 'quantmind/preprocess/pr_newswire.py'
+              - 'quantmind/preprocess/time.py'
+              - 'quantmind/preprocess/format/html.py'
+              - 'quantmind/preprocess/fetch/**'
+              - 'pyproject.toml'
+            pdf_rag:
+              - '.github/workflows/e2e.yml'
+              - 'scripts/verify_pdf_rag_e2e.py'
+              - 'quantmind/preprocess/fetch/arxiv.py'
+              - 'quantmind/preprocess/format/pdf.py'
+              - 'quantmind/rag/**'
+              - 'pyproject.toml'
+
   news:
+    needs: changes
+    if: github.event_name != 'pull_request' || needs.changes.outputs.news == 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 10
 
@@ -55,3 +97,33 @@ jobs:
 
       - name: Run live news E2E
         run: .venv/bin/python scripts/verify_news_e2e.py
+
+  pdf-rag:
+    needs: changes
+    if: github.event_name != 'pull_request' || needs.changes.outputs.pdf_rag == 'true'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v3
+        with:
+          enable-cache: true
+          cache-dependency-glob: pyproject.toml
+
+      - name: Create virtual environment
+        run: uv venv
+
+      - name: Install project runtime dependencies
+        run: uv pip install --python .venv/bin/python -e .
+
+      - name: Run live PDF RAG E2E
+        run: .venv/bin/python scripts/verify_pdf_rag_e2e.py

AGENTS.md

@@ -36,6 +36,7 @@ handoff all come from `openai-agents`.
 | `quantmind/library/` | Local persistence and semantic retrieval for canonical knowledge — depends only on `knowledge` |
 | `quantmind/configs/` | Operation cfg + typed input models or unions (`BaseFlowCfg`, `NewsWindow`, `PaperInput`) — depends only on `knowledge` |
 | `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` |
+| `quantmind/rag/` | Opinionated LlamaIndex document chunking and retrieval — depends only on `preprocess` |
 | `quantmind/flows/` | Apex layer: public library operations (`paper_flow`, `collect_news`, `batch_run`) |
 | `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` |
 | `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) |
@@ -75,19 +76,22 @@ the user explicitly authorizes it — fix the underlying issue instead.
 
 1. **Library, not framework** — functions over classes, `Protocol` over ABC,
    no plugin registries, no hook discovery, no CLI.
-2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
+2. **RAG data plane, not framework** — use LlamaIndex directly inside
+   `quantmind.rag`; keep upstream types private and do not add retriever,
+   vector-store, provider, or backend registries.
+3. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
    QuantMind-side facades over `from agents import ...`.
-3. **Schema models vs runtime evidence** — user/LLM inputs and configs use
+4. **Schema models vs runtime evidence** — user/LLM inputs and configs use
    extra-forbid Pydantic models; knowledge adds `frozen=True`; deterministic
    fetch, preprocessing, and collection values use frozen dataclasses when
    they do not need validation or JSON Schema (`Fetched`, `NewsBatch`).
-4. **Import boundaries are contracts** — `import-linter` (configured in
+5. **Import boundaries are contracts** — `import-linter` (configured in
    `pyproject.toml`) pins the dependency graph; never work around a failing
    contract.
-5. **Absolute imports** across module boundaries.
-6. **No meaningless wrappers** — a method must add logic, abstraction, or a
+6. **Absolute imports** across module boundaries.
+7. **No meaningless wrappers** — a method must add logic, abstraction, or a
    side effect beyond the call it wraps; otherwise inline it.
-7. **Name public operations by intent** — follow
+8. **Name public operations by intent** — follow
    `contexts/design/operations/naming.md`; use stage verbs, and reserve
    `pipeline` for deliberate multi-stage composition.
 

CLAUDE.md

@@ -37,6 +37,7 @@ handoff all come from `openai-agents`.
 | `quantmind/library/` | Local persistence and semantic retrieval for canonical knowledge — depends only on `knowledge` |
 | `quantmind/configs/` | Operation cfg + typed input models or unions (`BaseFlowCfg`, `NewsWindow`, `PaperInput`) — depends only on `knowledge` |
 | `quantmind/preprocess/` | Deterministic fetch / format / clean / time utilities — depends only on `utils` |
+| `quantmind/rag/` | Opinionated LlamaIndex document chunking and retrieval — depends only on `preprocess` |
 | `quantmind/flows/` | Apex layer: public library operations (`paper_flow`, `collect_news`, `batch_run`) |
 | `quantmind/magic.py` | `resolve_magic_input`: natural language → `(input, cfg)` |
 | `quantmind/mind/` | Cognitive layer (memory protocol); landing via the Agents SDK migration (#71) |
@@ -76,19 +77,22 @@ the user explicitly authorizes it — fix the underlying issue instead.
 
 1. **Library, not framework** — functions over classes, `Protocol` over ABC,
    no plugin registries, no hook discovery, no CLI.
-2. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
+2. **RAG data plane, not framework** — use LlamaIndex directly inside
+   `quantmind.rag`; keep upstream types private and do not add retriever,
+   vector-store, provider, or backend registries.
+3. **Do not rebuild the agent runtime** — use `openai-agents` directly; no
    QuantMind-side facades over `from agents import ...`.
-3. **Schema models vs runtime evidence** — user/LLM inputs and configs use
+4. **Schema models vs runtime evidence** — user/LLM inputs and configs use
    extra-forbid Pydantic models; knowledge adds `frozen=True`; deterministic
    fetch, preprocessing, and collection values use frozen dataclasses when
    they do not need validation or JSON Schema (`Fetched`, `NewsBatch`).
-4. **Import boundaries are contracts** — `import-linter` (configured in
+5. **Import boundaries are contracts** — `import-linter` (configured in
    `pyproject.toml`) pins the dependency graph; never work around a failing
    contract.
-5. **Absolute imports** across module boundaries.
-6. **No meaningless wrappers** — a method must add logic, abstraction, or a
+6. **Absolute imports** across module boundaries.
+7. **No meaningless wrappers** — a method must add logic, abstraction, or a
    side effect beyond the call it wraps; otherwise inline it.
-7. **Name public operations by intent** — follow
+8. **Name public operations by intent** — follow
    `contexts/design/operations/naming.md`; use stage verbs, and reserve
    `pipeline` for deliberate multi-stage composition.
 

contexts/design/README.md

@@ -26,6 +26,8 @@ implementation must preserve.
 |---|---|
 | Flow | [Paper extraction from input to validated result](flow/paper.md) |
 | Flow | [News collection](flow/news.md) |
+| Preprocess | [Page-aware multimodal PDF parsing](preprocess/pdf.md) |
+| RAG | [Page-aware document chunking and retrieval](rag/document.md) |
 | Library | [Local knowledge storage and meaning-based search](library/local.md) |
 | Operations | [Public operation naming](operations/naming.md) |
 

contexts/design/flow/paper.md

@@ -4,7 +4,7 @@
 
 - **Purpose**: Define how a paper input becomes a validated `Paper`.
 - **Read when**: Changing paper inputs, parsing, section trees, source tracking, page ranges, or future PageIndex support.
-- **Status**: Planned design; [Current Gaps](#current-gaps) lists what is not implemented yet.
+- **Status**: Mixed. Page-aware PDF parsing and document RAG are implemented; [Current Gaps](#current-gaps) lists the remaining paper assembly work.
 - **Core rule**: A model or PageIndex may suggest a section tree. Code creates the final IDs, links, order, page ranges, citations, and source-backed text.
 - **Page numbering**: PDF page ranges start at 1 and include both the first and last page.
 
@@ -57,6 +57,7 @@ across them, or write answers.
 | Work | Owner |
 |---|---|
 | Resolve identifiers, fetch bytes, parse pages, and hash source content | `quantmind.preprocess` |
+| Chunk or retrieve page-aware document evidence when requested | `quantmind.rag` |
 | Configure the operation and select the input variant | `quantmind.configs` |
 | Suggest a section tree and build the final `Paper` | `quantmind.flows` |
 | Define the `Paper`, `TreeKnowledge`, `TreeNode`, source, citation, and extraction models | `quantmind.knowledge` |
@@ -139,9 +140,12 @@ Required properties:
 - A page-based tree builder, including a future PageIndex integration, accepts
   only a source document whose range unit is `pdf_page`.
 
-`PaperSourceDocument` is a temporary value used during extraction, not another
-public knowledge model. The caller keeps raw PDF or HTML files; this design
-does not embed them inside `Paper`.
+For PDF inputs, [`ParsedDocument`](../preprocess/pdf.md) is the implemented
+deterministic source value. It preserves blocks, coordinates, and optional page
+screenshots as well as page text. `PaperSourceDocument` is the flow's temporary
+view over that value, not another public knowledge model. The caller keeps raw
+PDF, HTML, and screenshot files; this design does not embed them inside
+`Paper`.
 
 ## Which Source Provides Each Field
 
@@ -306,6 +310,10 @@ Future integration must preserve these decisions:
 6. Sibling page ranges may overlap, and a child range does not need to fit
    completely inside its parent range.
 
+A PageIndex adapter belongs with other opinionated document retrieval in
+[`quantmind.rag`](../rag/document.md). It still returns the limited draft above;
+it does not become the canonical tree or a generic retrieval backend.
+
 ## Fixed Paper Test Data
 
 The fixed test files live at:
@@ -331,8 +339,10 @@ rather than create a competing test paper.
 
 The repository does not yet guarantee the target pipeline above:
 
-- `pdf_to_markdown()` concatenates non-empty page text and drops page
-  boundaries and empty pages.
+- `pdf_to_markdown()` remains a compatibility view, while the primary
+  `parse_pdf()` path now preserves pages, blocks, coordinates, and artifacts.
+- `paper_flow()` has not yet adopted `ParsedDocument` or the document RAG
+  boundary; it still consumes the compatibility Markdown view.
 - `paper_flow()` sends the flattened document to one extraction agent and asks
   it to return the final `Paper` directly.
 - The model currently controls IDs, edges, citations, source fields, and

contexts/design/library/local.md

@@ -16,7 +16,8 @@
 - [What Can Match a Query](#what-can-match-a-query)
 - [When to Rebuild Search Data](#when-to-rebuild-search-data)
 - [Time Fields and Look-Ahead](#time-fields-and-look-ahead)
-- [Why SQLite and Simple Exact Ranking](#why-sqlite-and-simple-exact-ranking)
+- [Why SQLite and LlamaIndex Ranking](#why-sqlite-and-llamaindex-ranking)
+- [Independent tree navigation](#independent-tree-navigation)
 - [Out of Scope](#out-of-scope)
 
 ## Key Decisions
@@ -41,6 +42,7 @@ implementation proves which behavior is truly shared.
 |---|---|
 | `quantmind.knowledge` | Define immutable knowledge models and the text used for embeddings; perform no I/O |
 | `quantmind.library` | Store validated knowledge, maintain rebuildable search records, and return `SemanticHit` results |
+| [`quantmind.rag`](../rag/document.md) | Chunk and retrieve evidence within one parsed document without storing canonical knowledge |
 | `quantmind.flows` | Produce validated knowledge and optionally pass it to a library |
 | `quantmind.mind` or an agent application | Search the library and use matches to write answers |
 | Caller or source-specific pipeline | Retain raw PDF, HTML, media, and operational files |
@@ -104,16 +106,30 @@ or have no known availability time. Filtering only by `as_of_before` can still
 leak future information. Apply source kind, item type, confidence, tags, tree
 ID, and both time cutoffs before ranking results.
 
-## Why SQLite and Simple Exact Ranking
+## Why SQLite and LlamaIndex Ranking
 
 SQLite provides transactions, foreign keys, and reliable reconstruction of
-typed knowledge. NumPy can compare every embedding with cosine similarity at
-the current local data size. This simple exact ranking can later be replaced
-without changing user code.
-
-A future approximate or remote index may replace the private search
-implementation. It does not replace stored knowledge and must return the same
-`SemanticHit` type regardless of provider.
+typed knowledge. LlamaIndex owns the private collection-wide vector retrieval
+and ranking mechanics. On the first search after open or a write, private
+retrieval state is rebuilt from the filtered semantic records stored in SQLite;
+unchanged records reuse their persisted embeddings and are not sent to the
+embedding provider again.
+
+LlamaIndex nodes and retrievers remain implementation details. They do not
+enter `SemanticQuery`, `SemanticHit`, canonical Pydantic payloads, or public
+signatures. A future approximate or remote index may replace the private
+search implementation without replacing stored knowledge or changing the
+public result type.
+
+## Independent tree navigation
+
+`LocalKnowledgeLibrary` is canonical knowledge storage with rebuildable
+retrieval capabilities; it is not defined as a vector database. A future
+PageIndex path can select a paper through collection-wide semantic retrieval,
+then navigate that selected document's tree through a separate operation and
+separately rebuildable state. PageIndex does not have to be served through
+`search()` or LlamaIndex ranking. Opinionated document retrieval, including a
+future PageIndex adapter, belongs under [`quantmind.rag`](../rag/document.md).
 
 ## Out of Scope