feat(search): hybrid retrieval + FTS/vector indexes + grounded cited answer (folds #934) - #938
Conversation
…ts the scalability plan) /api/search previously computed ts_rank_cd minus cosine_distance over every owner-scoped row on every query — a full sequential scan that no index could serve (verified by EXPLAIN: the planner ignored GIN/HNSW even when present) and an unnormalized fusion of incomparable scales. Reshape (docs/plans/2026-07-05-search-scalability-hybrid-retrieval.md): - Candidate generation is now up to four index-eligible arms: lexical email/ attachment arms gated by @@ (GIN-served, ts_rank_cd ordered, LIMIT) and vector email/attachment arms as pure cosine-distance ORDER BY + LIMIT (HNSW-served). The embedding-failure fallback keeps working (lexical arms only). - Fusion moves to the app as reciprocal-rank fusion (RRF, k=60), replacing the score subtraction. Metadata + reply counts are fetched in a second bounded query for the fused candidate ids only, so joins never disqualify the ANN index. - Owner (user/organization) scoping preserved on every arm and the metadata query. Migration 0010 (postgresql-guarded, CONCURRENTLY in an autocommit block): GIN expression indexes on email/attachment text and hnsw (ivfflat below pgvector 0.5) vector_cosine_ops indexes, with the DDL shared between the migration and tests via db/search_indexes.py. Verification: - tests/test_search_pg_indexes.py (postgres marker) ran against a real pgvector 16 instance: EXPLAIN shows the lexical arm using ix_email_records_body_fts and the vector arm using ix_email_records_embedding_vec. - tests/test_search.py rewritten for the new shape (16 passed): endpoint behavior, owner scoping, fallback, session split, RRF fusion, arm shapes. - Regressions green: alembic/bootstrap (37), emails_api/db_session (66), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
Retrieval existed but nothing generated from it: search returned rows, never answers. This adds /api/search/answer — question answering over the user's own emails with the property that an answer cannot reference material retrieval did not surface. - services/rag_service.py: the model sees only retrieved, owner-scoped email content passed as JSON data (prompt-injection hardening, mirrors api/llm.py), must return cited_email_ids, and citations are validated against the retrieved set (unretrieved ids dropped + logged). Empty retrieval makes no LLM call at all. Context bounded to 5 emails x 1500 chars. - /api/search/answer reuses the hybrid retrieval arms + RRF fusion from the search reshape (owner scoping preserved), fetches bounded metadata for the fused top ids, and returns answer + citations (subject/sender/snippet) + provenance. Tests (6): no-context -> no LLM call; unretrieved citation ids dropped; context bounded to MAX_CONTEXT_EMAILS; long content truncated; endpoint happy path returns answer+citations from retrieved-only context; empty query short-circuits. Search suite still green (16), ruff clean. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_017RkKdtHRLG4wSLh6PVsp8J
seonghobae
changed the title
Resolve develop conflicts for #938, restore Windows runtime/env path stability, keep email import logging explicit, and make frontend build/smoke checks reproducible.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
PR governance metadata gate is not ready for
|
OpenCode Review Overview
Pull request overviewOpenCode reviewed the current-head bounded evidence and found no blocking issues. FindingsNo blocking findings. SummaryApproval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Changed-File Evidence Mapflowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Backend (6 files)"]
S1 --> I1["API and service runtime"]
I1 --> R1["Review risk: Backend (6 files)"]
R1 --> V1["backend tests"]
Evidence --> S2["Frontend (5 files)"]
S2 --> I2["browser runtime and bundle"]
I2 --> R2["Review risk: Frontend (5 files)"]
R2 --> V2["frontend tests"]
|
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including backend/api/search.py, backend/core/config.py, backend/core/env_paths.py, backend/scripts/private_mail_http_smoke.py, backend/scripts/start_backend.py, and 10 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects backend/api/search.py to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: PR introduces a well-implemented hybrid retrieval and grounded answer feature with comprehensive tests and no executable gaps.
- Head SHA:
7ce346a1ae41b2c866699918a097fbb1fb85d2b6 - Workflow run: 29155084621
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file: .gitattributes"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file: .gitattributes"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (10 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (10 files)"]
R2 --> V2["backend tests"]
Evidence --> S3["Frontend (5 files)"]
S3 --> I3["browser runtime and bundle"]
I3 --> R3["Review risk: Frontend (5 files)"]
R3 --> V3["frontend tests"]
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including backend/api/search.py, backend/core/config.py, backend/core/env_paths.py, backend/scripts/private_mail_http_smoke.py, backend/scripts/start_backend.py, and 10 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects backend/api/search.py to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: No blocking issues found in the current-head evidence.
- Head SHA:
7ce346a1ae41b2c866699918a097fbb1fb85d2b6 - Workflow run: 29154564739
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Changed file: .gitattributes"]
S1 --> I1["repository behavior"]
I1 --> R1["Review risk: Changed file: .gitattributes"]
R1 --> V1["required checks"]
Evidence --> S2["Backend (10 files)"]
S2 --> I2["API and service runtime"]
I2 --> R2["Review risk: Backend (10 files)"]
R2 --> V2["backend tests"]
Evidence --> S3["Frontend (5 files)"]
S3 --> I3["browser runtime and bundle"]
I3 --> R3["Review risk: Frontend (5 files)"]
R3 --> V3["frontend tests"]
|
Updated current-head evidence snapshot:
|
There was a problem hiding this comment.
Pull request overview
OpenCode reviewed the current-head bounded evidence and found no blocking issues.
Findings
No blocking findings.
Summary
Approval sufficiency: bounded evidence supplied affirmative approval evidence for changed files, coverage/docstring posture, risk surfaces, and current-head verification; approval is not based merely on the absence of known blockers.
Verification posture: CodeGraph evidence was initialized and bounded current-head evidence reviewed for changed-file evidence including backend/api/search.py, backend/core/config.py, backend/scripts/start_backend.py, backend/services/email_import_service.py, backend/services/rag_service.py, and 6 more.
Linter/static: workflow/static review evidence is bounded by the current-head GitHub Checks gate and changed-file evidence.
TDD/regression: coverage execution evidence and focused changed hunks were reviewed from bounded-review-evidence.md.
Coverage: coverage execution evidence reports supported repository test suites passed.
Docstring coverage: coverage execution evidence reports configured repository docstring gates passed or docstring coverage was advisory.
DAG: CodeGraph/source-backed behavior map connects backend/api/search.py to the affected review, runtime, or workflow path and required checks.
PoC/execution: coverage-evidence job executed on the current head and reported PASS.
DDD/domain: workflow and repository-governance invariants were reviewed against changed files in bounded evidence.
CDD/context: CodeGraph evidence, changed-file history, and focused hunks were reviewed from bounded-review-evidence.md.
Similar issues: changed-file history evidence was reviewed for comparable local precedents.
Claim/concept check: bounded evidence, repository source, current-head workflow evidence, and, where numeric, scientific, statistical, or literature-backed claims are affected, original-paper/formula evidence and parameter-recovery expectations were used for claims.
Standards search: standards and external-source checks are delegated to configured OpenCode web_search/Context7/DeepWiki sources when applicable; no evidence-backed standards blocker is present in bounded evidence.
Compatibility/convention: changed workflow/script conventions, object naming, and reserved-word safety for schema/API/config/code surfaces were checked in bounded evidence.
Breaking-change/backcompat: deployment evidence and changed-file history were checked for backward-compatibility risk.
Performance: changed surfaces were checked for performance risk in bounded evidence.
Developer experience: changed automation, review, test, setup, and maintenance surfaces were checked for helpful or obstructive DX impact in bounded evidence.
User experience: connected user, operator, API, CLI, documentation, review-comment, status-check, rendering, and workflow-reader behavior was checked for contradictions against code, docs, and tests in bounded evidence.
Visual/DOM: Playwright visual, DOM locator, ARIA snapshot, console, and responsive evidence were checked when a web UI surface was present; for non-web surfaces, API/CLI/log/docs/workflow interaction evidence was reviewed instead.
Accessibility/i18n: accessibility, localization, and human-readable text surfaces were checked where UI, CLI, API message, docs, logs, or review text changed.
Supply-chain/license: dependency, package, model, container, and external-tool changes were checked in bounded evidence.
Packaging: package, build, test, lint, and security contracts were checked in bounded evidence.
Security/privacy: workflow-token, review-gate, and repository-automation security/privacy boundaries were checked in bounded evidence.
- Result: APPROVE
- Reason: No blocking issues found in the current-head evidence.
- Head SHA:
7970aba308d6dd3862dd95a456136e0d267b71c6 - Workflow run: 29167260645
- Workflow attempt: 1
Changed-File Evidence Map
flowchart LR
PR["PR changed files"] --> Evidence["OpenCode bounded evidence"]
Evidence --> S1["Backend (6 files)"]
S1 --> I1["API and service runtime"]
I1 --> R1["Review risk: Backend (6 files)"]
R1 --> V1["backend tests"]
Evidence --> S2["Frontend (5 files)"]
S2 --> I2["browser runtime and bundle"]
I2 --> R2["Review risk: Frontend (5 files)"]
R2 --> V2["frontend tests"]
1 participant
What (the moat demo: cited answers over your own mailbox)
Retrieval existed but nothing generated from it.
/api/search/answeranswers a question from the user's own emails with a hard property: the answer cannot reference material retrieval did not surface.services/rag_service.py: model sees only retrieved, owner-scoped content as JSON data (injection-hardened, mirrorsapi/llm.py); must returncited_email_ids; citations validated against the retrieved set (unretrieved ids dropped + logged); empty retrieval → no LLM call at all; context bounded (5 emails × 1500 chars).answer + citations(subject/sender/snippet) + provenance.Verification (local)
6 new tests (no-context short-circuit · unretrieved-citation drop · context bounds · truncation · endpoint happy path with retrieved-only context · empty-query) + search suite 16 green + ruff clean.
Stack
#934 (hybrid retrieval) → this. Completes the moat chain: substrate (#927) → grounded extraction (#931) → scalable retrieval (#934) → cited generation.
🤖 Generated with Claude Code