feat: research public sources for post units

.env.example

@@ -39,6 +39,8 @@ MCP_RATE_LIMIT_WINDOW_SECONDS=
 # running contextual-orchestrator to turn the channels on.
 ORCHESTRATOR_BASE_URL=
 ORCHESTRATOR_API_KEY=
+SOURCE_RESEARCH_MAXIMUM_LEADS=
+SOURCE_RESEARCH_MAXIMUM_RESULTS=
 
 # GitHub workflows inject the canonical provider names from masked secrets.
 # Non-GitHub Compose runs also accept the operator's ~/.env compatibility

AGENTS.md

@@ -196,8 +196,10 @@ contextual-orchestrator owns model discovery and selection.
 
 `NullEmbeddingClient`, `NullAdjudicationClient`,
 `NullKeymanExtractionClient`, `NullEntityRelationshipClient`,
-`NullPostSummaryClient`, `NullPostChatClient`, and
-`NullCommitmentExtractionClient` (and any new channel client you add)
+`NullPostSummaryClient`, `NullPostChatClient`,
+`NullCommitmentExtractionClient`, `NullRelationVerificationClient`,
+`NullClaimVerificationClient`, and `NullSourceResearchClient`
+(and any new channel client you add)
 must set `available = False` and make their channel dropped +
 renormalized (`reconstruct.active_weights`), never silently return a
 placeholder score, invented Keyman, guessed relationship, fabricated
@@ -210,6 +212,14 @@ adjudication does -- never a raw LLM API. Demo TEPP seed goes through
 envelope is Failed (`tepp_not_available` / `tepp_result_not_persisted`),
 never a fabricated theta or a local psychometric substitute.
 
+Public source-reference research (ADR 0248) is a post-scoped write action
+on existing semantic units or image regions. Only `visibility_code=public`
+posts may send lead text to SearXNG or retrieve a result URL. Private posts
+fail closed without egress. Redirects and non-global targets are rejected.
+Unavailable search, retrieval, or adjudication is `research_unavailable`,
+never a fabricated supported/refuted judgment. Global Ask public
+verification (ADR 0215) still never fetches result URLs.
+
 The lineage `text` channel follows [ADR 0190](docs/adr/0190-lineage-text-channel-embedding-swap.md):
 when an embedding provider is configured, `reconstruct()` precomputes
 batched label embeddings once per reconstruction and scores cosine

ARCHITECTURE.md

@@ -826,6 +826,19 @@ against a deliberately fabricated one in the same request, asserting
 the former comes back `verify_corroborated` with a real evidence URL
 and the latter `verify_uncorroborated` with none.
 
+## Phase 6e: post-scoped source-reference research
+
+Issue #611's remaining ADR 0133 criterion is a different workflow from
+relation verification and from Global Ask snippet verification (ADR 0215).
+A public post may send an existing semantic unit or image-region excerpt
+to self-hosted SearXNG, retrieve one cited public page under SSRF and
+redirect rejection, and ask contextual-orchestrator to judge in
+`mode="verify"`. Private posts fail closed without egress. Citations
+persist to `source_research_citation` (migration 0236, ADR 0248). The
+reader next action is to open the cited public resource and compare it
+with the highlighted passage or image detail. Global Ask still never
+fetches result URLs.
+
 ## Phase 7: R&R's named actor is a PROV-O Agent, not always a person
 
 `post_summary.py`'s R&R extraction forced every named actor into a

CHANGELOG.d/2.19.0-post-scoped-source-reference-research.md

@@ -0,0 +1,17 @@
+# 2.19.0 — Post-scoped source-reference research
+
+## Added
+
+- Public posts can research a highlighted passage or image detail against a
+  cited public page (ADR 0248, remaining ADR 0133 / issue #611). The workflow
+  reuses self-hosted SearXNG, retrieves one public HTTP(S) target with
+  redirects disabled and non-global addresses rejected, and judges through
+  contextual-orchestrator `mode=verify`. Private posts fail closed without
+  egress. Deployments must set both source-research resource budgets explicitly;
+  otherwise the channel remains unavailable. Citations persist in 3NF
+  `source_research_citation`.
+- Reader next action: open the cited public resource, then compare it with
+  the highlighted passage or image detail. Supported or refuted judgments
+  without a cited URL downgrade to not enough information. Missing search,
+  retrieval, or adjudication is `research_unavailable`, never a fabricated
+  score.

CHANGELOG.md

@@ -51,6 +51,13 @@ All notable changes to this project are documented here. Format follows
 
 ### Added
 
+- Public posts can research a highlighted passage or image detail against a
+  cited public page (ADR 0248 / remaining ADR 0133). SearXNG finds candidates;
+  retrieval refuses redirects and non-global targets; contextual-orchestrator
+  judges in `mode=verify`. Private posts fail closed without sending content,
+  and absent explicit source-research resource budgets keep the channel unavailable.
+  After seed, open a public post and choose **Research public sources**, then
+  open the cited public resource and compare it with that highlighted content.
 - Expanded Voice-of-X post taxonomy (ADR 0246): the governed `voc_type`
   scheme adds Voice of Supplier, Employee, Business, Regulator, Investor,
   Society, and Process as source-post categories. Ontology, SHACL, the

CLAUDE.md

@@ -50,7 +50,7 @@ Create/start endpoint rules (ADR 0017 / 0021), tie-vs-miss similarity
 (ADR 0026), R&R catalog ids (ADR 0019 / 0027), leftover pairs
 (ADR 0048–0164 / 0182 / 0201), the text-channel embedding swap and cosine
 clamp (ADR 0190), per-edge channel-score persistence (ADR 0195),
-migration replay (ADR 0166), docstring coverage, and the measurement
-boundary are all stated in [AGENTS.md](AGENTS.md) -- read it before
-changing code, tests, or runtime policy rather than restating anything
-here.
+migration replay (ADR 0166), docstring coverage, source-reference
+research (ADR 0248), and the measurement boundary are all stated in
+[AGENTS.md](AGENTS.md) -- read it before changing code, tests, or runtime
+policy rather than restating anything here.

backend/app/config.py

@@ -58,6 +58,8 @@ class Settings:
     orchestrator_answer_timeout_seconds: float
     valkey_url: str
     searxng_base_url: str
+    source_research_maximum_leads: int | None
+    source_research_maximum_results: int | None
     tepp_transport_url: str
     tepp_api_key: str
     caldav_base_url: str
@@ -205,6 +207,12 @@ def load_settings() -> Settings:
         ),
         valkey_url=os.environ.get("VALKEY_URL", "redis://localhost:16379/0"),
         searxng_base_url=os.environ.get("SEARXNG_BASE_URL", ""),
+        source_research_maximum_leads=_optional_positive_int(
+            "SOURCE_RESEARCH_MAXIMUM_LEADS"
+        ),
+        source_research_maximum_results=_optional_positive_int(
+            "SOURCE_RESEARCH_MAXIMUM_RESULTS"
+        ),
         tepp_transport_url=os.environ.get("TEPP_TRANSPORT_URL", ""),
         tepp_api_key=os.environ.get("TEPP_API_KEY", ""),
         caldav_base_url=os.environ.get("CALDAV_BASE_URL", "").strip(),

backend/app/global_ask_queue.py

@@ -64,6 +64,7 @@
     gather_global_chat_sources,
     prepare_global_question_embedding,
 )
+from .source_research_ingestion import list_ask_source_references
 
 GLOBAL_ASK_STREAM_KEY = "global_ask_request_stream"
 
@@ -438,13 +439,18 @@ def can_see(row: asyncpg.Record) -> bool:
         verify_external=verify_external,
         client=verification_client,
     )
-    if knowledge_cutoff is None:
-        async with pool.acquire() as conn:
+    async with pool.acquire() as conn:
+        if knowledge_cutoff is None:
             lineage_graph = await lineage_graphs_for_posts(conn, can_see, cited_ids)
             images = await cited_post_images(conn, cited_ids)
-    else:
-        lineage_graph = {"nodes": [], "edges": [], "truncated": False}
-        images = []
+        else:
+            lineage_graph = {"nodes": [], "edges": [], "truncated": False}
+            images = []
+        source_references = await list_ask_source_references(
+            conn,
+            cited_ids,
+            checked_by=knowledge_cutoff,
+        )
     cited_posts = cited_post_summaries(usable_sources, cited_ids)
     cited_events = cited_post_events(usable_sources, cited_ids)
     cited_evidence = cited_post_evidence(usable_sources, cited_ids)
@@ -462,9 +468,15 @@ def can_see(row: asyncpg.Record) -> bool:
         "cited_events": cited_events,
         "cited_post_evidence": cited_evidence,
         "cited_post_images": images,
+        "cited_source_references": source_references,
         "source_post_ids": [source.post_id for source in sources],
         "lineage_graph": lineage_graph,
-        "delivery": build_ask_delivery(answer.answer_text, cited_posts, cited_evidence),
+        "delivery": build_ask_delivery(
+            answer.answer_text,
+            cited_posts,
+            cited_evidence,
+            source_references,
+        ),
         "external_verification_status": verification_status,
         "external_claims": [claim.to_payload() for claim in external_claims],
         "next_action": next_action,

backend/app/main.py

@@ -152,6 +152,10 @@
 )
 from backend.app.ranking_ingestion import load_visible_ranking_posts
 from backend.app.relation_verification_ingestion import verify_post_relations_from_pool
+from backend.app.source_research_ingestion import (
+    list_source_research_citations,
+    research_post_sources_from_pool,
+)
 from backend.app.report_ingestion import (
     GROUPING_KINDS,
     fetch_period_comparison,
@@ -239,6 +243,12 @@
     NullRelationVerificationClient,
     SearxngRelationVerificationClient,
 )
+from lineageweave.source_reference_research import (
+    PRIVATE_POST_UNAVAILABLE,
+    VISIBILITY_PUBLIC,
+    NullSourceResearchClient,
+    SearxngOrchestratedSourceResearchClient,
+)
 from lineageweave.semantic_hints import customer_hint_trust, format_semantic_hints
 from lineageweave.semantic_query import (
     ContextualOrchestratorSemanticQueryClient,
@@ -349,6 +359,27 @@ def _claim_verification_client_factory():
     return _claim_verification_client()
 
 
+def _source_research_client():
+    """Return the post-scoped public-research client, or its unavailable null."""
+
+    settings = load_settings()
+    if not (
+        settings.searxng_base_url
+        and settings.orchestrator_base_url
+        and settings.orchestrator_api_key
+        and settings.source_research_maximum_leads is not None
+        and settings.source_research_maximum_results is not None
+    ):
+        return NullSourceResearchClient()
+    return SearxngOrchestratedSourceResearchClient(
+        settings.searxng_base_url,
+        settings.orchestrator_base_url,
+        settings.orchestrator_api_key,
+        maximum_leads=settings.source_research_maximum_leads,
+        maximum_results=settings.source_research_maximum_results,
+    )
+
+
 def _organization_name_resolution_client():
     """Live orchestrator client when configured; otherwise the unavailable null."""
     settings = load_settings()
@@ -2459,6 +2490,112 @@ async def verify_post_entity_relationships(
     }
 
 
+@app.get("/api/posts/{post_id}/research-citations")
+async def read_post_research_citations(
+    post_id: str,
+    account: CurrentAccount = Depends(get_current_account),
+    pool: asyncpg.Pool = Depends(get_pool),
+) -> dict[str, Any]:
+    """Return persisted public-research citations for this post's source leads."""
+
+    post = await _load_visible_post(post_id, account, pool)
+    if str(post["visibility_code"]) != VISIBILITY_PUBLIC:
+        return {
+            "post_id": str(post["post_id"]),
+            "visibility_code": post["visibility_code"],
+            "unavailable_reason": PRIVATE_POST_UNAVAILABLE,
+            "citations": [],
+        }
+    async with pool.acquire() as conn:
+        citations = await list_source_research_citations(conn, post_id)
+    return {
+        "post_id": str(post["post_id"]),
+        "visibility_code": post["visibility_code"],
+        "unavailable_reason": None,
+        "citations": [
+            {
+                "lead_kind_code": row["lead_kind_code"],
+                "lead_source_unit_id": row["lead_source_unit_id"],
+                "lead_image_region_id": row["lead_image_region_id"],
+                "lead_excerpt_text": row["lead_excerpt_text"],
+                "search_query_text": row["search_query_text"],
+                "evidence_url": row["evidence_url"],
+                "evidence_title_text": row["evidence_title_text"],
+                "evidence_excerpt_text": row["evidence_excerpt_text"],
+                "judgment_code": row["judgment_code"],
+                "rationale_text": row["rationale_text"],
+                "next_action_text": row["next_action_text"],
+                "checked_at": row["checked_at"],
+            }
+            for row in citations
+        ],
+    }
+
+
+@app.post("/api/posts/{post_id}/research-citations")
+async def research_post_source_references(
+    post_id: str,
+    account: CurrentAccount = Depends(get_current_account),
+    pool: asyncpg.Pool = Depends(get_pool),
+    valkey: redis.Redis = Depends(get_valkey),
+) -> dict[str, Any]:
+    """Search and retrieve a public resource for this post's source leads.
+
+    Private posts fail closed without sending content. Gated by post_admin
+    because retrieval is a real external-search write action.
+    """
+
+    _require_post_admin(account)
+    post = await _load_visible_post(post_id, account, pool)
+    if str(post["visibility_code"]) != VISIBILITY_PUBLIC:
+        return {
+            "post_id": str(post["post_id"]),
+            "visibility_code": post["visibility_code"],
+            "unavailable_reason": PRIVATE_POST_UNAVAILABLE,
+            "citations": [],
+        }
+    client = _source_research_client()
+    if not client.available:
+        raise HTTPException(
+            status.HTTP_503_SERVICE_UNAVAILABLE,
+            "Public research is unavailable. Ask an administrator to enable it, "
+            "then try again.",
+        )
+    try:
+        with use_llm_metadata(build_post_llm_metadata(post_id, post)):
+            run = await research_post_sources_from_pool(
+                pool,
+                client,
+                post_id,
+                visibility_code=str(post["visibility_code"]),
+            )
+    except (HttpClientError, OSError, ValueError) as exc:
+        raise HTTPException(
+            status.HTTP_503_SERVICE_UNAVAILABLE,
+            "Public research could not be completed. Try again later or review "
+            "this post's existing evidence.",
+        ) from exc
+    except Exception as exc:  # noqa: BLE001 - provider boundary is fail-closed.
+        raise HTTPException(
+            status.HTTP_503_SERVICE_UNAVAILABLE,
+            "Public research could not be completed. Try again later or review "
+            "this post's existing evidence.",
+        ) from exc
+    await publish_activity_event(
+        valkey,
+        post_id,
+        "source_research_checked",
+        account.user_account_id,
+        f"Public sources reviewed: {len(run.citations)} item(s)",
+    )
+    return {
+        "post_id": run.post_id,
+        "visibility_code": run.visibility_code,
+        "unavailable_reason": run.unavailable_reason,
+        "citations": [citation.to_payload() for citation in run.citations],
+    }
+
+
 @app.post("/api/posts/{post_id}/extract-keymen")
 async def extract_post_keymen(
     post_id: str,

backend/app/mcp_server.py

@@ -246,7 +246,7 @@ async def lifespan(_: MCPServer) -> AsyncIterator[McpAppContext]:
         "lineageweave",
         title="LineageWeave",
         description="Authenticated provenance-bearing lineage intelligence.",
-        version="2.18.0",
+        version="2.19.0",
         lifespan=lifespan,
         token_verifier=token_verifier or KeyverseMcpTokenVerifier(resolved),
         auth=AuthSettings(