Skip to content

feat: authorized one-hop graph expansion with server-bounded examination (#151) - #202

Merged
stone16 merged 15 commits into
mainfrom
codex/issue-151-one-hop-expansion
Jul 31, 2026
Merged

feat: authorized one-hop graph expansion with server-bounded examination (#151)#202
stone16 merged 15 commits into
mainfrom
codex/issue-151-one-hop-expansion

Conversation

@stone16

@stone16 stone16 commented Jul 31, 2026

Copy link
Copy Markdown
Owner

What

Authorized one-hop graph expansion + evaluation slices (ADR-0093; migration 0052):

  • Supply-side link extraction at publication through the leased compiler-runner (engine/supply/link_graph.py): deterministic edges from validated v3 section source into revision_link_edge (six columns — refs/ordinals/kind only, no content, no per-edge ACL; path traversal blocked at construction; the stale LINK_OR_IMAGE refusal claim was re-verified and retired post-[M1] Tier-1 compiler-runner: vendor RAGFlow markdown parser #124).
  • One hop, never two — pinned at three independent levels: the caller's immutable main_projections snapshot; a Kernel-side guard refusing any anchor already in expanded_candidate_refs (no self-blinding — verified); and discovery's anchor-set check. The round-1 evaluation's surviving mutation (seeding expansion from the growing projection set) is now killed by a named test.
  • Same-Article/same-Revision inheritance after lineage verification; every cross-Article hop re-authorized through the UNCHANGED Kernel — with a mid-flight correction that removed an effective-scope pre-filter from discovery ("index filters never authorize"): everything pages through the Kernel. The one-hop definer function enforces role/limit/org/actor/membership checks with row_security = on inside the SECURITY DEFINER body.
  • Server-owned scanned-page ceiling (default 16, hard ceiling 64; exact-int validation, bool refused): total examination work is server-bounded (1,024 candidates by default) independent of corpus shape and refusal outcomes; a ceiling-hit is tenant-indistinguishable from neighbour exhaustion (differential test at the edge).
  • Expanded candidates COMPETE: rank-gated (relevance > 0.5) into [M2] Authorized ranking stage (ADR-0076) #131's single-source-of-order contract; no auto-inclusion; refused neighbours traceless in order, coverage, gaps, tenant bytes, and DecisionAudit — proven end-to-end over the REAL generated SDK (npm-packed into a live consumer).
  • Synthetic evaluation slices (adjacent-note + multi-hop with the two-hop note as a hard negative) through [M1] Golden set v1 + layered judges #129's loader and lock.

Verification (independent evaluator, both rounds, fresh)

Round 2: lint/typecheck clean · 2461 unit · 698 integration · catalog · M0 SECURITY PASS
Mutations 4/4 KILLED (incl. the round-1 survivor) · RLS recount re-derived: 73 total / 70 tenant = main + exactly revision_link_edge

Independent evaluation

Round 1: PASS with two MAJORs (unpinned one-hop bound — demonstrated by a surviving mutation; missing examination ceiling — the ADR-0083 resource class on a new path). Round 2: CONFIRM MERGE — both closed with mutation-kill evidence re-run by the evaluator. Follow-up filed: the co-resident UI's v1-pinned preview cannot ingest link-bearing notes (boundary decision documented in ADR-0093; leased preview replacement tracked separately).

Closes #151

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added rich Markdown v3 support for file imports, including deterministic revision-link extraction.
    • Added authorized one-hop navigation across linked revisions, with bounded expansion and relevance ranking.
    • Published linked-revision data atomically for consistent results.
  • Improvements

    • Compilation now runs during leased publication, improving scan scheduling and isolating compilation failures.
    • Scan reports no longer expose compilation refusal details.
  • Documentation

    • Added architectural guidance for rich Markdown publication and revision-graph navigation.
  • Tests

    • Expanded coverage for authorization, security, link extraction, publication, and migration behavior.

@stone16
stone16 merged commit 7f4af86 into main Jul 31, 2026
@stone16
stone16 deleted the codex/issue-151-one-hop-expansion branch July 31, 2026 13:47
@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: d0bb9696-b20b-4e36-818b-80e5f6c28230

📥 Commits

Reviewing files that changed from the base of the PR and between da7d2bf and a76d037.

📒 Files selected for processing (46)
  • STATUS.md
  • adapters/http/ui_api.py
  • applications/control.py
  • applications/file_scan.py
  • applications/leased_compiler_runner.py
  • docs/decisions/0093-activate-leased-rich-markdown-and-revision-link-graph.md
  • docs/decisions/README.md
  • engine/learning/governance.py
  • engine/persistence/configuration.py
  • engine/persistence/file_imports.py
  • engine/persistence/membership_context.py
  • engine/persistence/schema_security_manifest.yaml
  • engine/runtime/authorized_ranking.py
  • engine/runtime/construction.py
  • engine/runtime/materialized.py
  • engine/supply/compiler_runner.py
  • engine/supply/link_graph.py
  • engine/supply/markdown.py
  • eval/catalogs/m0-security-evidence.yaml
  • eval/golden/v1/synthetic-one-hop-graph-slices.json
  • eval/golden/v1/synthetic-one-hop-graph-slices.lock.json
  • migrations/versions/20260731_0052_revision_link_graph.py
  • scripts/provision_database_roles.py
  • scripts/security_gate/rls.py
  • tests/integration/test_file_dispatch.py
  • tests/integration/test_file_scan_operator_process.py
  • tests/integration/test_import_preview_requires_confirm.py
  • tests/integration/test_m0_security_gate_rls.py
  • tests/integration/test_membership_schema.py
  • tests/integration/test_migrations.py
  • tests/integration/test_multi_source_scan_status.py
  • tests/integration/test_no_false_deletes_at_bound.py
  • tests/integration/test_one_hop_graph_http_sdk.py
  • tests/integration/test_postgres_harness.py
  • tests/integration/test_revision_link_graph.py
  • tests/integration/test_z_egress_grant_file.py
  • tests/support/file_imports.py
  • tests/unit/test_compiler_runner_production_boundary.py
  • tests/unit/test_database_harness_contract.py
  • tests/unit/test_file_scan.py
  • tests/unit/test_golden_tracked_tree_is_synthetic.py
  • tests/unit/test_leased_compiler_runner.py
  • tests/unit/test_m0_rls_inventory.py
  • tests/unit/test_one_hop_runtime.py
  • tests/unit/test_revision_link_extraction.py
  • tests/unit/test_schema_security_manifest.py

📝 Walkthrough

Walkthrough

This PR activates ADR-0093: rich Markdown v3 compilation moves to a leased, isolated Supply-worker subprocess; file-scan preflight drops compilation and refusal classification; an immutable, content-free revision link graph is extracted, persisted, and secured with a graph-definer role; and the runtime gains bounded, authorized one-hop expansion with dedicated ranking.

Changes

Leased Markdown v3 and one-hop graph expansion

Layer / File(s) Summary
ADR-0093 and decision docs
STATUS.md, docs/decisions/0093-...md, docs/decisions/README.md
Adds ADR-0093 defining leased v3 compilation and one-hop graph semantics; updates status ledger and decisions index.
Remove compilation refusal from file scan
applications/file_scan.py, applications/control.py, adapters/http/ui_api.py, tests/integration/test_file_scan_operator_process.py, tests/integration/test_multi_source_scan_status.py, tests/integration/test_no_false_deletes_at_bound.py, tests/unit/test_file_scan.py
Replaces compilation/refusal classification in scan preflight with accepted-byte identity verification; scan reports no longer emit compilationRefusals.
Leased compiler runner subprocess
applications/leased_compiler_runner.py, engine/supply/compiler_runner.py, engine/supply/markdown.py, tests/unit/test_leased_compiler_runner.py, tests/unit/test_compiler_runner_production_boundary.py
Adds the isolated subprocess compiler runner, bumps active config version to v3, and validates timeout/crash/refusal handling.
Revision link extraction
engine/supply/link_graph.py, tests/unit/test_revision_link_extraction.py
Adds deterministic extraction of canonical revision links from rich Markdown with normalization, dedup, and rejection rules.
File-import publication routes through leased runner
engine/persistence/file_imports.py, tests/integration/test_import_preview_requires_confirm.py
Compiles v3 via the leased runner, validates effective config, and serializes extracted links with published documents.
Graph-definer role, schema manifest, migration
engine/persistence/configuration.py, engine/persistence/schema_security_manifest.yaml, migrations/versions/20260731_0052_revision_link_graph.py, scripts/provision_database_roles.py, scripts/security_gate/rls.py, tests/*schema*, tests/*rls*, tests/*migrations*, tests/support/file_imports.py
Adds the revision_link_edge table, graph-definer role/RLS policies, security-definer resolution functions, migration upgrade/downgrade guards.
Runtime one-hop discovery, authorization, ranking
engine/persistence/membership_context.py, engine/runtime/materialized.py, engine/runtime/construction.py, engine/runtime/authorized_ranking.py, tests/unit/test_one_hop_runtime.py
Adds bounded one-hop discovery, kernel authorization for same/cross-Article candidates, and lexical graph ranking.
End-to-end evaluation and integration coverage
eval/golden/v1/synthetic-one-hop-graph-slices*.json, eval/catalogs/m0-security-evidence.yaml, tests/integration/test_one_hop_graph_http_sdk.py, tests/integration/test_revision_link_graph.py, tests/integration/test_file_dispatch.py, tests/integration/test_z_egress_grant_file.py
Adds golden slices, HTTP/SDK integration coverage for authorized expansion, and updates dispatch/egress expectations for the v3 heading.
Golden lock validation
engine/learning/governance.py, tests/unit/test_golden_tracked_tree_is_synthetic.py
Validates tracked .lock.json files against sibling golden sets during governance checks.

Estimated code review effort: 5 (Critical) | ~150 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Worker as Supply Worker
  participant Runner as leased_compiler_runner (subprocess)
  participant Compiler as compile_rich_markdown
  participant LinkGraph as extract_revision_links
  participant DB as revision_link_edge table

  Worker->>Runner: launch subprocess (source, config, timeout)
  Runner->>Compiler: compile_rich_markdown(stdin)
  Compiler-->>Runner: parsed document or failure
  Runner-->>Worker: JSON envelope (base64 doc or failure code)
  Worker->>LinkGraph: extract_revision_links(document, source_path)
  LinkGraph-->>Worker: RevisionLink tuples
  Worker->>DB: persist revision snapshot + revision_link_edge rows
Loading
sequenceDiagram
  participant Runtime
  participant Kernel as AuthorizationKernel
  participant Membership as membership_context.discover_one_hop
  participant Graph as context_runtime_resolve_one_hop_graph (DB)
  participant Ranker as rank_authorized_one_hop

  Runtime->>Runtime: resolve authorized main-path results
  Runtime->>Membership: discover_one_hop(anchors, limit, offset)
  Membership->>Graph: query bounded one-hop edges
  Graph-->>Membership: candidate rows
  Membership-->>Runtime: MaterializedOneHopCandidate list
  Runtime->>Kernel: authorize_one_hop(candidates)
  Kernel-->>Runtime: AuthorizationDecision (expanded_candidate_refs)
  Runtime->>Ranker: rank_authorized_one_hop(query, projections)
  Ranker-->>Runtime: ranked CandidateRankEvidence
  Runtime-->>Runtime: select delivery excluding unranked expansions
Loading

Possibly related issues

Possibly related PRs

  • stone16/context-engine#54: Builds on the same engine/supply/markdown.py compiler contracts extended by v3 routing and revision-link extraction.
  • stone16/context-engine#56: Extends the same engine/persistence/file_imports.py and schema paths from v2 to leased v3 compilation.
  • stone16/context-engine#41: Extends the same engine/runtime/construction.py and engine/runtime/materialized.py authorization paths with one-hop discovery.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/issue-151-one-hop-expansion

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai

coderabbitai Bot commented Jul 31, 2026

Copy link
Copy Markdown

Caution

Failed to replace (edit) comment. This is likely due to insufficient permissions or the comment being deleted.

Error details
putComment timed out

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a76d0377ca

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +456 to +460
expanded.organization_id = expanded.anchor_organization_id
AND expanded.source_ref = expanded.anchor_source_ref
AND expanded.resource_ref = expanded.anchor_resource_ref
AND expanded.revision_id = expanded.anchor_revision_id
AND expanded.fragment_ref = expanded.anchor_fragment_ref

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Exclude every main anchor from graph results

When two authorized main-path results are linked—for example, A links to B and both fragments were returned by initial retrieval—this predicate excludes a row only when it matches its own originating anchor, so B can still be returned as a candidate rooted at A. _discover_materialized_one_hop then rejects any candidate equal to any anchor and raises ValueError, failing the entire Acquire request; exclude candidates matching the complete requested anchor set before returning the page.

Useful? React with 👍 / 👎.

Comment on lines +107 to +110
source_text = "\n".join(
section.text
for section in document.sections
if section.kind is not SectionKind.FENCED_CODE

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Extract links without altering section boundaries

When a long paragraph is split at the rich compiler's token ceiling, joining document.sections with a newline changes the original source at each split. If that boundary falls inside a whitespace-containing construct such as [[Target Note]] or an angle-bracketed reference target, the regexes—which reject newlines inside targets—silently omit a valid graph edge; adapters/parsers/ragflow_markdown.py:685-693 shows these splits can occur between any two non-whitespace tokens, so extraction should preserve the original inter-section source span or derive links before splitting.

Useful? React with 👍 / 👎.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[M2] Authorized one-hop graph expansion + evaluation slices

1 participant