diff --git a/docs/decisions/0094-admit-a-format-neutral-parsed-document-family.md b/docs/decisions/0094-admit-a-format-neutral-parsed-document-family.md new file mode 100644 index 00000000..040984e4 --- /dev/null +++ b/docs/decisions/0094-admit-a-format-neutral-parsed-document-family.md @@ -0,0 +1,118 @@ +--- +name: adr-0094-admit-a-format-neutral-parsed-document-family +version: "1.0.0" +description: > + Fix one format-neutral ParsedDocument family with nominal per-format source + locators, a closed structural-kind vocabulary including FIGURE, and explicit + versioned format profiles, so DOCX and PDF compilation can be built without + disguising their provenance as Markdown byte spans. Use when adding a + non-Markdown compiler or changing publication provenance contracts. Not an + approval of any third-party copy or of PDF model activation. +--- + +# 0094. Admit a format-neutral parsed document family + +- Status: accepted +- Date: 2026-07-31 +- Refines: ADR-0036, ADR-0038, ADR-0079 +- Related: ADR-0018, ADR-0066, ADR-0074 +- Decision input: `docs/research/2026-07-31-five-repository-implementation-blueprint.md` §5 (D1, decided 2026-07-31) and `docs/research/2026-07-31-ragflow-blueprint-evaluation.md` §3.1 (maintainer-local Room-A research; not public provenance) + +## Context + +The accepted Markdown compilers fix `ParsedDocument` around `SectionKind` and +UTF-8 byte spans into canonical source text: ADR-0036 (v1) and ADR-0038 (v2) +are the activated publication contracts, while ADR-0079 (v3) admits only a +local/acceptance transform whose production publication remains `NOT_ACTIVE`. +That representation is honest for Markdown but cannot express a DOCX zip +member (`part_uri` + block ordinal + XML digest) or a PDF region (page number, +bounding box, render digest, extraction method). The five-repository blueprint +evaluation established that the first non-Markdown lifts under ADR-0074 are +RAGFlow's `deepdoc/parser/docx_parser.py` and `extract_pdf_outlines`, and that +writing either compiler before the representation contract is fixed would force +PDF bounding boxes to masquerade as Markdown byte spans, corrupting Revision +provenance identity and every citation that derives from it. + +Maintainer decision D1 selects one format-neutral family contract with nominal +per-format locator subtypes and a new `FIGURE` structural kind, over parallel +sibling document types or indefinite deferral. + +## Decision + +1. **One family contract.** `ParsedDocument` is the format-neutral publication + representation: canonical serialization, all-or-nothing construction, + self-validating domain constructors, server-owned hard bounds (artifact + bytes, pages, pixels, blocks, cells, text length, runner wall time), a + closed typed refusal vocabulary, and two never-collapsed identities over the + whole canonical document: a content identity digest and a compilation + identity digest, generalizing ADR-0036's `content_hash`/`compilation_digest` + separation to every format. The Supply publication seam + (`prepared → indexed → active`, ADR-0018) accepts any family member through + the same protocol. +2. **Nominal source locator union.** Every structural unit carries one or more + typed locators from the closed union: `TextByteSpan(source_identity_digest, + start, end)` over the profile's declared canonical source text — for the + frozen v1/v2 Markdown profiles, ADR-0036's normalized canonical UTF-8, + preserving existing v1/v2 content identities (v3 falls under the next + branch, its byte spans round-tripping against the original input per + ADR-0079 clause 4); profiles that publish original-artifact round-trips bind + the original artifact digest; + `DocxXmlLocator(artifact_digest, part_uri, block_ordinal, xml_digest)`; + `PdfRegionLocator(artifact_digest, page_number, bbox_points, + page_render_digest, extraction_method)`. Tables and figures may carry + multiple ordered locators but remain one structural unit. Locators are + nominal types, not interchangeable dicts; a locator from another format is a + construction failure, not a fallback. +3. **Closed structural-kind vocabulary.** `HEADING`, `PARAGRAPH`, `LIST`, + `TABLE`, and `FENCED_CODE` are the existing frozen nominal kinds; `FIGURE` + is the only kind this decision adds. One + structural unit is exactly one Fragment; table cells, OCR words, and PDF + lines are typed metadata inside their unit, never independent Fragments. + Heading ancestry is copied into the same Fragment at compile time and counts + toward budget (ADR-0038); Runtime never fetches a "parent Fragment" to + restore headings. `FIGURE` units reference image bytes through a separate + bounded image-artifact policy; until that policy is admitted by its own + decision, a format profile either refuses figure-bearing artifacts or emits + a content-less figure descriptor (caption/locator only), never silent + omission and never inline bytes. +4. **Explicit versioned format profiles.** Each format compiles under an + immutable profile identity (the existing `context-engine-markdown-v*` + grammar family and `markdown-config-v*` configuration family for Markdown; + new formats add their own, for example `docx-config-v1` and + `pdf-text-outline-v1`) carried by `CompilationProfileRef`. + A profile fixes its grammar surface, locator usage, bounds, determinism + policy, and refusal categories. Unknown or unsupported profiles refuse before + artifact bytes are opened and before any model is loaded. +5. **Revision identity binds the full provenance chain.** `ContextRevision` + binds the original artifact SHA-256, the exact compiler/profile identity, + any model-bundle identity, and the complete canonical parsed-document + digest. A format's derived identities are versioned artifacts of that + profile, never recomputed silently. +6. **Frozen Markdown, no reinterpretation.** Markdown v1/v2/v3 fixtures and + semantics are frozen. The family contract must not reinterpret an existing + Revision; new formats receive their own format versions and the publication + seam distinguishes profiles explicitly. +7. **No copy and no activation inside this decision.** This ADR approves no + `third_party/` addition and no PDF profile activation. DOCX/outline copying + proceeds only under its own approval issue (decision D6) with ADR-0074 + registration; PDF profiles requiring model assets remain `NOT_ACTIVE` until + the asset gate closes under the D12 offline digest-bound bundle and single + runtime target decision, admitted by a separate activation record. + +## Consequences + +- DOCX and PDF-outline compilation become specifiable and testable against one + publication seam; PDF bbox provenance can never be disguised as a byte span. +- The ADR-0079 runner envelope (fixed deadline, typed `CompilationFailure`, no + network/DB/state) is reused unchanged for new format runners. +- `FIGURE` content policy is deliberately deferred; figure-bearing profiles are + honest about the gap instead of silently dropping images. +- Every new format pays the representation cost once (profile, locators, + fixtures, determinism proof) before product code. + +## Revisit trigger + +Revisit if a required format needs locator semantics that do not fit the closed +union, if the bounded image-artifact policy for `FIGURE` is admitted, or if +model-asset verification reopens PDF layout profiles under a different runtime +target policy. diff --git a/docs/decisions/0095-bridge-runtime-rerank-through-one-internal-package.md b/docs/decisions/0095-bridge-runtime-rerank-through-one-internal-package.md new file mode 100644 index 00000000..cc18f8e9 --- /dev/null +++ b/docs/decisions/0095-bridge-runtime-rerank-through-one-internal-package.md @@ -0,0 +1,178 @@ +--- +name: adr-0095-bridge-runtime-rerank-through-one-internal-package +version: "1.0.0" +description: > + Bridge Runtime model-backed rerank to the ADR-0052 Package-derived model + input rule through one internal, undelivered, audience-bound pre-rerank + ContextPackage, keeping one nominal AuthorizedModelInput contract with one + constructor per authorized process composition and at most one caller-visible + grant per resolve. Use when implementing the authorized rerank carrier + (ADR-0075 lift 5) or any future Runtime-internal model inference. Not an + activation of the rerank carrier and not a relaxation of ADR-0012, ADR-0046, + or ADR-0052 outside the exact carve-outs recorded here. +--- + +# 0095. Bridge Runtime rerank through one internal pre-rerank Package + +- Status: accepted +- Date: 2026-07-31 +- Refines: ADR-0012, ADR-0046, ADR-0052, ADR-0075, ADR-0076 +- Related: ADR-0077, ADR-0083 +- Decision input: `docs/research/2026-07-31-five-repository-implementation-blueprint.md` §5 (D2, decided 2026-07-31) and `docs/research/2026-07-31-onyx-blueprint-evaluation.md` §3.5 (maintainer-local Room-A research; not public provenance) + +## Context + +ADR-0052 fixes model-generation governance to one nominal `AuthorizedModelInput` +constructed from one complete current audience-bound ContextPackage plus a +matching one-shot `EgressGrant`, with the constructor realized as the private +TypeScript `prepareAuthorizedModelInput` factory inside the trusted Bot +application process. ADR-0012 fixes that inside Runtime, content-bearing rerank +and the other content-bearing stages accept `AuthorizedProjection` only, and +that the Package-derived `AuthorizedModelInput` belongs to the separate +downstream answer-generation boundary. ADR-0075 clause 4 repeats the +projection-only rule for content-bearing stages and schedules an authorized +rerank carrier (lift 5) inside the engine's governed model-inference port, +which lives in the Runtime process (Python), not in the Bot application. + +Taken literally and unreconciled, these decisions make the rerank carrier +unimplementable: the Bot-side constructor cannot be called from Runtime (the +accepted topology admits Bot → generated SDK → engine and forbids the reverse +import direction), while an engine-side constructor would be a second nominal +type unless the decisions are refined to say what "one nominal contract" means +across process compositions. The Onyx lift-5 evaluation confirmed that every +upstream rerank shape holds raw credentials, provider fallbacks, and plain +passage lists with no audience/package/grant/budget boundary, so porting any of +it is excluded; the gap is purely a ContextEngine composition question. + +Maintainer decision D2 selects the two-Package timing — form one internal +undelivered Package and feed it to the Package-derived model input rule — with +rerank output fixed as an exact Evidence permutation and no second nominal type +with the same name. This ADR records the exact refinements that make D2 +consistent with ADR-0012, ADR-0046, ADR-0052, ADR-0075, and ADR-0076. + +## Decision + +1. **One internal pre-rerank Package.** After authorization and the authorized + ranking stage (ADR-0076), Runtime composes one **internal, undelivered** + ContextPackage over exactly the admitted `AuthorizedProjection`s. It carries + every delivered-Package invariant: current Organization/Membership, + audience, purpose, Policy snapshot/epoch, Block↔Evidence one-to-one closure, + expiry, and package digest. Its purpose is the server-owned closed value + `pre_rerank_model_input`; callers never supply or select it (the purpose + binding discipline of ADR-0022 and ADR-0046 applies). It is never delivered + and never caller-visible; its sole content egress is the exact internal + model hop of clause 4, after the mandatory internal `EgressGate` and + one-shot grant redemption — every other egress remains prohibited. It + produces no operator-visible Package; its digest and its meter reservations + never enter ContextRun or DecisionAudit; the one ContextRun of the resolve + binds only the final delivered Package digest (ADR-0031). +2. **One nominal contract, one constructor per authorized process + composition.** The ADR-0052 rule — exactly one complete current + audience-bound ContextPackage, a matching one-shot `EgressGrant`, the closed + question envelope, trusted time, and the Release-manifest-bound versioned + model profile, yielding `AuthorizedModelInput` — is one nominal contract, + not one process. One nominal contract means one contract definition + (construction rule, canonical serialization, and digest), not one + language-runtime type: digest-equivalent twins validated under shared + fixtures are how this repository names cross-composition contract identity, + as ADR-0052's own Python/TypeScript digest authorities already establish. + Decision D2's prohibition on "a second nominal type with the same name" + forbids any second contract under this name with different construction + rules — projection-fed, candidate-fed, or caller-authored input; no such + contract exists. The Bot-application TypeScript factory remains the sole + constructor for generation hops and its private boundary is unchanged. For + the Runtime rerank port, the engine exposes the same nominal + `AuthorizedModelInput` contract with exactly one engine-side constructor + that applies the identical rule to the internal pre-rerank Package; the + Bot-side and engine-side constructors are proven digest-equivalent under + shared fixtures (ADR-0052 digest-twin discipline), including the versioned + profile binding. No other constructor exists. `AuthorizedProjection`s, + `CandidateRef`s, duck-typed packages, two-Package batches, and inputs + missing the closed question envelope, trusted time, or release-bound profile + cannot construct `AuthorizedModelInput` in either process. +3. **Exact-permutation output contract.** The rerank provider returns an exact + permutation of **all** input Evidence indices — nothing else. Invented refs, + duplicate or out-of-range indices, and non-finite scores are construction + failures. Subset selection happens only during final Package construction + under the budget rules; the rerank result itself is a permutation. Returned + scores never enter authorization decisions, never become public Package + fields, and never leave the governed port except as closed digest/category + trace. +4. **Grant issuance (refines ADR-0046).** ADR-0046's sentence "Runtime issues + at most one variant for a resolve" is replaced, for resolves in which the + rerank carrier is active, by: Runtime issues at most one **internal + model-hop grant** — issued inside the retained current-UserActor + transaction after internal Package construction, budget, provenance, and + current-epoch validation pass a mandatory **internal `EgressGate`** + mirroring the final gate inside Runtime; bound with the complete relevant + ADR-0046 binding set (Organization, internal Package digest, canonical + payload digest, purpose, audience digest, Policy Epoch, hop variant + `internal_model`, retention and sensitivity profiles, issuer Runtime, + consumer the governed rerank port, provider/model/region, issuance, expiry, + profile lineage); redeemed under the same one-shot digest-only retention + and generic zero-byte failure discipline as the final hop; redeemed inside + Runtime and never returned to any caller or transport — plus at most one + **final-hop grant** carrying the same complete binding set against the + delivered Package. Callers still receive at most one grant variant (the + final hop); the internal-only default for resolves without rerank is + unchanged; the mandatory final `EgressGate` still runs after final Package + construction; grants are never reused across hops or resolves. Multi-hop + batches beyond this exact two-hop sequence remain deferred under ADR-0046's + revisit trigger. +5. **Order precedence (refines ADR-0076).** Rerank consumes the ADR-0076 + authorized-ranking-stage order as its sole ordering input. While the rerank + carrier is active, selection, budget packing, and assembly order derive + solely from the exact rerank permutation; while it is inactive, the + authorized-ranking-stage order governs unchanged. This is the permitted + reading of ADR-0076 clause 4's "read rank only from this stage" once rerank + is active; rerank scores remain non-authoritative, never enter + authorization, and never become Package fields. ADR-0096 clause 5 mirrors + this precedence for token accounting and packing. +6. **Projection-rule carve-out (refines ADR-0012 and ADR-0075 clause 4).** + Inside Runtime, the model-backed rerank port is the single content-bearing + stage that consumes the nominal `AuthorizedModelInput`, constructed only over + the internal Package whose Evidence is exactly the admitted + `AuthorizedProjection`s. Every other content-bearing stage — dedupe, token + accounting, expansion hydration, Assembler, ordinary trace, ContextRun — + keeps the ADR-0012/ADR-0075 rule: `AuthorizedProjection` only. The + `CandidateRef → AuthorizationKernel → AuthorizedProjection` order is + preserved: Kernel authorization and ADR-0076 ranking both precede internal + Package composition. A static gate proves the previous projection-fed rerank + request shape cannot reach any provider. +7. **Release-bound profile, fail closed.** The rerank profile is an immutable + Release-manifest-bound Runtime profile, validated at composition activation + and on every request (ADR-0068 clause 6 discipline). Its unavailability + behavior is frozen in the profile and defaults to closed unavailability. No + fallback model call and no silent degradation to retrieval order exist + unless a separate recorded maintainer decision defines them and their + activation evidence. +8. **Activation stays deferred.** The rerank carrier remains `NOT_ACTIVE`. Its + owning issue must register, before activation: the security oracle that + denied/cross-Organization candidates mixed into an authorized set contribute + zero content bytes to the rerank gateway and assembler; release-binding and + per-request validation of the rerank profile; one-shot grant redemption + evidence for both hops; exact-permutation negative tests; and cumulative + shared-meter usage in the final Package (ADR-0096). Real provider network + calls remain a separate activation gate. + +## Consequences + +- ADR-0052's letter holds: every `AuthorizedModelInput` derives from one + current audience-bound ContextPackage through one nominal contract; "one + constructor per authorized process composition, digest-twin validated" is the + recorded meaning across the Bot and engine boundaries. +- ADR-0046's caller-facing guarantees hold: a caller still sees at most one + grant variant per resolve; the internal model-hop grant never leaves Runtime. +- ADR-0012 and ADR-0075 keep the projection-only rule for every content-bearing + stage except the single carved-out model-backed rerank port. +- Onyx lift 5 is unblocked without porting any upstream rerank code. +- One extra internal Package composition per reranked resolve is the accepted + price of one nominal contract and one governance story. + +## Revisit trigger + +Revisit if measured internal-Package overhead becomes material, if a second +Runtime-internal model use case appears that would generalize the internal +purpose envelope or the per-composition constructor rule, or if a rerank +fallback policy or a multi-hop batch beyond this exact two-hop sequence is +proposed (each requires its own decision under the ADR-0046 revisit trigger). diff --git a/docs/decisions/0096-bind-package-token-accounting-to-release-tokenizer.md b/docs/decisions/0096-bind-package-token-accounting-to-release-tokenizer.md new file mode 100644 index 00000000..29bcc4d0 --- /dev/null +++ b/docs/decisions/0096-bind-package-token-accounting-to-release-tokenizer.md @@ -0,0 +1,158 @@ +--- +name: adr-0096-bind-package-token-accounting-to-release-tokenizer +version: "1.0.0" +description: > + Bind Package token accounting to a ReleaseManifest-fixed tokenizer profile + and one resolve-owned cumulative budget meter (tokens, provider calls, cost, + elapsed) published in the final Package and ContextRun, through a reviewed + new contract version that never mutates frozen v0. Use before activating any + model-backed Runtime carrier (rewrite, rerank, select). Not an activation of + those carriers and not a tokenizer vendor choice. +--- + +# 0096. Bind Package token accounting to a release tokenizer + +- Status: accepted +- Date: 2026-07-31 +- Refines: ADR-0047 (the release-lineage tokenizer reference and the `utf8-byte-budget-v1` accounting profile it currently fixes on v0); ADR-0048 (generated-SDK coexistence under the new reviewed contract version) +- Related: ADR-0012, ADR-0022, ADR-0031, ADR-0033, ADR-0066, ADR-0067, ADR-0068, ADR-0079 +- Decision input: `docs/research/2026-07-31-five-repository-implementation-blueprint.md` §5 (D4, decided 2026-07-31) and `docs/research/2026-07-31-onyx-blueprint-evaluation.md` §3.6 (maintainer-local Room-A research; not public provenance) + +## Context + +The delivered-Package token accounting currently fixed on v0 release lineage +(ADR-0047) is the byte-based accounting profile `utf8-byte-budget-v1`. A +separate constant, `utf8-byte-token-v1`, exists only as code in the +not-yet-active model-inference port (`engine/runtime/model_inference.py`) and +has never been admitted by any ADR; it is not the delivered-Package accounting +profile. (ADR-0066 governs Fragment embeddings and contains no token +accounting.) The byte profile is honest for the loopback dogfood carrier +(ADR-0068), which makes no provider calls and reports zero-filled usage +honestly, but it cannot govern model-backed Runtime carriers (`rewrite`, +`rerank`, `select`): their profiles need true token counts, and their cost must +be metered cumulatively with assembly bytes. The atomic `PackageBudgetMeter` +(reserve/commit/cancel) already exists; what is missing is one resolve-owned +meter shared by all stages, a Release-bound real tokenizer identity, and +publication of cumulative usage. The Onyx lift-6 evaluation showed upstream +budgeting helpers (fixed 75-token metadata estimates, best-effort BPE +trimming) are unsafe as hard budgets, so nothing is ported. + +Maintainer decision D4 selects introducing a real ReleaseManifest-bound +tokenizer **before** any model-backed carrier activates, with one +schema/OpenAPI migration, over keeping byte accounting indefinitely or +publishing two accounting dimensions side by side. + +## Decision + +1. **Tokenizer profile in the ReleaseManifest.** Token accounting is governed + by an immutable tokenizer profile referenced by the active ReleaseManifest: + pinned tokenizer artifact digest, vocabulary/normalization identity, and + accounting version. Package `tokens` are counted only under the manifest's + tokenizer. The tokenizer artifact is a pinned, hash-verified dependency; no + network fetch at resolve time; provider-side tokenizer APIs never decide a + hard budget. **Fail closed:** a missing, unknown, unavailable, or + hash-mismatched tokenizer artifact; a carrier profile naming a different + tokenizer than the active ReleaseManifest; or mixed tokenizer identities + within one resolve — each refuses before any counting and before the first + provider byte, with zero fallback to `utf8-byte-budget-v1`, to a provider + tokenizer API, to cached counts, or to a stage-local meter. This ADR + governs Package accounting only; ADR-0079's representation-bound compile-time + splitting counter remains profile-local and unchanged. +2. **Ordered migration, new contract version, one deadline.** The migration + completes **before** any model-backed Runtime carrier is activated, in this + order: (a) admit an immutable tokenizer-profile record with digest bindings + in the release lineage, giving substance to the ADR-0047 tokenizer + reference; (b) land cumulative usage semantics and tokenizer profile + references through a **reviewed new public contract version** per ADR-0047 — + historical v0 artifacts are never mutated — refining ADR-0048 as follows: + the new version is a separate generated tree with its own checksum gate; + the frozen v0 generated tree, facade export, and `/v0/resolve` client are + unchanged and coexist with the new-version client until the owning + contract-version issue records migration; package-consumer negative gates + are regenerated for both versions; activation and migration ownership lies + with that owning issue, not with any carrier activation; (c) fix read + semantics for + Packages produced under `utf8-byte-budget-v1` (their accounting profile is + recorded and preserved, never recounted); (d) make tokenizer identity part + of ReleaseManifest compatibility, so a tokenizer change composes a new + manifest generation through the sole promote path (ADR-0033); (e) refuse + downgrade and mixed-generation resolves (a resolve never counts under two + tokenizer identities). Until the migration completes, `utf8-byte-budget-v1` + remains the explicit named accounting profile of the current surface — never + an implicit default. +3. **One resolve-owned cumulative meter.** Each resolve creates exactly one + `PackageBudgetMeter` over the effective budget intersection. Query rewrite, + query embedding, rerank, selection, and assembly reserve their profile + maximum before any provider byte, commit actuals, charge the frozen maximum + when a call occurred but its result is unusable, and cancel on pre-call + refusal. Metered dimensions are tokens, provider calls, cost, and elapsed, + alongside the existing block/evidence/latency bounds. Stage-local meters, + usage resets, and reconstruction of usage from block bytes alone are + removed. +4. **Cumulative usage is published and digest-covered.** The final + ContextPackage and its ContextRun publish the same cumulative meter usage; + the package digest commits to that usage. A consumer can read the cumulative + usage and verify the Package digest binds it; provider calls, cost, and + elapsed are execution facts that a consumer cannot recompute from Package + blocks alone, and this ADR does not claim otherwise. If independent + verification of those facts is ever required, it is served by a separate + restricted digest-only settlement proof admitted by its own decision, never + by Package-body reconstruction. +5. **Accounting seam rules only.** Package-content tokenization and packing + consume `AuthorizedProjection`s only, and count with the Release tokenizer + exactly; non-content provider stages (query rewrite, query embedding) meter + their closed request/response payloads under the same Release tokenizer and + the same resolve-owned meter, accepting no `CandidateRef` and no + unauthorized source content. Ordering follows the precedence fixed by + ADR-0095 clause 5 (authorized ranking stage order, or the exact rerank + permutation while the rerank carrier is active). Packing selection, + trimming policy, and any gap semantics remain under the frozen v0 contract — + whose `gaps` field stays empty — until a separate decision admits new + packing policy through a new reviewed contract version. This clause adds no + selection algorithm and no public-contract field. +6. **No retroactive recounting.** Delivered Packages and released manifest + generations keep the accounting profile under which they were produced. + Changing the tokenizer is a new manifest generation: it composes a new + release candidate, re-evaluates, and promotes through the sole + release-operator path (ADR-0033); it never mutates an active profile in + place and never recounts history. +7. **Determinism evidence.** A network-free CI twin proves cross-process count + equality under the pinned artifact; the activation record for the first + tokenizer profile records the twin digest and the exact counting contract. + Semantic equivalence to any provider tokenizer is out of scope — accounting + determinism, not vendor parity, is the gate. +8. **Activation evidence.** Before any model-backed carrier reports cumulative + usage as active, the owning issue registers executed evidence for: every + clause-1 fail-closed refusal (missing, unknown, unavailable, + hash-mismatched, carrier-mismatched, and mixed tokenizer identities, each + with its own activation oracle); one-meter identity across rewrite, + embedding, rerank, selection, and assembly (stage-local meters and usage + resets are statically and dynamically rejected); concurrent reservation + behavior (over-limit concurrent reserves admit exactly one; cancels do not + leak reservations); maximum charging when a call occurred but its result is + unusable; zero provider bytes before reservation; and exact equality of + final Package and ContextRun cumulative usage with digest binding. + +## Consequences + +- Model-backed carriers gain a complete meter: every provider call, cost, and + elapsed interval is reserved, settled, and published with the Package, + closing the ADR-0067 and ADR-0068 revisit triggers on Runtime-enforced + provider-call, cost, and elapsed accounting. +- One reviewed new contract version (tokenizer profile references plus + cumulative usage semantics) happens before model-backed activation, rather + than mutating v0 or migrating twice. The budget usage field shape already + exists on v0; the migration changes its accounting profile binding and + cumulative semantics, not its existence. +- `STATUS.md` reporting for `rewrite`/`rerank`/`select` can move from + zero-filled usage to cumulative usage only after this profile is fixed. +- Byte-accounting history remains auditable: old Packages stay recomputable + under `utf8-byte-budget-v1`. + +## Revisit trigger + +Revisit if a replacement tokenizer is required (new manifest generation, same +promote path), if cost accounting needs finer dimensions than calls/cost/ +elapsed, if packing or gap policy is proposed (separate decision, new contract +version), or if multi-target determinism evidence forces a counting-contract +change. diff --git a/docs/decisions/README.md b/docs/decisions/README.md index 062befa2..673dabc3 100644 --- a/docs/decisions/README.md +++ b/docs/decisions/README.md @@ -64,6 +64,9 @@ kernel, capability separation, and publication visibility model. | Exact Package egress | [0046 — Bind egress to one exact Package hop](0046-bind-egress-to-one-exact-package-hop.md) | One digest-only grant binds one exact audience-bound Package to one model or channel preflight hop and redeems atomically | Treating Package construction as disclosure authority, arbitrary content at egress, cross-hop reuse, or bypassing final policy | | Public OpenAPI v0 | [0047 — Freeze OpenAPI v0 through one Runtime path](0047-freeze-openapi-v0-through-one-runtime-path.md) | One public `/v0/resolve` schema and a hidden provisional v1 bridge share the same sealed Runtime; Package release lineage is read-only from the Learning-published active manifest | Two authorization compositions, caller-authored release facts, Runtime publication/fallback, or in-place mutation of historical snapshots | | Local evidence console | [0090 — Admit a co-resident local evidence console](0090-admit-a-co-resident-local-evidence-console.md) | One explicitly authenticated server-rendered UI calls typed HTTP carriers inside the API ingress while frozen OpenAPI v0 stays unchanged; Control jobs additionally consume one separate exact-operation Control call | Anonymous process-identity inheritance, Membership-as-Control, direct UI-to-engine calls, denied rank/score leakage, or feedback publication authority | +| Format-neutral compilation | [0094 — Admit a format-neutral parsed document family](0094-admit-a-format-neutral-parsed-document-family.md) | One ParsedDocument family with nominal per-format locators, a closed structural-kind vocabulary including FIGURE, versioned format profiles, and Revision identity binding the full provenance chain | PDF or DOCX provenance disguised as Markdown byte spans, cross-format locator fallback, silent figure omission, or any third-party copy or PDF model activation inside this decision | +| Runtime rerank bridge | [0095 — Bridge Runtime rerank through one internal pre-rerank Package](0095-bridge-runtime-rerank-through-one-internal-package.md) | One internal undelivered audience-bound pre-rerank Package feeds the single nominal AuthorizedModelInput contract through a mandatory internal EgressGate and one-shot internal model-hop grant, with rerank output an exact Evidence permutation | A second same-name contract with different construction rules, caller-visible internal grants or Packages, rerank scores entering authorization or Package fields, or projection-fed provider requests | +| Release-bound token accounting | [0096 — Bind Package token accounting to a release tokenizer](0096-bind-package-token-accounting-to-release-tokenizer.md) | One ReleaseManifest-pinned tokenizer profile and one resolve-owned cumulative meter (tokens, calls, cost, elapsed) published and digest-bound in the final Package through a reviewed new contract version | Byte-profile or provider-tokenizer fallback, stage-local meters and usage resets, frozen-v0 mutation, retroactive recounting, or mixed tokenizer identities within one resolve | | Generated TypeScript SDK | [0048 — Generate the TypeScript SDK behind a closed facade](0048-generate-typescript-sdk-behind-a-closed-facade.md) | OpenAPI v0 generates the semantic contract and internal fetch client; a package export map exposes only a metadata-safe facade and the contract checksum | Handwritten wire schemas, raw generated transport exports, arbitrary caller headers, floating generator/runtime versions, or claiming inactive capability redemption | | Citation open | [0051 — Reauthorize opaque citation opens](0051-reauthorize-opaque-citation-opens.md) | Digest-only multi-use locators recover only content-free target lineage; every open uses a current UserActor and the sealed Kernel to produce a replacement Package | Bearer authority, prior-decision reuse, source URLs, locator consumption on denial, or content before exact reauthorization | | Private model egress | [0052 — Gate model generation by one authorized Package](0052-gate-model-generation-by-package.md) | One nominal SDK-Package input and exact one-shot model grant gate the deterministic private ModelGateway, bounded answer, Package-subset citations, and digest-only audit | Arbitrary prompt context, multiple Packages, direct Gateway invocation, invented citations, effect intents, or unaudited answer release | @@ -204,3 +207,6 @@ touched: - [0091 — Reconcile connector ACL freshness at Supply acceptance](0091-reconcile-connector-acl-freshness-at-acceptance.md) - [0092 — Authorize Feishu subjects and bound Mirrored freshness](0092-authorize-feishu-subjects-and-bound-mirrored-freshness.md) - [0093 — Activate leased rich Markdown and the Revision link graph](0093-activate-leased-rich-markdown-and-revision-link-graph.md) +- [0094 — Admit a format-neutral parsed document family](0094-admit-a-format-neutral-parsed-document-family.md) +- [0095 — Bridge Runtime rerank through one internal pre-rerank Package](0095-bridge-runtime-rerank-through-one-internal-package.md) +- [0096 — Bind Package token accounting to a release tokenizer](0096-bind-package-token-accounting-to-release-tokenizer.md) diff --git a/docs/research/2026-07-31-dify-blueprint-evaluation.md b/docs/research/2026-07-31-dify-blueprint-evaluation.md new file mode 100644 index 00000000..f475d52c --- /dev/null +++ b/docs/research/2026-07-31-dify-blueprint-evaluation.md @@ -0,0 +1,465 @@ +# Dify → ContextEngine 可复刻蓝图评估（clean-room） + +> **决策状态**：本文开放问题已由维护者于 2026-07-31 全部决定（D5），结果见 [`five-repository-implementation-blueprint.md`](./2026-07-31-five-repository-implementation-blueprint.md) §5；正文推荐项为评估时刻的状态。 + +> **Room-A 研究产物 — 维护者本地研究，非公开 provenance；Room-B 实现者只读本报告的规格与 oracle，不读 Dify 源码** +> +> 本报告是固定 checkout 的静态源码观察与 ContextEngine 独立设计规格，不是公开引用权威，也不是法律意见。公开 prior-art 主张仍只回引 +> [`2026-07-19 four-repository evidence baseline`](./2026-07-19-four-public-repositories-evidence.md)。除明确标为本仓已激活的能力外，上游动态正确性、性能、故障恢复与生产安全均为 **[未取证]**。 + +## 1. 固定 commit 与许可证核验 + +### 1.1 固定范围与取证方法 + +- 上游仓库：`https://github.com/langgenius/dify.git`。 +- 固定 commit：[`120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5`](https://github.com/langgenius/dify/tree/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5)。Room-A 在 `/tmp/dify-bp` checkout 后以 `git rev-parse HEAD` 核对为该值。 +- 证据类型仅为固定 commit 的 **[一手静态]** source/license/tree 观察；未启动 Dify 服务、数据库、队列、向量后端或 tracing provider，未做故障注入与 benchmark。 +- 本报告不得成为 Room-B 对 Dify root code 的间接逐行翻译。Room-B 只实现下文 ContextEngine 自有 DTO、状态机、失败语义和 oracle；不得打开上游 root-licensed 实现来补足含糊处。 + +### 1.2 root license：全部 root-licensed code 只允许 clean-room + +Dify 固定 commit 的根 [`LICENSE`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/LICENSE#L1-L19) 声明“modified Apache License 2.0”，并对未经书面授权使用源码运营 multi-tenant environment 增加商业许可条件。ContextEngine 的产品目的正是 multi-tenant context delivery，因此按 +[`ADR-0074`](../decisions/0074-adopt-controlled-third-party-code-reuse.md) 的已接受裁决： + +1. `api/`、`web/` 及其他未被路径内独立许可证覆盖的代码全部是 **clean-room only**； +2. 本报告可保留可观察行为、接口形状和测试 oracle，不得复制实现、常量表、控制流或测试代码； +3. root license 结论不因单个文件缺少版权头而改变，也不能以“Apache-2.0 部分”绕过附加条件； +4. public provenance 不引用本报告新增的仓库外研究结论，只引用四仓 evidence baseline 或可复核的一手 permalink。 + +### 1.3 SDK 子树逐路径核验 + +| 固定路径 | 路径内证据 | 结论 | 本次处置 | +|---|---|---|---| +| `sdks/nodejs-client/**` | 子树有独立 [`LICENSE`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/nodejs-client/LICENSE#L1-L21)，正文为 MIT；[`package.json`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/nodejs-client/package.json#L13-L42) 同时声明 `license: MIT`、仓库子目录和 root-only export。 | 在该固定 commit、该路径边界内具备 ADR-0074 所要求的独立 MIT 路径证据；若复制仍须逐文件 hash、nested dependency/notice scan、审批、SBOM 和 artifact inclusion。 | **法律层可进入 copy+patch 审查，架构层候选为 none。** 不复制。 | +| `sdks/php-client/**` | [`README`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/php-client/README.md#L91-L95) 自述 MIT；但该子树 tree 中没有路径内 `LICENSE`，[`composer.json`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/php-client/composer.json#L1-L9) 也没有 `license` 字段。 | **[未取证]**。README 文字不足以替代 ADR-0074 要求的 exact path license-region verification。 | do-not-take；且 ContextEngine 没有 PHP SDK 需求。 | +| `sdks/` 其余说明 | 固定 [`sdks/README.md`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/README.md#L1-L25) 将 Java/Go/Ruby 指向外仓，并把 Python/PHP/Node 标为待迁移说明。 | 外仓不在本任务固定 commit 的许可证核验范围，不能继承 Dify 子树结论。 | do-not-take；若未来需要，重新固定各自 repo/commit/path。 | + +### 1.4 copy+patch 结论：none；生成式 SDK 路径严格更优 + +`sdks/nodejs-client` 虽有 MIT 路径证据，但不适合 ContextEngine： + +- 它手写 routes、wire types 和 client methods，而 ADR-0047/0048 要求 OpenAPI v0 是唯一语义源； +- 它从 root 导出 [`HttpClient`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/nodejs-client/src/index.ts#L88-L103)，并且 base client 暴露任意 endpoint/header 的 + [`sendRequest`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/nodejs-client/src/client/base.ts#L33-L68)；这与 ContextEngine “只导出 closed facade，调用者不能制造任意 trusted metadata/header”的要求冲突； +- 它的 KnowledgeBase client 面向 mutable Dataset/Document/Segment CRUD 与手写 indexing-status route，例如 + [`getDocumentIndexingStatus`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/sdks/nodejs-client/src/client/knowledge-base.ts#L319-L329)，不是 `ContextPackage` consumer； +- 本仓已经按 [`ADR-0048`](../decisions/0048-generate-typescript-sdk-behind-a-closed-facade.md) 激活 pinned OpenAPI codegen、generated-tree digest、closed export map、clean consumer install 与 compile-negative fixtures。复制会引入第二份 wire truth 和不必要的补丁负担。 + +因此本任务的 **copy+patch candidates = none**，无需创建 `third_party/dify/`。若未来架构决策被重开，下面仅是治理登记的最小模板，**不是复制批准**；每个 `[[files]]` 必须列举实际选中的文件与固定 commit 内容 hash，`approval` 在批准前不得填占位值后落库： + +```toml +repository = "https://github.com/langgenius/dify.git" +commit = "120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5" +source_paths = ["sdks/nodejs-client/"] +excluded_paths = ["api", "web", "sdks/php-client"] +reuse_mode = "copy-patch" +approval = "" +license = "MIT" + +[[files]] +upstream_path = "sdks/nodejs-client/" +vendored_path = "third_party/dify-node-sdk/" +sha256 = "" +``` + +实际重开还必须同时加入 `LICENSE.upstream`、`MODIFICATIONS.md`、nested notices、CycloneDX SBOM、`THIRD_PARTY_NOTICES.md` 聚合与 wheel/sdist/npm/container artifact completeness；仅有 `UPSTREAM.toml` 不构成合规。 + +## 2. 能力盘点 → ContextEngine 区域映射表 + +| Dify 可观察能力 | 固定上游入口 | ContextEngine 区域 / seam | 复用分类 | 决策摘要 | +|---|---|---|---|---| +| Document indexing lifecycle、pause/error/retry | [`api/core/indexing_runner.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/indexing_runner.py)；[`IndexingStatus`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/models/enums.py#L129-L139) | Supply：immutable ContextRevision、File publication checkpoint/job events、outbox、WorkerLease、dual watermarks、Control status | **clean-room Room-A spec** | 学习可观察 phase；不拿 mutable Document/Segment、先删旧 index 再重建或 exception text。 | +| 单库 LLM routing | [`single_retrieve`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L602-L738) | Runtime `QueryPlanner` + server-derived `AuthorizedSourceCapabilitySet` | **clean-room Room-A spec** | router 只能在已授权 capability 集中收窄；source 描述必须 content-free，route 不是授权。 | +| 多库 fan-out、hybrid、weighted/model rerank | [`multiple_retrieve`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L740-L887)；[`RetrievalService._retrieve`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/retrieval_service.py#L796-L931) | Candidate discovery、rank evidence、RRF、AuthorizationKernel、authorized ranking/rerank、PackageBudget | **clean-room Room-A spec** | 仅 content-free fusion 可在 Kernel 前；content-bearing rerank/dedupe/assembly 必须在 `AuthorizedProjection` 后。 | +| Candidate hydration、parent/child/summary restoration | [`format_retrieval_documents`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/retrieval_service.py#L488-L791) | CandidateRef → Article-level Kernel → lineage-verified same-Article expansion / cross-Article reauthorization | **do-not-take implementation；仅负面 oracle** | 上游 formatter 直接读取 segment/child content；这是必须杀死的授权前水合 premise。 | +| 普通 app 的 context string concat | [`DatasetRetrieval.retrieve`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L455-L600) | `ContextPackage` 是唯一 online deliverable；生成在 BotDelivery/上层 | **do-not-take** | 不返回裸字符串，不丢 Evidence、audience、policy、budget、provenance、TTL。 | +| Index processor factory | [`BaseIndexProcessor`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/index_processor/index_processor_base.py#L46-L108)；[`IndexProcessorFactory`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/index_processor/index_processor_factory.py#L10-L29) | DocumentCompiler + Supply execution seam + typed immutable DTO | **clean-room Room-A spec** | 学习 variation isolation；不拿混合 I/O、mutable ORM Dataset/Document 或“大而全”base class。 | +| Vector backend factory/entry point | [`BaseVector`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/vdb/vector_base.py#L18-L76)；[`vector_backend_registry.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/vdb/vector_backend_registry.py#L28-L87) | V1 native PostgreSQL FTS + pgvector；data-only CandidateIndex seam | **do-not-take portability；clean-room contract oracle only** | 在第二个真实 backend 前不承诺 portability；同名 search/filter/delete 不代表 tenant、filter 或原子语义一致。 | +| Datasource runtime/factory | [`datasource_manager.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/datasource/datasource_manager.py) | ContextProvider / connector-runner、SupplyDocumentEnvelope/ChangePage/checkpoint、per-connector twin | **clean-room Room-A spec** | connector 只提议 observation；engine durable acceptance 与 source authorization 仍是唯一真相。 | +| Workflow knowledge-retrieval node | [`KnowledgeRetrievalNode`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/workflow/nodes/knowledge_retrieval/knowledge_retrieval_node.py#L67-L182)；[`Source`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/workflow/nodes/knowledge_retrieval/retrieval.py#L12-L82) | Agent/Bot consumer 经 generated SDK 调一个 sealed Runtime；ContextPackage/ContextRun | **clean-room Room-A spec** | 学习“retrieval 是版本化产品对象”；不拿 caller-authored tenant/user/dataset IDs 或 raw source-rich output。 | +| REST retrieval/hit-test | [`service_api/dataset/hit_testing.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/controllers/service_api/dataset/hit_testing.py#L17-L68) | 公开 `POST /v0/resolve`；Control/UI hit test 也走最高 public seam | **clean-room Room-A spec** | 不建第二条“test retrieval”授权路径；同一 Runtime 返回同一 Package/error 语义。 | +| SDK | `sdks/nodejs-client/**` | immutable OpenAPI v0 → generated client → closed TypeScript facade | **copy+patch (MIT) eligibility，candidate none** | 路径许可可核验，但 generated path 严格更符合 ADR-0048。 | +| MCP product surface | [`MCPAppApi`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/controllers/mcp/mcp.py#L44-L96) | 可选 MCP ingress → 同一 `ContextRuntime.resolve` | **do-not-take now** | 本仓真实 caller 出现且 parity/security suite 完成前保持 `NOT_ACTIVE`，不复制 App MCP server。 | +| Trace provider selection 与 retrieval trace | [`OpsTraceProviderConfigMap`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/ops/ops_trace_manager.py#L216-L300)；[`dataset_retrieval_trace`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/ops/ops_trace_manager.py#L1094-L1180) | authorized-only digest-only ContextRun + restricted DecisionAudit + content-free metrics | **clean-room Room-A spec；raw trace do-not-take** | 保留 provider variation lesson；不保留 raw query、full documents、denied refs/counts、workspace/user display fields或 credential metadata。 | + +## 3. 逐能力蓝图 + +以下工期均为**相对当前 `STATUS.md` 激活面上的增量 engineer-days**，包括实现、单元/contract test、真实 PostgreSQL 测试与文档，不含 production identity、live connector tenant、外部法务或大规模 benchmark 等外部等待时间。 + +### 3.1 Indexing 状态机 → immutable Revision publication + +**上游路径与观察。** Dify 固定枚举公开 `waiting → parsing → cleaning → splitting → indexing → completed`，另有 `paused`/`error`（[`models/enums.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/models/enums.py#L129-L139)）。runner 在 extract 后写 splitting、保存 segment 后写 indexing、index worker 完成后写 completed，并把 exception text 写回 Document（[`indexing_runner.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/indexing_runner.py#L58-L67)、[`#L483-L491`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/indexing_runner.py#L483-L491)、[`#L676-L687`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/indexing_runner.py#L676-L687)）。retry/sync 可先删除旧 segment/vector 再重新处理（[`retry_document_indexing_task.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/tasks/retry_document_indexing_task.py#L81-L118)、[`document_indexing_sync_task.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/tasks/document_indexing_sync_task.py#L123-L160)）。这些只证明 UI/运营 phase 的价值；不会被继承为 publication semantics。 + +**本仓 seam / ADR。** `CONTEXT.md` 的 `ContextResource → immutable ContextRevision → ContextFragment`；ADR-0018（old-or-new pointer）；ADR-0040/0041（replacement plan + durable recovery boundary）；ADR-0043（acquisition checkpoint / publish watermark 分离）；ADR-0059/0060（exact WorkerLease dispatch/reclaim）；ADR-0072（content-free status）。 + +**Room-A 行为规格。** Room-B 实现下列独立状态域，不把它们压成一个 mutable `status`： + +1. **Durable acceptance / outbox。** 一个 accepted `SourceChange` 事务必须同时写入 immutable change record、按 source sequence 的 acquisition checkpoint、唯一 durable import job 和可 claim outbox/queue row。事务失败则四者都不存在；事务成功但 worker 未收到通知时，scheduler 仍能从 durable row claim，禁止“commit 后直接 `.delay()` 是唯一交付”。 +2. **Job/recovery checkpoint。** job 内部 checkpoint 只允许 `acquired → prepared → ready → completed` 单调前进；`interrupted`/`reclaimed` 是 immutable job events，不倒退 checkpoint。每次执行绑定 `(organization, sourceVersion, job, operation, resource?, revision?, leaseGeneration, nonce, expiry)` exact WorkerLease；高 generation reclaim 后旧 lease 任何 effect 为 0。 +3. **Revision lifecycle。** `ContextRevision` 只允许 `prepared → indexed → active`。`error`、`paused`、`retrying` 不是 Revision state：失败属于 job/refusal，暂停属于调度/Control，旧 active Revision 不被修改。prepared 必须固定 content/compilation/profile digest 与完整 Fragment lineage；indexed 必须证明所有 required index artifacts 完整且与该 Revision 一致。 +4. **Activation。** replacement `ready` 后，在同一 Organization publication exclusive barrier 下重新验证 current Source active、SourceVersion、job/lease generation、current acquisition authority、完整 artifact digest 与 expected previous active Revision；随后单事务 compare-and-swap `ContextResource.active_revision_ref`、append `active` publication event、supersession edge、publish completion并完成 job。读事务持 shared barrier，始终看完整旧版或完整新版。 +5. **CAS conflict。** 若另一并发任务已激活 bit-identical complete artifact，当前 job可在 guard lock 下分类为 `unchanged` 并零 publication effect 完成；若 active lineage 不同则保留 `ready`/产生 closed conflict category，重新取权威状态，不得覆盖。 +6. **失败与取消。** compile/index/embedding 失败：新 Revision永不 active，旧 Revision继续服务；仅持久化 closed refusal category，不存 source bytes、parser diagnostic 或 exception text。Source disable：同事务禁用 source、推进 Policy Epoch、取消 nonterminal jobs、写 cleanup intent；此后旧 lease effect 为 0。崩溃发生在任一已提交 boundary 后：新 generation 从 checkpoint 继续，不重复已证明的 deterministic work；未提交 work 视为未发生。 +7. **Operational status projection。** Control 可显示 `accepted/claimed/preparing/indexing/ready/reclaiming` 的 content-free aggregate、acquisition checkpoint、contiguous publish watermark、active Resource count、`never | last_success_at/age`、in-flight count、closed refusal category及 ADR-0072 已允许的当前 canonical path/digest/length。不要表面化 raw parsing text、exception、Fragment/denied identity或非连续“最新完成”水位。terminal `activated/unchanged/refused/cancelled` 可按 opaque job ref查看；它们不进入 Runtime authorization。 + +**ContextEngine 接口形状草图（不是 Dify API）。** + +```python +@dataclass(frozen=True, slots=True) +class PublicationWork: + organization_ref: OrganizationRef + source_ref: ContextSourceRef + source_version_ref: SourceVersionRef + job_ref: FileImportJobRef + acquisition_ref: AcquisitionRef + expected_resource_ref: ContextResourceRef | None + expected_previous_revision_ref: ContextRevisionRef | None + content_identity_digest: Digest + +class SupplyPublication: + def accept(change_page: AcceptedChangePage, actor: TrustedControlCall) -> AcceptanceReceipt: ... + def prepare(work: PublicationWork, compiled: CompiledRevision, lease: WorkerLease) -> PreparedReceipt: ... + def mark_indexed(work: PublicationWork, artifacts: IndexArtifactSet, lease: WorkerLease) -> ReadyReceipt: ... + def activate(work: PublicationWork, lease: WorkerLease) -> ActivationOutcome: ... + def status(call: TrustedSourceStatusCall) -> SourceOperationalStatus: ... + +ActivationOutcome = Activated | Unchanged | LeaseRejected | AuthorityChanged | Conflict | RetryableUnavailable +``` + +`CompiledRevision` 和 `IndexArtifactSet` 必须是 versioned、immutable、无 ORM/session/callback 的 domain DTO；只有 Supply module 能构造 publication transaction。`status` DTO 无 Runtime capability。 + +**测试 oracle。** + +- 在 accept commit 后、dispatch 前 kill；scheduler 最终 claim 同一 job且只产生一个 target Revision。 +- 在 `acquired`、`prepared`、`ready` 后逐点 kill；reclaim generation `n+1` 完成，同一 checkpoint不回退，generation `n` 重放 effect=0。 +- index/embedding 第 N 个 Fragment 失败：active pointer 仍指旧版；候选、Fragment 与旧 Package无混合。 +- activation commit 前/后 kill：读者观测集合只能是 all-old 或 all-new，不能出现 hybrid；publish watermark只在 visibility commit 后跨过该 sequence。 +- 两个相同与两个不同 replacement 竞争：相同产生一个 active + 一个 unchanged；不同只有 CAS winner，loser 不覆盖。 +- Source disable 与 activation/lease redemption 竞争：线性化后若 disable 先提交则 publication=0；若 activation先提交，随后的 disable 仍使 Runtime不可见并推进 epoch。 +- refusal status 只含 closed category；fixture 的 source substring、exception、Fragment id、credential在 DB row、JSON、log capture 中出现次数为 0。 +- acquisition checkpoint可领先 publish watermark；后续 sequence completion不能越过前序 gap；recovery闭合 gap后 contiguous watermark推进。 + +**验证命令。** 纯状态/DTO/治理：`make lint && make typecheck && make test && make catalog`。publication、WorkerLease、RLS、crash-boundary和 old-or-new 证据：先 `make db-up`，再 `make integration && make security-gate`，最后 `make db-down`；本报告任务未执行这些数据库命令。 + +**工作量与依赖。** 6–9 engineer-days。依赖现有 ADR-0040/0041/0059/0060 实现、PostgreSQL publication barrier、File import job/outbox ownership、ADR-0072 status DTO；若新增通用非 File outbox，另加 3–5 天并先写 ADR，不能把 File carrier泛化成已激活 ProviderPort。 + +### 3.2 Dataset retrieval orchestration → authorized planner + +**上游路径与观察。** 单库模式先为 available datasets 生成工具描述，由 LLM router选择一个 id，再按该库 retrieval config执行（[`dataset_retrieval.py#L602-L738`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L602-L738)）。多库模式并行每个 dataset / text-or-attachment branch，检查 indexing technique/embedding model compatibility，最后进行跨库 rerank或 score排序（[`#L740-L887`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L740-L887)、[`#L1798-L1909`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L1798-L1909)）。单 dataset 内 hybrid 同时跑 keyword/full-text/vector，dedupe 后 weighted/model rerank（[`retrieval_service.py#L796-L931`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/retrieval_service.py#L796-L931)）。 + +**必须明确的 hydration anti-pattern。** Dify vector/search 结果携带 `page_content`；formatter 随后读取 `DocumentSegment`、`ChildChunk.content`、summary、attachments并构造内容对象（[`retrieval_service.py#L488-L791`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/retrieval_service.py#L488-L791)）。普通 app 路径再把这些正文排序后以 newline concat 返回（[`dataset_retrieval.py#L474-L600`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L474-L600)）。固定 inner service 甚至明确只验证 tenant ownership、不检查 user-level dataset permission（[`knowledge_retrieval_inner_service.py#L1-L12`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/services/knowledge_retrieval_inner_service.py#L1-L12)）。ContextEngine 必须把“tenant/dataset ownership = audience authorization”和“hit 后可直接 hydrate”作为 must-kill premises。 + +**本仓 seam / ADR。** ADR-0012/0024/0025（sealed projection + EffectiveScope + same-transaction projection）、ADR-0067/0068（content-free vector discovery）、ADR-0076（rank evidence在授权后 exact rejoin）、ADR-0077（Article authorization atom）、ADR-0081（data-only discovery session）、Implementation Design §3 Runtime order / §8 retrieval。 + +**Room-A 行为规格。** planner严格分成 plan、discover、authorize、rank/assemble 四层： + +1. `AuthorizedSourceCapabilitySet` 由当前 `UserActor` transaction 内的 mandatory EffectiveScope、active SourceVersion、ReleaseManifest/RuntimeProfile、Source capability declaration 和 optional RequestNarrowing交集产生。它只说明“哪些 source/ref + retrieval mode 可以被尝试”，不是 Resource grant。caller 不能直接传 dataset/source set；RequestNarrowing只能删除已有 ref/mode。 +2. `QueryPlanner.plan` 输入 ContextNeed、上述 capability set、ReleaseObservation 和 effective PackageBudget，输出 frozen `RetrievalPlan`。source descriptor仅含 opaque source ref、closed resource kinds、mode、language/embedding/profile compatibility和content-free operator label；不得把 source description正文、title/path、ACL或credential交给 router/model。 +3. `ROUTE_ONE` 模式只允许从 capability set 选零或一个 source。router返回未知/越界 source、无结果或模型不可用时，分别映射 `InvalidPlan`、合法 empty plan或 profile-declared unavailable；绝不能扩大为全部 source。router model调用必须经 governed model-inference port、EgressGrant与 shared PackageBudgetMeter。 +4. `FAN_OUT` 模式按 plan固定的 `max_sources × per_source_k × max_rankers` 有界并发。每个 source只能使用声明 capability；不支持的 mode在 candidate I/O 前返回 `UnsupportedCapability`。V0 对任何 required branch failure采用 fail-closed `RetryableUnavailable`，不静默降级到 weaker source/ACL/mode；将来若允许 optional ranker，必须由 versioned profile预声明且在 Package记录 gap。 +5. candidate discovery query由 CandidateIndex准备、Runtime在 retained UserActor transaction执行；replaceable index只收到 `CandidateDiscoverySession` 的 primitive results。每个 hit是 `CandidateRef + RankEvidence`，二者均无 body/title/path/source metadata。lexical/vector/hybrid fusion（建议 deterministic RRF）只操作这些值。 +6. Runtime以 deterministic canonical CandidateRef order调用 sealed `AuthorizationKernel`；Kernel看不到 rank。每个 ref验证 Organization、active Source/Resource/Revision、ArticleAccessPolicy、SourceAclEvidence、Membership/Agent/purpose/audience/field ceiling并只对成功项构造 `AuthorizedProjection`。missing/denied/cross-org均进入同一个 empty-compatible路径；denied detail不进 ContextRun。 +7. `AuthorizedRanker` 按 exact CandidateRef 把 rank evidence rejoin到成功 projection；denied rank立即丢弃。weighted fusion仍可使用 authorized rank；任何看正文的 reranker、dedupe、tokenizer、summary/parent/neighbor hydration、selection和assembler只接受 `AuthorizedProjection`。 +8. same-Article/current-Revision expansion可在核验 lineage后继承已作出的 Article decision；跨 Article expansion必须回到新的 CandidateRef并重新走 Kernel。任何 summary/attachment/child/parent只要触及另一 Article都不得继承 first hit。 +9. assembly按 effective PackageBudget确定性选择；final Policy Epoch和egress veto在 delivery 前重验。输出只可能是 authorized `ContextPackage`、canonical empty Package、closed unavailable/refusal；永不输出 context string或 raw ranked documents。 + +**ContextEngine 接口形状草图。** + +```python +@dataclass(frozen=True, slots=True) +class AuthorizedSourceCapability: + source_ref: ContextSourceRef + source_version_ref: SourceVersionRef + allowed_modes: frozenset[DiscoveryMode] + allowed_resource_kinds: frozenset[ResourceKind] + index_profile_ref: IndexProfileRef + projection_ceiling: ProjectionCeiling + +class QueryPlanner(Protocol): + def plan( + self, + need: ContextNeed, + capabilities: tuple[AuthorizedSourceCapability, ...], + release: ReleaseObservation, + budget: PackageBudget, + ) -> RetrievalPlan: ... + +class CandidateIndex(Protocol): + def prepare(self, plan: RetrievalPlan, scope: ContentFreeDiscoveryScope) -> PreparedDiscovery: ... + def shape(self, result: CandidateDiscoverySession) -> tuple[RankedCandidateList, ...]: ... + +class AuthorizedRanker(Protocol): + def rank( + self, + query: AuthorizedRelevanceQuery, + items: tuple[AuthorizedProjectionWithRank, ...], + meter: PackageBudgetMeter, + ) -> tuple[AuthorizedProjection, ...]: ... +``` + +`AuthorizedRelevanceQuery` 是经 retention/egress policy允许的 request-scoped值；它不允许 ranker恢复数据库/session/locator。`RankedCandidateList` 的 type constructor拒绝任何 `content/title/path/metadata` 字段。 + +**测试 oracle。** + +- capability set `{S1}` + caller narrowing `{S1,S2}`：plan仍只能含 `S1`；未知/empty/malformed router output不会触发 S2 call。 +- malicious CandidateIndex尝试通过对象图获取 projection session/connection/locator：静态 capability-graph test和运行时 probe都不可达。 +- 每个 ranker混入 same-org denied、cross-org、stale revision、tombstoned resource；正文进入 pre-Kernel fusion可为 0（它本就无正文），进入 rerank/tokenizer/assembler/model的 denied bytes严格为 0。 +- rank evidence map含 denied candidate；authorized output排序只由 admitted candidate的 exact join决定，删除 denied ref不改变 delivered order/shape。 +- hybrid三个 branch乱序完成：相同 inputs/profile产生 byte-stable fused order；相同 rank/tie以 opaque canonical ref稳定打破。 +- required branch exception、timeout、unsupported、live ACL unavailable各映射 closed unavailable，且不会退回 Weak；零 hit则为 canonical empty Package，不伪装 unavailable。 +- parent/child fixture：same Article/current Revision expansion成功；stale Revision或cross Article必须重授权；cross Article denied内容进入 consumer=0。 +- final epoch在 authorization 与 delivery间推进：Package不交付，ContextRun不宣称 delivered；已消耗的模型/provider budget按失败 lineage安全记录，不含 denied内容。 +- top-k/score不产生 protected-object enumeration：unknown、denied、cross-org probes经允许归一化字段后 Package相等；不声明 timing equivalence。 + +**验证命令。** `make lint && make typecheck && make test && make catalog`；real pgvector/FORCE-RLS/cross-org/epoch/hydration证据需 `make db-up && make integration && make security-gate`（结束 `make db-down`）；HTTP/generated SDK最高 seam另跑 `make smoke && make openapi-check && make sdk-check && make sdk-test`。 + +**工作量与依赖。** 12–18 engineer-days（ROUTE_ONE 3–4，bounded FAN_OUT + deterministic RRF 4–5，authorized rank/rerank 3–5，failure/budget/trace gates 2–4）。依赖 ADR-0081 data-only CandidateIndex、ADR-0076 rank rejoin、complete Article policy/field projection、shared PackageBudgetMeter、frozen eval slices；real model rerank carrier保持 `NOT_ACTIVE` 直到 ablation与egress证据独立通过。 + +### 3.3 Factory / adapter 轴 → typed Provider contract 与 conformance suite + +**上游路径与观察。** Dify 将 paragraph/QA/parent-child processors经 factory选择（[`index_processor_factory.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/index_processor/index_processor_factory.py#L10-L29)），`BaseIndexProcessor` 同时拥有 extract/transform/summary/load/clean/index/preview（[`index_processor_base.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/index_processor/index_processor_base.py#L46-L108)）。vector backend 以同一 abstract create/add/search/delete面和 entry points隔离（[`vector_base.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/vdb/vector_base.py#L18-L76)、[`vector_backend_registry.py`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/datasource/vdb/vector_backend_registry.py#L39-L87)）。Datasource manager和 trace config map分别选择插件 runtime与 trace provider。可学习的是“变化轴有独立入口、unsupported显式”；不可继承的是 broad ORM/session接口和“方法同名即语义等价”。 + +**本仓 seam / ADR。** Implementation Design §3.3 `ContextProvider` typed interface；ADR-0075 connector-runner + three keystone seams；ADR-0078 将 contract ownership 放回 Supply execution seam并要求每 connector自带 deterministic twin；ADR-0081固定 CandidateIndex data-only能力；V1 vector implementation固定 PostgreSQL，不建 backend marketplace。 + +**Room-A 行为规格。** 将变化轴拆成三个互不相认的端口： + +1. **Pure compiler axis：** `DocumentCompiler.compile(CanonicalSourceBytes, CompilationProfile) -> CompiledRevision | CompilationFailure`。无网络、DB、clock、Organization lookup、credential或publication authority；同 bytes/profile输出完全相同的 typed structures、source spans和digests。 +2. **Connector-runner axis：** runner接收 exact WorkerLease-bound `ConnectorJobEnvelope`，调用一个 connector读取 source，输出 bounded `ChangePage[SupplyDocumentEnvelope, DeleteObservation, AclObservation, OpaqueCheckpointProposal]`，独立持久化为 0。checkpoint仅是 proposal；engine全页 durable accept 后才签发 continuation。connector不得发布 Revision、推进 watermarks、tombstone Resource或构造 Runtime authority。 +3. **Runtime discovery/projection axis：** Materialized File走 native PostgreSQL candidate discovery；未来 live/federated provider使用 `describeCapabilities/discover/authorizeAndProject` 的 closed outcomes。candidate discovery返回 content-free CandidateRef；projection evidence交给 Kernel，不让 adapter直接构造 AuthorizedProjection。V1不抽象多个 vector store；PG FTS/pgvector是一个固定实现，filter/delete parity问题不被 factory掩盖。 + +`ProviderOutcome` 只能是 `Ok | Unsupported | RetryableUnavailable | InvalidCheckpoint | GenericDenied`。Unsupported/denied/unavailable不能变成 empty success；SourceAclEvidence mode是 SourceVersion声明，Live/Mirrored失败不可退成 Weak。 + +**ContextEngine 接口形状草图。** + +```python +class DocumentCompiler(Protocol): + def compile(self, source: CanonicalSourceBytes, profile: CompilationProfile) -> CompilationOutcome: ... + +class ConnectorRunner(Protocol): + def execute(self, job: ConnectorJobEnvelope, lease: WorkerLease) -> ConnectorRunOutcome: ... + +class ContextProvider(Protocol): + def describe_capabilities(self, source: ContextSourceRef) -> ProviderOutcome[CapabilityDeclaration]: ... + def read_changes( + self, source: ContextSourceRef, cursor: ChangeCursor | InitialScan, limit: ChangeLimit + ) -> ProviderOutcome[ChangePage]: ... + def discover( + self, ticket: ContextAccessTicket, plan: RetrievalPlan, limit: CandidateLimit + ) -> ProviderOutcome[CandidatePage]: ... + def authorize_and_project( + self, ticket: ContextAccessTicket, refs: tuple[CandidateRef, ...], ceiling: ProjectionCeiling + ) -> ProviderOutcome[SourceProjectionBatch]: ... +``` + +DTO全部 `frozen + slots + closed enum/discriminated union`，只携带 domain refs/values；禁止 `Session`、ORM row、untyped dict、callback、raw credential和ambient tenant。connector registry只从 server-owned SourceVersion解析实现，不接受 caller plugin path。 + +**每 connector 必过的 conformance suite。** + +| Suite | 硬 oracle | +|---|---| +| Capability honesty | 未声明 operation返回 `Unsupported` 且 provider/source I/O计数符合声明；声明的 ACL mode、batch/field/cursor limits与实际相同。 | +| Identity/tenant binding | 缺 Organization/SourceVersion、wrong source/job/workload/operation、expired/replayed lease的 source call或durable effect为 0；无 default tenant/schema。 | +| Cursor/checkpoint | 同 cursor同 fixture得到同 ordered page；未全页 durable accept不发 next cursor；old cursor不后退；SourceVersion变化令旧 cursor `InvalidCheckpoint`。 | +| DTO/provenance | 同 source bytes/metadata/profile产生相同 canonical path、Resource key、content digest、ACL observation、delete observation和ordered envelope；不含 credential/provider internal object。 | +| ACL semantics | Live同请求检查或声明 verify-before/after；Mirrored带 exact aclAsOf/version/freshness；Weak仅在源确无更强语义；Live/Mirrored outage永不回退。 | +| Projection consistency | discovery/projection的 SourceConsistencyRef exact match；missing/mixed/stale/changed ref被Kernel拒绝；batch部分 denial不泄露denied object detail。 | +| Delete semantics | connector只产 content-free delete observation；接受 observation不会 tombstone/advance epoch/cleanup，唯一 Control tombstone authority另行重验 current scan。 | +| Retry/replay | page、job和provider response重放不重复 Resource/Revision/effect；reclaim generation替换旧 generation；ambiguous external read映射 closed retryable，不能伪造 empty。 | +| Boundedness | path count、page size、bytes、fields、depth、wall time都由 server profile限制；越界 all-or-nothing closed refusal，无 partial publish。 | +| Twin/live tiers | deterministic twin跑完全套但只得 contract-verified；sandbox/live同一 suite通过后才分别升 tier，不能由 mock数量替代。 | +| Capability graph | connector/compiler/CandidateIndex对象图均不能到达publication pointer、AuthorizationKernel constructor、projection session、ActionPlane或release promotion。 | +| Trace redaction | provider error、credential、source body、denied ID在ordinary logs/metrics/ContextRun=0；只保留closed category/digest与authorized lineage。 | + +**测试 oracle。** 除表中套件外，做 metamorphic tests：重排 provider page input仍按 canonical order输出；同名 backend的 `delete_by_metadata_field` 不算 conformance，必须对“删除后Runtime立即不可见”证明由 tombstone/epoch而非index delete完成；filter支持必须用 cross-org混入与 underfilled ANN fixtures证明候选仍过Kernel，不能以 adapter self-report验收。 + +**验证命令。** DTO/compiler/registry/twin：`make lint && make typecheck && make test && make catalog`。third-party governance（若实际复制任何 permissive subtree）：`make third-party-check && make third-party-artifacts`。真实 connector sandbox另有其 owning make target后才能升 tier；当前数据库 seam用 `make db-up && make integration && make security-gate`，结束 `make db-down`。 + +**工作量与依赖。** shared contract/conformance harness 8–12 engineer-days；每个新 connector的 twin + mapping + failure matrix另计 5–10天（不含 source API本身）。依赖 ADR-0075 runner serialization、ADR-0078 per-connector ownership、WorkerLease/ChangePage DTO、SourceVersion CapabilityDeclaration和credential broker。不要创建已被 ADR-0078 dissolved 的 `contract_kit/`。 + +### 3.4 Workflow / REST / SDK / MCP → 一个 sealed Runtime 产品对象 + +**上游路径与观察。** Workflow node将 dataset ids、single/multiple config、metadata filters与query/attachments组合成 retrieval request，输出含 dataset/document/segment名字、score、hit count、hash、child content、raw doc metadata的 `Source[]`（[`knowledge_retrieval_node.py#L184-L294`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/workflow/nodes/knowledge_retrieval/knowledge_retrieval_node.py#L184-L294)、[`retrieval.py#L12-L82`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/workflow/nodes/knowledge_retrieval/retrieval.py#L12-L82)）。Service API同时暴露 hit-test/retrieve；SDK把 KnowledgeBase CRUD做成产品面；MCP以 App server/JSON-RPC提供另一入口（[`mcp.py#L44-L96`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/controllers/mcp/mcp.py#L44-L96)）。可学习的是“retrieval config/node/run是用户可见产品对象”；不可继承多套auth/输出语义。 + +**本仓 seam / ADR。** ADR-0017 closed access set；ADR-0047唯一公开 `POST /v0/resolve` + hidden bridge同handler；ADR-0048 generated SDK facade；ADR-0031 ContextRun；engine output固定 ContextPackage；MCP目前 `NOT_ACTIVE`。 + +**Room-A 行为规格。** 产品对象分三层，只有一条 domain执行： + +1. **Configuration object。** AgentVersion/RuntimeProfile持有 server-validated retrieval strategy、source delegation ceiling、profile refs和PackageBudget ceiling。它可被Control/Release UI展示和version，但不是Principal或grant；发布仅由ContextLearning promote active ReleaseManifest。 +2. **Invocation object。** 所有 activated ingress只构造 `AuthenticatedInvocation + TrustedDeliveryContext + ResolveWire`。body是 closed `Acquire | Continue | OpenCitation`：Acquire只含need、optional smaller budget、optional narrowing；Organization/Principal/Membership/purpose/audience/ACL/source mode不得在body。远程Bot只携带authenticated metadata中的opaque `DeliveryEvidenceRef`。 +3. **Result object。** 唯一内容输出是 `ResolutionOutcome.Resolved(ContextPackage)`；Package含opaque package/run/decision refs、audience digest、policy snapshot/epoch、release/profile/tokenizer/schema lineage、asOf/expiry、budget usage、authorized Blocks ↔ Evidence一一闭合、citations/coverage/gaps和package digest。它不返回raw dataset/source/document names、pre-auth score/hit count、denied details或plain context string。ContextRun在同UserActor transaction、response前以digest-only授权 lineage提交。 + +HTTP `/v0/resolve`、generated TS SDK、loopback consumer、BotDelivery和未来 MCP都调用同一个 `ContextRuntime.resolve` composition。Control hit-test若存在也必须用HTTP/generated SDK最高public seam获取Package，不能直接调CandidateIndex/formatter。MCP只有在真实caller、trusted-context construction、Package/error parity、unknown/denied convergence、metadata-injection negative tests完成后激活；在此前 server能力报告和状态继续为 `NOT_ACTIVE`。 + +**ContextEngine 接口形状草图。** + +```python +class ContextRuntime: + def resolve( + self, + invocation: AuthenticatedInvocation, + delivery: TrustedDeliveryContext, + request: Acquire | Continue | OpenCitation, + ) -> ResolutionOutcome: ... + +# generated semantic types come from frozen OpenAPI; facade does not handwrite ResolveWire +export interface ContextEngineClient { + resolve(args: { + requestId: string + deliveryEvidenceRef?: string + request: ResolveWire + }): Promise +} +``` + +SDK facade constructor只接受base URL与transport authentication；不导出generated implementation subpath、generic fetch client、arbitrary headers、raw request method或handwritten Package schema。Future MCP adapter只做 protocol decode/encode + trusted ingress redemption，然后调用同一 `resolve`；不得生成MCP特有授权或source-rich result。 + +**失败模式。** malformed/unknown union/duplicate singleton headers在trusted context前以frozen generic wire error失败且content I/O=0；auth/evidence redemption失败generic且I/O=0；known but inactive capability在candidate/source I/O前closed unavailable；policy denied/missing/cross-org candidate收敛为canonical empty Package；ContextRun commit失败则不返回成功/decisionRef；SDK transport failure与closed HTTP domain outcome是不同union；MCP notification/protocol错误未来不得绕过Runtime。 + +**测试 oracle。** + +- server OpenAPI只有一个 public resolve operation；hidden bridge与v0 handler/composition对象identity相同，不能注入第二Kernel。 +- HTTP、installed npm tarball client、loopback consumer对同seeded Acquire返回相同Package security fields/digest；transport失败与domain unavailable可区分。 +- compile-negative：body加organization/principal/purpose/audience/ACL/unknown variant失败；SDK加raw header、deep import generated client、generic request method编译失败。 +- valid empty/authorized Package在response前都有同org durable ContextRun；commit fault返回generic unavailable且没有optimistic decisionRef。 +- workflow/Bot consumer只能导入SDK package root；repo scan禁止engine internal/CandidateIndex/ORM imports；每个Block保留exact Evidence ref。 +- future MCP parity gate逐字段比较Package/error；wrong service/destination/request DeliveryEvidenceRef与caller-authoredtrusted fields均在content I/O前失败。 + +**验证命令。** `make openapi-check && make openapi-breaking-check && make sdk-check && make sdk-build && make sdk-test && make sdk-pack && make smoke`；consumer/Bot：`make bot-build && make bot-test && make ui-build && make ui-test`；真实authorized wire需 `make db-up && make integration && make security-gate` 后 `make db-down`。 + +**工作量与依赖。** 当前HTTP/OpenAPI/SDK已激活，补 retrieval product object与consumer parity约4–6 engineer-days；MCP不计入当前实现，未来activation 8–12天并需独立ADR/threat-model/evidence。依赖 production authentication另行激活、DeliveryEvidenceRef redemption、frozen Package schema、ContextRun commit-before-response和PackageBudget。 + +### 3.5 MIT SDK 子树 → 不复制，守住 generated client + +**上游路径与观察。** 许可证据见第1节。产品形状上，Node client手写routes/types，root导出generic `HttpClient`和base `sendRequest`；KnowledgeBase client把Dataset/Document/Segment CRUD与retrieval status捆成宽SDK。固定 commit 的PHP子树许可边界未充分核验。以上不形成可复制架构价值。 + +**本仓 seam / ADR。** ADR-0047 immutable OpenAPI v0；ADR-0048 pinned generator + closed facade + generated-tree digest + package consumer test；ADR-0074 third-party registration仅在实际copy时适用。 + +**Room-A 行为规格。** Room-B不实现Dify SDK，不创建`third_party/dify`。保持以下 generation contract： + +1. `openapi/v0/openapi.json` 是historical immutable semantic source，已有目录不能覆盖；server schema与snapshot structural equality/checksum必须相等。 +2. pinned Node/npm/generator/TypeScript versions从clean temp dir生成；file set、每字节、tree digest和bundled OpenAPI checksum任一漂移即失败。 +3. public package export map只开放facade与checksum；generated implementation不可deep import。 +4. facade只接收base URL、transport auth、request id、optional opaque DeliveryEvidenceRef和generated `ResolveWire`；任意header/endpoint/request-option通道不存在。 +5. npm tarball在clean temp consumer安装、compile、调用real local HTTP；tarball包含所需contract/license/SBOM材料且cache不入包。 +6. Continue/OpenCitation可被generated类型表达但未激活carrier时只返回closed unavailable；codegen存在不等于capability active。 + +**接口形状草图。** 延续3.4的 `ContextEngineClient.resolve`；语义类型必须 `export type { ResolveWire, ResolutionOutcome, ContextPackage } from ` 的受控再导出，facade不得重新声明字段。 + +**测试 oracle。** 修改snapshot/server/generated任一侧、generator version、export map、tree digest、checksum都使对应gate红；tarball consumer无法import internal generated path或构造trusted body/header；packed client对authorized seeded Package recompute digest成功；`npm pack --dry-run` file list不含credential/cache/source map（除非明确批准）且包含contract/license notices。 + +**验证命令。** `make openapi-check && make openapi-breaking-check && make sdk-generate && make sdk-check && make sdk-build && make sdk-test && make sdk-pack`。若未来真的copy，额外 `make third-party-check && make third-party-artifacts && make build`；当前结论none，因此不要创建登记。 + +**工作量与依赖。** 0 engineer-days copy工作；1–2天用于将本报告的negative oracle补进现有SDK tests（若当前未覆盖generic header/client export）。依赖ADR-0048与pinned toolchain；不依赖Dify代码或package。 + +### 3.6 Trace / observability → authorized-only digest lineage + +**上游路径与观察。** Dify retrieval完成后可更新hit count并将完整`Document`交给trace task（[`dataset_retrieval.py#L889-L1009`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L889-L1009)）；query audit在独立事务中保存raw query/attachment id（[`#L1029-L1082`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/rag/retrieval/dataset_retrieval.py#L1029-L1082)）。trace manager将inputs、full documents、tenant/app/user/display names、embedding/rerank model写入trace info（[`ops_trace_manager.py#L1094-L1180`](https://github.com/langgenius/dify/blob/120c38bad8d27cbe1e6a1d5522fd66f5caf6d0d5/api/core/ops/ops_trace_manager.py#L1094-L1180)），再由provider map适配多种trace backend。它证明operator想看latency/model/profile/result的产品价值，但其payload不适合ContextEngine。 + +**本仓 seam / ADR。** ADR-0031 authorized-only ContextRun + seven-field restricted DecisionAudit；ADR-0076 refused rank evidence不外露；Implementation Design §7.3；ContextRun query只存organization-bound keyed digest，Package retention固定digest-only。 + +**Room-A 行为规格。** 观测拆为三个不同可见面： + +1. **Tenant-visible / Learning-safe ContextRun（只在成功形成并交付Package时写）。** 可存：opaque run/decision/auth/request refs；trusted purpose与audience digest；PolicySnapshot ref/epoch/asOf；effective-scope digest；active ReleaseManifest generation/digest和Content/Index/Runtime/Curation/tokenizer/Package schema refs；effective/used PackageBudget；stage timing/call/cost aggregate；terminal `delivered_authorized | delivered_empty`；selected authorized Evidence refs；Package digest/retention mode；accepted/finalized/expiry。raw query只存versioned Organization-bound HMAC digest和key version。 +2. **Restricted DecisionAudit。** 对 `delivered_empty` 当前只存 Organization/run/decision、PolicySnapshot/epoch、closed `no_authorized_evidence`、recorded_at；不存query/digest、Candidate/Fragment/Resource ref/body/name/score、denial reason或candidate/denied count。其他pre-auth transport失败不伪造ContextRun。 +3. **Ordinary operator metrics/logs。** 只允许content-free aggregate：resolve outcome class、stage latency buckets、budget usage、provider call outcome category、lease/retry phase、active profile/ref digest、source status watermarks/refusal category。labels不能含Organization/user/source/resource/fragment raw id、query、title/path/body、credential、opaque token或high-cardinality denied detail。跨Organization aggregate必须经过另一个明确privacy/retention decision，当前不默认激活。 + +**operator-visible surviving signals。** `runRef/decisionRef`（需exact authorized read seam）、time range/latency、delivered authorized或canonical empty、authorized Evidence count/refs、Package digest、budget effective/used、release/profile/tokenizer lineage、policy epoch/snapshot、audience digest、closed provider/Runtime outcome、source acquisition/publish watermarks和closed current compilation refusal。**不 survive**：raw query、prompt、Package/body副本、pre-auth scores/order、denied/missing object identity/count/reason差异、source/document display name、workspace/user name、credential/model secret、DeliveryEvidenceRef/ContextAccessTicket/ActionTicket/WorkerLease bearer。 + +**接口形状草图。** + +```python +@dataclass(frozen=True, slots=True) +class AuthorizedRunObservation: + run_ref: ContextRunRef + decision_ref: DecisionRef + terminal_outcome: Literal["delivered_authorized", "delivered_empty"] + policy_snapshot_ref: PolicySnapshotRef + policy_epoch: int + audience_digest: Digest + release: ReleaseObservation + budget: BudgetObservation + stage_metrics: tuple[ContentFreeStageMetric, ...] + authorized_evidence_refs: tuple[EvidenceRef, ...] + package_digest: Digest + query_digest: KeyedDigest + +class ContextRunWriter(Protocol): + def append_before_delivery( + self, observation: AuthorizedRunObservation, tx: CurrentUserActorTransaction + ) -> DurableRunReceipt: ... +``` + +只有sealed Runtime构造该DTO；candidate index、provider、trace adapter不能调用writer。External telemetry exporter若未来存在，只能接受另一个经过allowlist projection的content-free DTO，不能接收ContextRun ORM row或Package。 + +**测试 oracle。** + +- canary strings放入query、authorized body、denied body、path/title、credential/token；ContextRun只命中query HMAC与authorized Evidence refs，ordinary log/metric/trace sink命中正文/secret=0，DecisionAudit全部canary=0。 +- same-org denied/cross-org/missing candidate三种empty运行的public Package和restricted audit category相同（只归一化server refs/times/digest）；不做timing equivalence宣称。 +- authorized run的Evidence refs与Package Blocks一一闭合；Package digest可重算；digest-only retention中不存在Package JSON/body副本。 +- query相同但Organization不同，HMAC digest不同；key version rotation改变comparison domain；无key/default secret时fail closed。 +- ContextRun insert/commit fault：HTTP成功响应=0；unauthenticated/malformed request ContextRun=0。 +- refused candidate的rank/score加入hostile fixture后，tenant-visible/operator output中该rank/score=0；authorized排序仍正确。 +- metric cardinality test拒绝raw ids/query/path作为labels；provider exporter异常不影响authorization，也不能导致raw fallback logging。 + +**验证命令。** `make lint && make typecheck && make test && make catalog && make smoke`；真实RLS、commit-before-response、redaction和security operator exact-read需 `make db-up && make integration && make security-gate`，结束 `make db-down`。如新增telemetry artifact，必须将它纳入catalog与secret/redaction tests后再声明active。 + +**工作量与依赖。** 5–8 engineer-days（safe stage metrics 2–3，export projection/cardinality guard 1–2，redaction/adversarial suite 2–3）。依赖现有ADR-0031 schema/writer、retained UserActor transaction、Package digest、release observation、PackageBudgetMeter；production operator identity、retention/export/delete policy仍是独立前置，不可由trace provider配置代替。 + +## 4. 不可借鉴清单与必须杀死的隐含前提 + +| 学习行为 | ContextEngine 独立实现 | 必须杀死的隐含前提 | +|---|---|---| +| 可观察 parsing/splitting/indexing phase | Revision三态 + job checkpoint/events + dual watermarks + content-free Control status | 一个mutable Document status同时能表达execution、visibility、retry和authorization。 | +| commit后异步dispatch | acceptance/outbox/job同事务，scheduler从durable state claim | DB commit后直接queue dispatch永不丢；broker ack等于business commit。 | +| retry前clean旧segments/vector | build immutable new Revision，旧active保留到CAS activation；cleanup异步 | 重建中短暂无正文/混合索引对在线读者可接受。 | +| exception text写Document.error | closed refusal category + restricted diagnostics另行决策 | parser/provider异常适合tenant/operator普遍展示或长期保留。 | +| Dataset tenant ownership过滤 | current Membership + Agent ceiling + ArticleAccessPolicy + SourceAclEvidence + purpose + audience + field projection交集 | tenant/dataset同属天然等于每个Principal/audience可读。 | +| LLM router选择dataset | router只在server-derived AuthorizedSourceCapabilitySet内收窄 | model输出是可信source authority；未知选择可回退到全库。 | +| metadata filter先缩小documents | optional request narrowing只缩小EffectiveScope；exact auth仍逐Article执行 | filter命中/索引ACL足以授权，missing filter可视为unrestricted。 | +| keyword/vector/full-text并行 | content-free CandidateRef + RankEvidence，deterministic RRF | 检索返回的page_content/title/path可在授权前供fusion/debug。 | +| score threshold / top-k | rank与auth隔离，授权后exact rejoin和budget selection | 分数高可抵消授权失败；top-k之外无需考虑side channel或underfill。 | +| retrieval formatter水合Segment/Child/Summary | CandidateRef先过Kernel；same Article/current Revision lineage核验，cross Article重授权 | 首个child hit的许可自动覆盖parent、neighbor、attachment、summary或另一Article。 | +| rerank/fusion读取Document body | pre-Kernel仅ref fusion；正文rerank只接AuthorizedProjection | 相关性模型是“内部服务”所以可以先看denied bytes。 | +| newline concat context | expiring、audience-bound、budgeted ContextPackage + Evidence closure | 裸context string足以表达授权、provenance、TTL、revocation和citation。 | +| Workflow输出dataset/document/segment metadata | Package只给opaque、安全必需、authorized lineage；ContextRun digest-only | source metadata越丰富越可观测，不会泄露existence、names、ranks或ACL。 | +| REST hit-test另调retriever | Control/UI也通过唯一public resolve seam | “测试”路径可以弱化auth、budget、audit或输出更多denied详情。 | +| 每transport单独封装retrieval | HTTP/generated SDK/future MCP都映射同一sealed Runtime | SDK或MCP是新的domain implementation，可各自filter/auth。 | +| caller body携带tenant/user/dataset | ingress构造AuthenticatedInvocation/TrustedDeliveryContext；body closed | trusted identity/audience/purpose可以由caller自报并在service层校验。 | +| MCP App server直接激活 | 保持NOT_ACTIVE直到真实caller与完整parity/security gate | protocol支持存在即等于安全产品能力存在。 | +| broad BaseIndexProcessor | pure compiler + runner + publication分别typed | extract/transform/load/clean/index同一base class仍是deep module，session/ORM不会泄漏authority。 | +| vector backend同名CRUD/search | V1固定PG；每个真实adapter跑semantic conformance | 同名`filter/delete/search`具有相同tenant、atomicity、score、consistency和failure semantics。 | +| provider返回empty表示各种失败 | closed ProviderOutcome；Unsupported/Unavailable/Denied不变empty success | 空列表可安全吞掉credential、ACL、checkpoint、backend故障。 | +| connector返回checkpoint | engine整页durable accept后才签发continuation | provider观察/返回cursor等于engine已接受或内容已发布。 | +| connector delete | content-free observation → sole Control tombstone authority重验 | source adapter/index delete本身能决定Runtime不可见与epoch。 | +| trace记录raw query/full documents | Org-bound keyed query digest + authorized Evidence refs + digest-only Package | observability天然是受信面，保存输入/输出不会成为第二内容库。 | +| trace记录dataset/user/workspace/model字段 | allowlisted content-free metrics与profile digests | display name/raw id是低风险label；credential lookup失败可fallback raw。 | +| hit_count副作用 | 若需要只对authorized selected Evidence、同transaction/明确retention记录aggregate | candidate命中即是可见/可学习事件；异步副事务失败不影响lineage真实性。 | +| root code看似Apache | Dify root永远Room-A clean-room；只看exact separately licensed regions | 去掉附加条件后可当普通Apache复制；产品级license可替代路径级核验。 | +| MIT Node SDK可复制 | 继续OpenAPI-generated closed facade；copy candidate none | permissive许可自动意味着架构适配且维护成本更低。 | +| PHP README写MIT | 保持[未取证]并不复制 | README一句license声明等于exact path license region与完整notice链。 | + +## 5. 推荐实现顺序 + 给 coordinator 的开放问题 + +### 推荐顺序 + +1. **先冻结 oracle 与 capability graph（2–3天）。** 把第3.2的“pre-Kernel正文=0”、第3.3 per-connector conformance、第3.6 redaction canary写入catalog/contract tests；保持所有新carrier `NOT_ACTIVE`。这是Room-B唯一允许的Dify观察输入。 +2. **补齐 Supply publication/status 投影（6–9天）。** 在现有File replacement/recovery/lease基础上确认accept+outbox原子性、closed operational phases和每durable boundary crash tests；不增加Revision状态，不泛化ProviderPort claim。 +3. **固定 planner DTO和AuthorizedSourceCapabilitySet（3–4天）。** 只实现server-derived capability narrowing、ROUTE_ONE deterministic twin、unsupported/failure semantics；router model默认无carrier。 +4. **实现bounded FAN_OUT + content-free RRF（4–5天）。** 使用ADR-0081 data-only session；保留rank evidence但Kernel rank-blind；先用FTS/pgvector现有PG实现，不引入vector factory。 +5. **实现授权后rank/rerank/expansion（5–8天）。** exact rank rejoin、same-Article/current-Revision lineage、cross-Article reauth、PackageBudgetMeter；rerank twin先contract-verified，真实model须ablation/egress后另激活。 +6. **收口产品面（4–6天）。** Workflow/UI/Bot只经installed generated SDK消费Package；补compile-negative/generic-header禁止与transport parity；不做Dify SDK迁移。 +7. **最后补safe observability（5–8天）。** 先ContextRun allowlist/stage aggregate，再external exporter projection；redaction/retention/operator identity未闭合前不输出raw trace。 +8. **MCP继续不排期。** 只有真实caller证明HTTP/SDK不能满足protocol需要时，另立ADR和8–12天activation slice；不得把Dify App MCP结构当捷径。 + +总增量约 **29–43 engineer-days**（不含MCP、live connector、production auth和外部等待）；可按 `Supply status` 与 `Runtime planner` 两条不共享文件的工作流并行，但共同合入前必须跑一次最高public seam + real PostgreSQL security gate。 + +### 给 coordinator 的开放问题 + +1. **Planner的第一个真实pulling workload是什么？** 当前STATUS只激活loopback single-Membership File pgvector Acquire；是否先交付“单Source lexical+vector hybrid”，还是直接要求多Source fan-out？后者会把AuthorizedSourceCapabilitySet、cross-source budget和failure policy一起提前。 +2. **ROUTE_ONE是否允许模型router？** 建议首版只用deterministic rule/twin并把model router `NOT_ACTIVE`；若确有自然语言source routing需求，需要确定ModelGateway/EgressGrant、profile、budget和fallback语义的owning ADR。 +3. **Required branch失败是整次unavailable，还是允许profile-declared partial Package gap？** 本报告默认V0整次fail closed，避免silent quality downgrade。若业务必须partial，需预先冻结哪些branch可选、gap wire字段、Quality/Budget gate和不泄露source existence的归一化。 +4. **Source operational status要不要增加per-job opaque view？** ADR-0072已允许source aggregate与current refused paths；本报告允许terminal phase按opaque job ref读取。若UI只需source-level视图，应删掉per-job surface以减少枚举面。 +5. **Authorized rank/timing保留到什么层级？** 建议ContextRun只存selected authorized refs与stage aggregate，不存逐Evidence raw score；若evaluation要逐项rank，需独立retention/privacy schema，且refused rank永不进入。 +6. **是否要把“Dify SDK copy candidate none”记录为长期决策？** ADR-0048已实质覆盖；建议不新增ADR，只在未来有人提议迁移Dify SDK时以本报告和ADR-0048拒绝，避免为“未做的copy”制造normative artifact。 +7. **PHP subtree是否值得进一步legal取证？** 当前产品无PHP SDK需求，建议保持 `[未取证]` 且不投入；只有出现明确caller后再向upstream/license owner核实exact path grant并固定独立repo/commit。 +8. **公开材料如何引用本报告？** 建议答案是“不引用”。任何可公开的Dify结构主张回到四仓evidence baseline；本报告留在maintainer-local research边界，Room-B提交只引用本仓requirements/ADR/tests，不把Dify当安全guarantee或public provenance。 diff --git a/docs/research/2026-07-31-five-repository-implementation-blueprint.md b/docs/research/2026-07-31-five-repository-implementation-blueprint.md new file mode 100644 index 00000000..b1eb66cc --- /dev/null +++ b/docs/research/2026-07-31-five-repository-implementation-blueprint.md @@ -0,0 +1,159 @@ +# 五仓借鉴总蓝图 — ContextEngine 可复刻实施路线 + +> 日期：2026-07-31 · 作者：coordinator（Claude）融会贯通五份独立 codex worker 评估 +> +> 输入报告（同目录，全部固定 commit 静态取证，未运行上游系统）： +> - [`2026-07-31-ragflow-blueprint-evaluation.md`](./2026-07-31-ragflow-blueprint-evaluation.md)（Apache-2.0，copy+patch 可选区） +> - [`2026-07-31-dify-blueprint-evaluation.md`](./2026-07-31-dify-blueprint-evaluation.md)（Room-A） +> - [`2026-07-31-maxkb-blueprint-evaluation.md`](./2026-07-31-maxkb-blueprint-evaluation.md)（Room-A） +> - [`2026-07-31-onyx-blueprint-evaluation.md`](./2026-07-31-onyx-blueprint-evaluation.md)（MIT 非 ee 区，copy+patch 可选） +> - [`2026-07-31-openviking-blueprint-evaluation.md`](./2026-07-31-openviking-blueprint-evaluation.md)（Room-A，AGPLv3） +> +> **证据纪律**：公开 reference claim 仍只回引 [`2026-07-19 四仓证据基线`](./2026-07-19-four-public-repositories-evidence.md)。Dify / MaxKB / OpenViking 三份是 **Room-A 维护者本地研究**，Room-B 实现者只读其中规格与 oracle、不读上游源码；OpenViking 尚未纳入公开基线。RAGFlow / Onyx 报告中的复制候选在 `third_party/` 注册闭合前同样只是候选。本文不是法律意见，不替代逐项 legal review。 + +## 1. 五仓许可证矩阵与复用模式总账 + +| 仓库 | 固定 commit | 许可证事实 | 复用模式 | 本次 copy+patch 结论 | +|---|---|---|---|---| +| RAGFlow | `4391e03` | 根 Apache-2.0；**模型资产不在 Git 树内，许可未取证** | 源码区 copy+patch 可选；资产门未闭合 | **DOCX + PDF outline 两文件可进入注册流程**；PDF/OCR 九文件 blocked（模型/依赖审计） | +| Onyx | `2fb3dd1` | 非 `ee/` 全 MIT；`ee/` 企业许可 | MIT 区 copy+patch 可选；ee 仅 clean-room | **至多一个极窄 ABC 形状**（`interfaces_new.py`）待决策；推荐原生 Protocol + permalink 等价 | +| Dify | `120c38b` | 根 Apache-2.0 + 多租户附加限制，与本产品目的冲突 | 全部 root 代码 clean-room | **none**（`sdks/nodejs-client` MIT 可核验，但 generated SDK 严格更优） | +| MaxKB | `32b2d88` | GPLv3 §5(c) | 纯 clean-room，零代码/零依赖 | **none** | +| OpenViking | `49b1820` | 主项目 **AGPLv3**（§13 覆盖网络服务）；`crates/ov_cli` MIT↔Apache **冲突未消歧**；examples 逐路径混合（openwebui-plugin 明确 AGPL） | 主项目严格 clean-room | **none**（当前；generated SDK 路径更优，冲突消除前不复制） | + +**已 vendored（不重复提议）**：`third_party/ragflow/deepdoc/parser/markdown_parser.py`（ADR-0079）、`third_party/onyx/connectors/` 四文件（ADR-0075）。 + +**一句话总账**：真正可能进入 `third_party/` 的新增代码只有 RAGFlow 的 `docx_parser.py` + `utils.py`（outline）两个文件，且要先过 format-neutral representation ADR 与依赖许可证门；其余全部价值以 clean-room 行为规格 + 接口形状 + 测试 oracle 的形式吸收。这与 ADR-0074/0075 的"按区域不按产品、Kernel 切割、唯一真相"完全一致。 + +## 2. 能力 → 主参考源归属表（融会贯通版） + +每个 ContextEngine 区域只指定**一个主参考源**避免多源污染，负 oracle（反例）同样有价值。 + +| ContextEngine 区域 | 主参考源 | 辅助输入 | 负 oracle（必须杀死的形状） | +|---|---|---|---| +| Supply 编译：PDF/DOCX/结构保真 | **RAGFlow** deepdoc（copy+patch 候选） | — | loose dict/HTML 输出、parser 成功=发布、cell/word 成 Fragment、runner 联网下模型 | +| Supply 编译：Markdown v1/v2/v3 | 已冻结（ADR-0036/0038/0079） | — | 新 factory 静默重解释旧 Revision | +| Supply 执行/checkpoint/lease 生命周期 | **Onyx** connector framework（已 vendored）+ RAGFlow task_executor 行为 oracle | Dify indexing 状态可见性（Room-A） | Redis unacked=lease、heartbeat 续权、missing-tenant 落默认 schema（Onyx app_base 反例） | +| Supply 状态可见性（运营 UX） | **Dify** indexing lifecycle（Room-A） | MaxKB preview 状态 | mutable Document.status 同时是执行/可见性/授权真相 | +| Candidate discovery：单源 hybrid | **Onyx** lift 2（接口形状；原生 PG FTS+pgvector） | — | Vespa/OpenSearch SQL 移植、`bypass_acl`、index filter 授权 | +| Pre-Kernel fusion/dedupe | **Onyx** lift 1 差分 oracle | RAGFlow weighted_sum 对照 fixture | pre-Kernel 加权（ADR-0083 已禁止）、denied 位置影响合法顺序 | +| Runtime 多源编排（planner） | **Dify** dataset_retrieval（Room-A） | — | LLM router 输出是 source authority、caller 传 dataset set、hydration 前授权 | +| Same-Article expansion | **Onyx** lift 3 行为 oracle | RAGFlow TOC/parent（clean-room） | first-hit 授权继承 parent/neighbor（`search_utils.py:149` 显式反例）、document_id 继承权限 | +| Authorized rerank | **Onyx** lift 5 行为 oracle | RAGFlow rerank（clean-room） | provider wrapper 持 key 直调、float score 授权、`list[str]` 输入 | +| Query rewrite | **Onyx** lift 4 编排形状 | — | 任意 memory/history 进 rewrite、模型删掉 original query | +| PackageBudget | **Onyx** lift 6 行为 oracle | OpenViking 逐 hop 预算 | 固定 75-token 估算、stage-local meter、usage 写 0 | +| Progressive disclosure（密度分层） | **OpenViking** L0/L1/L2（Room-A） | — | depth/tier 产生权限、目录已搜尽声明 | +| Curation 运营 UX：preview/confirm | **MaxKB**（Room-A） | Dify 状态机 | mutable row 同时 source+index+release、临时 file ID=发布权 | +| Curation 知识单元：annotation/snapshot | **MaxKB** Paragraph/Problem/Tag/Termbase（Room-A） | — | mutable Tag row 双真相、LLM enrichment 直入索引、hit_num 回写 | +| Learning：session→候选 | **OpenViking** session extraction（Room-A） | MaxKB feedback improve | 模型 create/merge/delete active memory、从 ContextRun digest 反推正文 | +| Release：evaluate/promote/canary/rollback | 已固定（ADR-0033/0073/0080/0082）；**Onyx** swap_index 行为 oracle | MaxKB Hit Test/direct return | 两次 commit 切换、手测=门禁、quality 抵消 security | +| 交付：HTTP/SDK/MCP 产品面 | **Dify** workflow node 产品化（Room-A） | OpenViking MCP 13 tools 生命周期 | 每 transport 一套 auth、context string concat、宽 MCP 写面（remember/forget） | +| Observability/trajectory | **Dify** trace 变体轴 + **OpenViking** trajectory（二者收敛到同一红线） | — | raw query/full document/denied count 进 trace（Dify ops_trace_manager 反例） | +| Operator console UX | **OpenViking** Studio/Helper 映射到 ADR-0090 七类 job | MaxKB Hit Explorer | 同进程直调 DB、匿名浏览器继承 dogfood principal、score 显示 | +| Eval/parity 方法 | **RAGFlow** comparator 迁移方法（clean-room） | Onyx tests/README 四层 → 本仓 make 分层 | 任意 Python object 差异少=等价、回放生产写返回值 | +| 战略定位 | OpenViking = **产品层相邻竞品、协议层条件互补、安全架构仅战略参考** | — | 把 agent memory FS 当 runtime foundation | + +## 3. 统一实施路线图（Wave 0–6） + +五份报告各自的推荐顺序存在依赖与重叠，合并为七波。**退出条件（exit gate）全部是本仓已注册的 make 证据，不是上游一致性。** 工作量为增量 engineer-days（单人；已扣除三 seam 与两个 vendored subtree 的沉没成本）。 + +### Wave 0 — 治理与架构决策（3–6d，阻塞后续开工） + +D1–D12 已于 2026-07-31 全部决策完毕（§5）。本 Wave = 把决定落盘为 ADR / issue / legal 动作，不写产品代码： + +1. **D1 落盘：format-neutral representation ADR**（按 RAGFlow 报告 §3.1 配方）：族契约 + nominal locator 子类型（`TextByteSpan` / `DocxXmlLocator` / `PdfRegionLocator`）、FIGURE kind、hard bounds、refusal vocabulary。**ADR 落盘前不写 DOCX/PDF 产品代码。** +2. **D2 落盘：rerank-bridging ADR**：两 Package 时序（内部 pre-rerank Package → ADR-0052 `AuthorizedModelInput` → final Package），解除 lift 5 阻塞。 +3. **D4 落盘：tokenizer-profile ADR + migration 计划**：ReleaseManifest 绑定真实 tokenizer，替换 `utf8-byte-token-v1` 的时机与 OpenAPI/schema 迁移步骤。 +4. **D9 落盘：OpenViking 有条件准入流程**：legal 复核（AGPL §13 + `ov_cli` 消歧请求）→ 逐 claim permalink 固定 → 四仓基线修订为版本化五仓基线（更新 PLAN/STATUS attribution）。 +5. **D6 落盘：新开 DOCX/outline 批准 issue**（不复用 #124）；**Room-A/B 边界登记**：三份 clean-room 报告标注为维护者本地研究，Room-B 提交只引本仓 requirements/ADR/tests。 + +### Wave 1 — 零代码风险的证据增量（5–9d，可与 Wave 0 并行） + +1. RRF provenance 纠偏 + 差分 oracle（Onyx lift 1 + RAGFlow §3.2）：固定"上游 `search.py` 是 weighted_sum 不是 RRF"的文档事实；补 Onyx `test_search_utils` 等价 fixture（重写而非 vendoring 上游 test model）。退出：denied candidate 任意插入不改变合法顺序（`make test` + catalog）。 +2. Adapter parity gate 通用骨架（RAGFlow §3.4）：canonical serializer + mutation-complete comparator + `ParityFixtureManifest`；首批覆盖 Markdown v3、SupplyChangePage、candidate fusion。退出：mutation matrix 全红/全绿符合预期（`make test`）。 +3. Redaction canary 进 catalog（Dify §3.6 + OpenViking §3.4）：query/body/denied/credential canary 扫描 ContextRun/log/metric/DecisionAudit。退出：canary 命中=0（`make catalog`）。 + +### Wave 2 — Runtime 检索链收口（Onyx 固定顺序，19–31d） + +严格按 ADR-0075 lift 顺序关闭**剩余 gate**（不是重写）： + +1. **lift 2 hybrid**（3–5d）：接口 provenance 决策（vendoring 极窄 ABC vs 原生 Protocol——推荐后者，见决策 D2）；真实 PG17 benchmark（EXPLAIN、underfill、recall）；production activation 记录。external query embedding 保持 `NOT_ACTIVE` 直到计量闭合。 +2. **lift 3 expansion**（4–6d）：把已完成的 `expand_fragment_window` 接入 resolve 编排；cross-Article refs 回送 Kernel；合并重叠窗口。退出：real PG + HTTP 证明 same-Article/current-Revision 继承、cross-Article 重授权、old-Revision/tombstone 拒绝。 +3. **lift 4 rewrite**（5–8d）：`RewriteProfile` + one-shot EgressGrant + 共享 meter；最多 1 semantic + 3 keyword，original query 固定保留。carrier 在 final usage 可验证前保持 `NOT_ACTIVE`。 +4. **lift 5 rerank**（7–10d + ADR 1–2d）：依赖 Wave 0 决策 D2 通过；混入 denied → gateway/assembler content bytes = 0 是 release veto。 +5. **lift 6 budget**（4–6d）：每 resolve 单一 meter，替换 `construction.py` 中 usage=0 的 ad-hoc 计数；final Package + ContextRun 发布同一累计 usage。**这是前四个 model-backed lift 转 active 的总开关。** + +### Wave 3 — DOCX + PDF outline copy+patch（10–15d，依赖 Wave 0 决策 D1） + +按 RAGFlow §3.1 配方：复制 `docx_parser.py` + `utils.py::extract_pdf_outlines` → `third_party/ragflow/deepdoc/parser/`；patch 移除 rag_tokenizer/LazyImage/Pandas/吞异常；OOXML block order 遍历；typed locator；双进程 digest 确定性证明；依赖（python-docx、pypdf）逐包许可证文本补齐；SBOM + 制品内 notice。PDF/OCR 九文件**保持 blocked**：只做 `/tmp` source-only spike（8–12d，可并行），模型资产（layout/det/rec/tsr.onnx、xgb.model）许可/hash 任一门失败即永久停止，不把 sunk cost 当准入理由。产品化另需 15–22d 且不在本蓝图承诺内。 + +### Wave 4 — 多源 planner + 运营/curation UX（22–36d） + +1. **Dify planner**（12–17d）：`AuthorizedSourceCapabilitySet`（server-derived，caller 只收窄）→ `QueryPlanner`（ROUTE_ONE 首版用 deterministic twin，model router `NOT_ACTIVE`）→ bounded FAN_OUT（required branch 失败整次 fail closed）。依赖 Wave 2 的单源 hybrid。 +2. **MaxKB preview/confirm Wave A**（8–12d）：digest-bound `RevisionPreviewRef`、受权 content inspection、edit-as-new-digest、confirm 只建 durable job（接现有 Supply，不建第二 queue）。退出：active Revision/Release pointer delta = 0。 +3. **alternate_query annotation 薄切片**（10–15d，C1）：propose → citation validation → human audit → immutable CurationSnapshot → frozen on/off eval。Tag/Termbase/direct-return 明确排后（Termbase 改 tokenizer，必须最后）。 + +### Wave 5 — Progressive disclosure + console + thin MCP（21–32d，依赖 Wave 2/4） + +1. **tiered AssemblyProfile**（6–9d，OpenViking §3.1）：`abstract/overview/detail` 只是**信息密度**不是存储层/ACL；每 hop 预留/结算同一 meter；同 Article lineage 继承、跨 Article 重授权。 +2. **Evidence Console 七类 job**（7–11d，OpenViking §3.5 + ADR-0090）+ Manual Hit Explorer（MaxKB §3.3，3–5d）：只走 `/v0/resolve`；score/rank 明示 unavailable；Control 一次一操作。 +3. **Authorized browse**（8–12d，OpenViking §3.2）：**等待 OpenCitation carrier 正式激活后才做**；在此之前 console 只对当前 Package 纯投影。 +4. **thin MCP parity**（6–9d，OpenViking §3.6 + Dify §3.4）：初版恰好一个 `resolve_context`，首个 consumer 锁定 **Codex**（D10）；HTTP/SDK/MCP 对同请求 Package digest 等价；不复制 read/list/remember/forget。 + +### Wave 6 — 长期/条件项（不承诺排期） + +- PDF/OCR 产品化（27–42d）：受模型资产门 + 单一 runtime target 决策双重否决。 +- session-derived Learning input（10–15d + retention 系统另估）：须先接受独立 consent/加密/retention ADR（D8），否则永久 `NOT_ACTIVE`。 +- Onyx ee 权限同步编排：永远 clean-room，仅作行为规格。 + +**总量**：Wave 0–5 约 **80–129 engineer-days**（不含 Wave 6、production auth、live connector、外部 legal 等待）。Wave 1/2 与 Wave 0 部分并行；Wave 3 与 Wave 2 不共享文件可并行；但每次合入共享 schema 前必须跑一次 `make integration` + `make security-gate` 最高 seam 证据。 + +## 4. 跨报告一致性核对（agent 互证结果） + +五份独立报告在以下点**互相印证或纠正**，可信度高： + +1. **RRF 归属纠正**（RAGFlow 发现，Onyx 印证）：RAGFlow `search.py` 用后端 `weighted_sum(0.05,0.95)`，**不是 RRF**；真正的 weighted RRF 函数在 Onyx `search_utils.py`。本仓 owned RRF 的 provenance 文档不应错误归因 RAGFlow。 +2. **pre-Kernel 加权禁令**（Onyx ADR-0083 + RAGFlow cut line + Dify §3.2 三方一致）：fusion 只携 content-free evidence，权重只在授权后 admitted positions 上计算。 +3. **hydration 反例收敛**（Dify formatter + Onyx `_retrieve_adjacent_chunks:149` + RAGFlow `retrieval_by_children`）：三个上游都显式存在"首 hit 授权覆盖扩展"的隐含前提，全部列为 must-kill。 +4. **observability 红线收敛**（Dify §3.6 + OpenViking §3.4）：两份独立报告给出逐字段相同的 ContextRun 存活/禁止信号表 → 直接作为 redaction catalog 输入。 +5. **copy+patch=none 收敛**（Dify SDK + OpenViking CLI/examples + Onyx 大多数 lift）：generated SDK / 原生实现 + permalink 在四个独立评估中都胜出，只有 RAGFlow 两个解析文件值得注册。 +6. **现状感知**（Onyx 独有，已代码核对属实）：三 keystone seam 与 lift 1/2 已在 `engine/runtime/{prekernel_fusion,authorized_ranking,budget,model_inference}.py`、`engine/persistence/supply_execution.py`、`adapters/{hybrid,fts,pgvector}.py` 落地；路线图因此是"关 gate"而非"重写"。 +7. **新发现的许可证事实**：OpenViking `ov_cli` Cargo.toml 自报 MIT 与父目录/README Apache-2.0 冲突；Dify `sdks/php-client` 仅 README 称 MIT（未取证）；RAGFlow ONNX/XGBoost 资产不在 Git 树（未取证）。三者都按"冲突/未知即停止"处理。 + +## 5. 决策记录（2026-07-31 stometa 全部确认） + +12 项决策已全部作出（均采纳推荐方案）。Wave 0 的剩余工作 = 把这些决定**落盘为 ADR / issue / legal 动作**，而不是继续讨论。 + +| # | 决策结果 | 落盘动作 | 来源 | +|---|---|---|---| +| D1 | **format-neutral 族契约 + nominal locator 子类型**：`ParsedDocument` 提升为格式中立族（共享 publication interface、canonical serialization、hard bounds），locator 分 `TextByteSpan`/`DocxXmlLocator`/`PdfRegionLocator`；**新增 FIGURE structural kind**，image bytes 走独立 bounded artifact policy | 写 representation ADR（4–6d），未决前不写 DOCX/PDF 产品代码 | RAGFlow Q1/Q3 | +| D2 | **两 Package 时序**：Kernel → 内部不交付的 pre-rerank Package → ADR-0052 `AuthorizedModelInput`（+ one-shot model EgressGrant）→ rerank = Evidence exact permutation → final Package（独立 final-hop grant）。不发明第二种同名 nominal type。**协调者诠释（2026-07-31，review round 2 分歧裁决，claude 认同 / codex 保留）**："单一 nominal contract" 指单一契约定义（构造规则 + canonical serialization + digest），跨进程以 digest-equivalent twin 实例化（ADR-0052 双语言 digest 权威先例）；D2 禁止的是同名不同构造规则的第二契约（如 projection-fed），不禁止同一契约的 twin。已写入 ADR-0095 clause 2 | 写 rerank-bridging ADR，解除 lift 5 阻塞 | Onyx Q1 | +| D3 | **原生 Protocol + permalink**：不 vendor Onyx ABC；在 ADR/MODIFICATIONS 记录 `adapters/hybrid.py` 是 Onyx `interfaces_new.py@2fb3dd1`（sha256 `2285d0cb…`）HybridCapable 形状的等价独立实现 | ADR 补 provenance 段落；无 third_party 注册 | Onyx Q2 | +| D4 | **model-backed lift 激活前引入 ReleaseManifest 绑定的真实 tokenizer**（pinned 制品 + digest），lift 6 一步做成完整 meter（tokens + calls/cost/elapsed 累计），含一次 schema/OpenAPI migration | tokenizer-profile ADR + migration 计划 | Onyx Q3 | +| D5 | **推荐组合**：先单源 hybrid 收尾（Wave 2）；ROUTE_ONE 只用 deterministic rule/twin，model router `NOT_ACTIVE`；required branch 失败整次 fail closed，无 partial Package | planner 规格按此冻结；model router 重开需新 ADR | Dify Q1/Q2/Q3 | +| D6 | **新开 DOCX/outline 专用批准 issue**（不复用 #124）：列 source_paths、固定 commit、双 hash、依赖许可证文本 owner、SBOM checklist | 开 issue；python-docx/pypdf 许可文本随注册 PR 落盘 | RAGFlow Q6 | +| D7 | **窄 Curation/Intake application service**（不扩 ContextControl canonical Interface）；**双角色分离**：reviewer 可 confirm import，只有 release operator 可 promote；一人流程也产生两份 nominal authority + 两次审计 | preview seam 设计文档；privilege test 证明无越权 | MaxKB Q1/Q2 | +| D8 | **立项但排 Wave 6**：写独立 consent/最小字段/加密/retention/删除-export/RLS ADR；此前 Learning 候选只从 authorized Package evidence + explicit feedback 产生，原始 transcript extraction 保持 `NOT_ACTIVE` | retention ADR 骨架进 Wave 6 排期 | OpenViking Q4 | +| D9 | **有条件准入 OpenViking 为第五仓**：白名单 = context FS/L0–L2 tiering、session→candidate UX、observable trajectory、agent exposure；黑名单 = 多租户授权/安全证明、runtime foundation、任何 copy+patch。流程：legal 复核 → 逐 claim permalink 固定 → 四仓基线修订为版本化五仓基线；同流程向 upstream/legal 发 `ov_cli` MIT↔Apache 消歧（即使消歧也不复制） | legal 复核任务 + 基线修订 PR | OpenViking Q1/Q7 | +| D10 | **首个 thin MCP consumer = Codex**：工具面恰好一个 `resolve_context`；成功标准 = HTTP/generated SDK/MCP 对同请求 digest 等价或同一 closed refusal | 更新 exposure-spike-mcp-vs-api 研究文档 | OpenViking Q6 | +| D11 | **首切片锁 `alternate_query`**（propose → citation validation → human audit → CurationSnapshot → frozen on/off eval）；**direct return = `verbatim_authorized_block`**，versioned AssemblyProfile、BotDelivery 消费 audience-bound Package 带 citation、默认关闭经 canary 激活；Tag/Termbase 排后 | C1 切片 issue；引擎不生成答案边界不变 | MaxKB Q5/Q6 | +| D12 | **离线安装 + 单 runtime target**：模型 bundle 经独立受控安装取得，逐项固定 revision/sha256/license/model card，只读 digest-bound；runner 零网络；首版只支持一个 CPU/ONNX Runtime/arch 组合；资产门任一失败 PDF/OCR 永久停止 | 资产门 checklist 进 D6 父级跟踪 | RAGFlow Q4/Q5 | + +## 6. 合规与公开口径清单（Definition of Done 附加项） + +- [ ] 任何新增 copy+patch：`third_party//` 具备 `LICENSE.upstream`、`UPSTREAM.toml`（repo/commit/paths/excluded/hashes/mode/approval）、`MODIFICATIONS.md`、`patches/`、`sbom.cyclonedx.json`，且 wheel/sdist/npm/container **物理包含** notice + SBOM（Git-only attribution 不合规）。 +- [ ] Dify/MaxKB/OpenViking 相关 PR 描述不引用三份 Room-A 报告为公开 provenance；只引本仓 ADR/tests/四仓基线。 +- [ ] OpenViking 准入前，README/PLAN/STATUS/设计文档不出现 OpenViking 作为 authority 的引用。 +- [ ] 每个 Wave 的 STATUS.md 更新区分 Active / `NOT_ACTIVE`；bounded proof 不升级为 general claim；`make smoke` 绿不升级为 publication/security 证明。 +- [ ] RAGFlow PDF/OCR：资产门（ONNX/XGBoost 逐项 revision+hash+license+model card）未闭合前，profile 报 `NOT_ACTIVE`，spike 代码留在 `/tmp` 或 runtime-tree 外并留删除证明。 +- [ ] 全局不变量逐 release 复核：Unauthorized Evidence = 0、wrong-Organization effect = 0、missing tenant context = fail closed。 + +## 7. 如果只做一件事 + +每个仓库对 ContextEngine 的**单一最高价值贡献**，按可开工顺序： + +1. **Onyx**：六 lift 的剩余 gate（尤其 lift 6 统一 meter）——把已建成的检索链从 "NOT_ACTIVE" 变成可证明 active 的最后一公里（24–37d）。 +2. **RAGFlow**：DOCX + PDF outline 两文件 copy+patch——用最小法律风险打开第一个非 Markdown 格式（10–15d，依赖 D1）。 +3. **MaxKB**：preview→confirm 运营心智——把 ingestion 变成可审计运营动作，且全程 pointer delta=0（Wave A 8–12d）。 +4. **Dify**：AuthorizedSourceCapabilitySet + bounded fan-out——多源编排的唯一干净形状（12–17d，依赖 Wave 2）。 +5. **OpenViking**：tiered AssemblyProfile + 单工具 thin MCP——progressive disclosure 与 agent exposure 的产品化，但安全上仅战略参考（12–18d 首批）。 diff --git a/docs/research/2026-07-31-maxkb-blueprint-evaluation.md b/docs/research/2026-07-31-maxkb-blueprint-evaluation.md new file mode 100644 index 00000000..da329bb1 --- /dev/null +++ b/docs/research/2026-07-31-maxkb-blueprint-evaluation.md @@ -0,0 +1,617 @@ +# Room-A 研究产物 — 维护者本地研究，非公开 provenance；Room-B 实现者只读规格与 oracle，不读 MaxKB 源码 + +> **决策状态**：本文开放问题已由维护者于 2026-07-31 全部决定（D7/D11），结果见 [`five-repository-implementation-blueprint.md`](./2026-07-31-five-repository-implementation-blueprint.md) §5；正文推荐项为评估时刻的状态。 + +> 本文是 GPLv3 clean-room 两室协议中的 Room-A 规格。它只记录固定版本的可观察行为、接口形状与测试 oracle，不授权复制 MaxKB 代码、依赖、schema、SQL、提示词、UI 实现或命名。除本仓 accepted ADR 与一手 ContextEngine 要求外，本文不得作为公开 provenance。 + +## 1. 固定 commit 与许可证核验 + +### 固定输入与结论 + +| 项目 | 静态核验结果 | 对 Room-B 的约束 | +|---|---|---| +| 上游仓库 | `1Panel-dev/MaxKB` | 只允许从本文规格与 oracle 独立实现 | +| 固定 commit | [`32b2d885e47ad04639abd7a18490bf5937f9c072`](https://github.com/1Panel-dev/MaxKB/commit/32b2d885e47ad04639abd7a18490bf5937f9c072) | 所有上游事实均只针对此 commit；后续版本不得自动外推 | +| 根许可证 | 固定 commit 的 [`LICENSE`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/LICENSE) 是 GNU GPL version 3；其中 §5(c) 要求以 GPLv3 许可整个被传递的组合作品 | 根据 ADR-0074，MaxKB 全部实现区为 **clean-room only**；零代码、零依赖、零 schema/SQL/UI 复制 | +| 单独许可子树 | 对 checkout 中 `LICENSE*`、`COPYING*`、`NOTICE*`、`.gitmodules`、常见 `vendor`/`third_party` 目录及 SPDX/Apache/MIT 授权头的静态扫描，没有发现可从 MaxKB 仓内单独取用的许可子树 | **可用子树：无**。依赖自身许可证不等于 MaxKB 对其粘合代码、配置或产品实现作出了单独授权 | +| 研究方式 | 只读源码和 UI 静态链路；未启动 MaxKB、未跑动态请求、故障注入、并发测试或 benchmark | 动态效果、性能、恢复可靠性与安全保证一律保持 `[未取证]` | + +核验边界：仓内仅发现根 `LICENSE` 这一份许可证文本；`ui/src/api/system/license.ts` 是业务文件名，不是许可授予。没有发现单独许可子树是本次固定 checkout 的有限静态结论，不是法律意见；隐藏在生成物、依赖包或仓外分发物中的 notice 完整性仍为 `[未取证]`。即使未来发现 permissive dependency，也只能独立从其原始上游、固定版本和许可边界评估，不能从 MaxKB checkout 拷贝。 + +本报告遵守 [ADR-0074](../decisions/0074-adopt-controlled-third-party-code-reuse.md)：Room-A 可观察行为，Room-B 不读 MaxKB 源码。公开材料如需描述 MaxKB，只能回引本仓已批准的[四仓证据基线](2026-07-19-four-public-repositories-evidence.md)，不能引用本文新增的维护者本地研究作为公开 authority。 + +## 2. 能力盘点 → ContextEngine 区域映射表 + +| MaxKB 可观察能力或产品形状 | 固定 commit 一手路径 | ContextEngine 区域 / 所属 seam | 判定 | 独立实现结论 | +|---|---|---|---|---| +| 上传 → 分段规则 → 预览 → 编辑/删除 → 导入 | [`UploadDocument.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/UploadDocument.vue)、[`SetRules.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/upload/SetRules.vue)、[`ParagraphList.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/knowledge/component/ParagraphList.vue) | Supply candidate Revision + Control UX adapter；ADR-0018；随后由 ReleaseCandidate / ContextLearning 决定在线 release | **clean-room Room-A spec** | 保留“先看再确认”的操作心智；改成 digest-bound 候选、分离内容审阅与 release 门禁、全程不可变 lineage | +| `/document/split` 接收文件、规则、长度、过滤开关并返回可编辑分段 | [`views/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/views/document.py)、[`serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py) | versioned CompilationProfile → deterministic CompiledRevision preview | **clean-room Room-A spec** | 输入 profile 必须版本化；重复 preview 产生同 digest；preview 不改 active pointer、不进入生产 index | +| preview 时把源文件持久化并用 `source_file_id` 串联 confirm | [`serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py) | tenant-owned preview blob / candidate lineage | **do-not-take / anti-pattern** | 临时内容必须有 Organization、actor、purpose、TTL、digest 和清理状态；临时文件 ID 不是发布或授权能力 | +| Paragraph 作为可编辑检索单元 | [`models/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/models/knowledge.py) | immutable ContextFragment | **clean-room Room-A spec**（产品心智） | UX 可以显示“段落/内容块”；持久层不得原地改 Fragment，任何表示变化产生新 ContextRevision | +| Problem 与 Paragraph 多对多关联、手工或 LLM 生成相关问题 | 同上；[`serializers/paragraph.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/paragraph.py) | proposed CurationAnnotation `alternate_query` / golden-case candidate | **clean-room Room-A spec** | 关联问题可进入候选召回或评测覆盖；先审计，再进入 CurationSnapshot；绝不直接写生产索引 | +| document-scoped key/value Tag 及筛选 UX | [`models/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/models/knowledge.py)、[`TagDrawer.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/tag/TagDrawer.vue) | source metadata 或 CurationAnnotation `facet_tag`，二者必须明确分型 | **clean-room Room-A spec** | 源事实随新 Revision；人工/模型治理标签随 CurationSnapshot。禁止同一 mutable Tag row 同时承担两种真相 | +| Termbase 参与分词向量保存时的搜索向量构造 | [`models/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/models/knowledge.py)、[`vector/pg_vector.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/vector/pg_vector.py) | audited `term` annotation → immutable CompilationProfile / IndexProfile rebuild | **clean-room Room-A spec** | 术语表值得进入 curation UX；变化必须创建新 profile 并重编译/重索引，不得让 mutable row 改写现存索引语义 | +| Hit Test：query、top N、相似度、embedding/keywords/blend，返回命中内容和分数 | [`serializers/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/knowledge.py)、[`views/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/views/knowledge.py) | ContextRuntime tracked resolve + exploration report；golden intake | **clean-room Room-A spec** | 保留可解释调试面；命中内容必须来自 ContextPackage，手测结果不构成 release gate | +| direct return：document mutable flag + threshold，命中后跳过生成 | [`base_search_knowledge_node.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/flow/step_node/search_knowledge_node/impl/base_search_knowledge_node.py)、[`base_chat_step.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/chat_pipeline/step/chat_step/impl/base_chat_step.py) | versioned AssemblyProfile / RuntimeProfile；仍只消费 AuthorizedProjection / ContextPackage | **clean-room Room-A spec**（需收紧） | “verbatim authorized block”可作为显式 release 候选；阈值和策略必须版本化、评测、canary，不能由 document row 即时改生产行为 | +| 点赞/点踩与原因；把回答标注为改进内容 | [`chat_record.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/chat/serializers/chat_record.py)、[`application_chat_record.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/serializers/application_chat_record.py) | authorized-only ContextRun feedback → triage → golden candidate / CurationAnnotation candidate | **clean-room Room-A spec** | 保留反馈采集和人工归因；反馈只生成候选，必须绑定 Package、release generation 与 citations；不能直接造 Fragment | +| Document/Paragraph 落库后由 decorator/Celery 触发 embedding，删除旧向量再建新向量 | [`serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py)、[`task/embedding.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/task/embedding.py)、[`listener_manage.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/common/event/listener_manage.py) | WorkerLease-bound Supply + ADR-0066 embed-before-publication | **do-not-take / anti-pattern** | 采用完整 embedding readiness 的行为目标；拒绝“先可见内容、后补向量”、delete/recreate 活跃索引和 ambient Celery authority | +| 11 个 Django app 的 `tests.py` 是三行空桩 | [`apps/knowledge/tests.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/tests.py) | highest public seam + real PG17/invariant catalog | **do-not-take / anti-pattern** | UI 手测、文件存在和测试数量均不能证明行为；active invariant 未执行即 FAIL | +| serializer/view/decorator 同时承载校验、权限、事务、状态、任务派发和向量副作用 | [`serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py) | Deep Modules；sealed Runtime；narrow Supply/Learning ports | **do-not-take / anti-pattern** | transport 只做受信输入构造和 DTO 映射；策略与发布只能在所属 Module 的封闭事务里执行 | + +全表的总约束是：**candidate 永远不等于 active behavior**。Supply 的原子 `ContextResource.active_revision` 只发布完整不可变内容；在线 Runtime 采用哪个 corpus/profile/curation 仍由唯一的、release-operator-authorized `ContextLearning.promote` 激活 `ReleaseManifest`。`ContextControl` 只治理 source/access/policy，不能发布 profile；评测 executor 只产报告，也不能发布。 + +## 3. 逐能力蓝图 + +### 3.1 preview → confirm ingestion：从候选 Revision 到两段原子可见性 + +**上游路径 + permalink** + +- 三步 UI（上传、规则/预览、成功）和最终批量导入：[`ui/src/views/document/UploadDocument.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/UploadDocument.vue)。 +- intelligent/advanced 分段，advanced 可选 pattern、长度、特殊字符过滤；右侧按文件展示分段：[`ui/src/views/document/upload/SetRules.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/upload/SetRules.vue)。 +- 每个分段展示 title、content、字符数，并可编辑/删除：[`ui/src/views/knowledge/component/ParagraphList.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/knowledge/component/ParagraphList.vue)。 +- preview API 与 confirm API 是两条路径：[`apps/knowledge/views/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/views/document.py)、[`apps/knowledge/serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py)。 + +**映射的本仓 seam / ADR** + +- `ContextSource → ContextResource → ContextRevision → ContextFragment`，`prepared → indexed → active`：[ADR-0018](../decisions/0018-immutable-revision-publication.md)。 +- embedding 完整才允许 publication：[ADR-0066](../decisions/0066-embed-fragments-before-publication.md)。 +- post-Revision curation 单独形成 `CurationSnapshot`：[ADR-0014](../decisions/0014-curation-snapshot-and-release-ownership.md)。 +- 当前 corpus 形成 generation-bound `ReleaseCandidate`：[ADR-0073](../decisions/0073-compose-explicit-release-candidates-from-current-corpus.md)。 +- 只有 `ContextLearning.promote` 改 active Release pointer：[ADR-0033](../decisions/0033-promote-organization-releases-through-one-learning-owner.md)。 + +**Room-A 行为规格** + +状态机分成三个不可混写的状态族： + +```text +Preview: uploaded -> compiled_preview -> reviewed -> confirmed | abandoned | expired +Supply: accepted_job -> acquired -> prepared(embeddings complete) -> indexed -> resource_active +Release: candidate -> executor_evaluated(PASS|FAIL|REFUSED) -> operator_authorized -> promoted +``` + +1. `uploaded` 接受受信 Organization/actor/source binding、一个或多个 source blob refs、不可变 `CompilationProfileRef` 和 preview TTL；不接受 caller 自报 Organization、active Revision 或 release generation。 +2. `compiled_preview` 必须固定 source digest、compiler/profile digest、规范化 Revision digest、Fragment 有序清单 digest、warning/refusal category；相同输入必须得到相同 digest。preview 失败时没有 ImportJob、Revision、Fragment、index candidate 或 active-pointer 效果。 +3. reviewer 可以对候选执行 `keep | edit_as_new_candidate | split | merge | remove | attach_alternate_query | propose_tag | propose_term`。任何内容编辑都生成新的 candidate digest，不原地改已审候选；旧候选留审计 lineage。 +4. `confirm` 使用 optimistic compare：`PreviewRef + expected_candidate_digest + decision_reason_digest`。digest 已变、preview 过期、actor 权限撤销、SourceVersion 变化或 Resource tombstoned 时拒绝，业务效果为零。confirm 只创建 durable acquisition/import job；它不直接写 active pointer。 +5. Supply worker 只能用与 exact durable job、Organization、Source、Resource、operation、generation、nonce 完全绑定的 `WorkerLease` 继续。编译、embedding、indexing 任一步不完整时旧 Revision 保持在线。 +6. `resource_active` 是 ADR-0018 的一次 PostgreSQL pointer transaction；读者只见完整旧版或完整新版。它使完整 Revision 成为当前 corpus 事实，但**不会自动改变 promoted ReleaseManifest**。 +7. ContextLearning 从当前 corpus 观察生成 generation-bound `ReleaseCandidate`，由 ADR-0080 executor 运行权威评测。报告 `REFUSED`/`FAIL` 或四门任一不通过时不可 promote。 +8. release operator 看到精确 base generation、manifest/corpus/profile digests、四门状态、compatibility、capability coverage、commands、报告时间和 executor seam 后，才授权 `ContextLearning.promote`。promotion 原子更新 Release pointer 并追加 success audit；并发 loser 或 stale base 的效果为零。 +9. curation 不阻塞 Revision publication。审计通过的 annotation 另组 `CurationSnapshot`；curation-on 只有作为兼容 `ReleaseManifest` 的 `CurationProfileRef` 经同一 promote 路径才生效。缺失/失败 curation 退回正常非 curation retrieval，而不是弱化授权。 + +**operator-visible signals** + +| 面 | 可见内容 | 不能显示 / 不能代表 | +|---|---|---| +| candidate inspection（受权 reviewer） | **content-bearing**：文件名的安全显示名、每个候选 Fragment 的 source-ordered title/body、字符/token 估算、structural path、source span、拟关联问题/tag/term、before/after diff、compiler warning；默认分页且不把内容写入普通日志 | denied source 内容、其他 Organization 内容；preview 内容不代表已授权 Runtime Evidence | +| queue/status 列表 | **content-free**：PreviewRef/JobRef 的 opaque 或 digest 形式、source/resource refs、candidate digest、Fragment count、byte/token totals、profile refs、状态、generic failure category、created/expiry time | 正文、原始路径、secret、provider error、denied candidate rank | +| release decision | **content-free 为主**：expected base generation、manifest/corpus/profile/eval digests、PASS/FAIL/REFUSED、四门、slice counts/uncertainty、stale-lineage count、executor seam、commands、candidate diff counts | “手测通过”、caller 上传 counters、单一 aggregate score；任何内容摘要都不能替代受权 inspection | + +**审计链** + +追加且不可变地记录 `PreviewCreated → CandidateRecompiled → ReviewerDecision → ImportJobAccepted → LeaseIssued/Reclaimed → acquired/prepared/indexed/active events → ReleaseCandidateObserved → ExecutorReport → PromotionAuthorized → PromotionCommitted`。每条至少绑定 Organization、actor/service、source/resource、old/new digest、profile refs、base generation、UTC instant、reason/refusal category；restricted audit 可持 decision detail，普通运营审计只持 content-free digest。失败 promotion 不写 success audit；denied details 不进入 ContextRun 或 Learning corpus。 + +**接口形状草图（独立设计，不是 MaxKB API 翻译）** + +```python +class RevisionPreviewPort(Protocol): + def create( + self, + call: AuthorizedCurationCall, + source_blobs: tuple[SourceBlobRef, ...], + compilation_profile: CompilationProfileRef, + ) -> RevisionPreviewRef: ... + + def inspect( + self, call: AuthorizedCurationCall, preview: RevisionPreviewRef, page: PageRequest + ) -> RevisionPreviewPage: ... # content-bearing, separately authorized + + def revise( + self, call: AuthorizedCurationCall, preview: RevisionPreviewRef, + expected_digest: Sha256, edits: tuple[CandidateEdit, ...] + ) -> RevisionPreviewRef: ... # new digest, immutable predecessor + + def confirm( + self, call: AuthorizedCurationCall, preview: RevisionPreviewRef, + expected_digest: Sha256, reason_digest: Sha256 + ) -> ImportJobRef: ... # no publication effect + +class RevisionReadinessEvaluator(Protocol): + def evaluate(self, candidate: CandidateRevisionRef) -> RevisionReadinessReport: ... + +class ContextLearning: + def evaluate(self, candidate: ReleaseCandidateRef) -> ReleaseEvaluation: ... + def promote(self, call: TrustedPromotionCall) -> PromotionReceipt: ... +``` + +`RevisionReadinessReport` 只证明 deterministic compilation、embedding/index completeness 和 compatibility；它不是 ADR-0080 的权威 release evaluation，也没有 publication authority。 + +**测试 oracle 清单** + +- 相同 bytes + profile 产生相同 candidate/Fragment ordering/digest；任一 edit 产生不同 digest，旧 preview 不变。 +- preview、revise、abandon、expire 的 active Revision pointer、active Release pointer、生产 index row 数变化均为 0。 +- unauthorized/cross-Organization preview inspect 返回同类 generic refusal；正文、路径和 item count 不泄漏。 +- confirm 时 actor/source/profile/digest 任一 stale：ImportJob=0、pointer=0、success audit=0。 +- 在 acquired/prepared/indexed/pointer-before-commit 每个 fault point 终止：旧 Revision 完整可读；恢复只由新 lease generation 继续；旧 nonce 效果为 0。 +- missing/wrong-dimension embedding 不能进入 `prepared/indexed/active`。 +- resource activation 后、Release promotion 前，Runtime 仍按旧 promoted manifest 行为；候选不等于 active behavior。 +- `evaluate` 的 PASS 也不改 pointer；非 release operator、过期 grant、stale generation、四门任一失败或 REFUSED 的 promote 效果为 0。 +- curation pipeline、ContextControl、migration、bootstrap、evaluator 均不能直接写 Release pointer；rollback 走新的 candidate + 同一个 promote。 +- UI content-bearing inspect 只走受权服务 seam；列表/日志/metrics 不含 source bytes、denied detail 或 corpus path。 + +**验证命令** + +```bash +make lint +make typecheck +make test +make ui-test +make catalog +make smoke +make integration # 真实 PG17：fault points、RLS、CAS、publication visibility +make security-gate # 需先 make db-up；hard-oracle 与 release evidence +``` + +**工作量与依赖** + +- 估算：**18–27 engineer-days**（1 名工程师；preview contract/storage 5–7，受权 inspection/UX 4–6，confirm/job/audit 4–6，release wiring 与 tests 5–8；不含新 parser、真实 provider procurement 和 UX research）。 +- 依赖：现有 File acquisition/WorkerLease/recovery、ADR-0066 embedding、release candidate/evaluation/promotion、trusted operator authentication、owner-only preview blob store、local Evidence Console 或等价受权 UI。若要激活 curation-on，另依赖 C1 annotation audit 与 CurationSnapshot 实现。 + +### 3.2 Paragraph / Problem / Tag / Termbase：知识单元与 curation UX + +**上游路径 + permalink** + +- `Paragraph(content,title,status,hit_num,is_active,position,chunks)`、`Problem(content,hit_num)`、mapping、`Tag(key,value)`、`Termbase(content)`：[`apps/knowledge/models/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/models/knowledge.py)。 +- Paragraph 关联/解除 Problem、创建/编辑/删除以及 embedding 副作用：[`apps/knowledge/serializers/paragraph.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/paragraph.py)。 +- document tag 的 key/value 与关联文档 UX：[`ui/src/views/document/tag/TagDrawer.vue`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/ui/src/views/document/tag/TagDrawer.vue)。 +- Termbase 在向量保存时参与全文 `search_vector`：[`apps/knowledge/vector/pg_vector.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/vector/pg_vector.py)。 + +**映射的本仓 seam / ADR** + +- `Paragraph` 产品概念 → `ContextFragment`，继承唯一 Article/ContextResource 与 Revision lineage；不能独立授权。 +- `Problem` → `CurationAnnotation(kind=alternate_query)` 或 golden-case candidate。 +- `Tag` → 必须二选一：source-declared metadata 进入新 Revision，或 `CurationAnnotation(kind=facet_tag)` 进入 snapshot。 +- `Termbase` → `CurationAnnotation(kind=term)`；接受后组成不可变术语快照并进入 versioned `CompilationProfile`/`IndexProfile`。它若影响 assembly 展示，另由 versioned `AssemblyProfile`/`RuntimeProfile` 引用，不能一个 profile 包办所有变化域。 +- [ADR-0014](../decisions/0014-curation-snapshot-and-release-ownership.md)、[ADR-0033](../decisions/0033-promote-organization-releases-through-one-learning-owner.md)；Article 是内容授权原子的约束见 ADR-0077/本仓 threat model。 + +**哪些概念值得进入 curation UX** + +| UX 概念 | 保留方式 | operator 看到什么 | 生效条件 | +|---|---|---|---| +| Fragment inspector | 显示 immutable Fragment 与 Revision lineage，不称 mutable paragraph | content-bearing title/body、structural path、span、source/revision/profile refs、active/superseded 标记 | 内容只能通过新 Revision 改；reviewer 不能直接 patch active Fragment | +| Alternate query / related problem | 一对多 annotation candidate，来源可为 human、feedback、LLM | query 文本、目标 Fragment、生成来源、citation、confidence、duplicate warning | human audit + frozen on/off eval + CurationSnapshot + promote | +| Facet tag | typed key/value annotation；明确 source_fact 或 curated | tag、scope、目标 Article/Fragment、proposer、evidence、冲突/覆盖关系 | source_fact 随 Revision；curated 随 snapshot；均不参与授权 | +| Term | term、aliases、language、match mode、作用域和 tokenizer impact | term 内容、命中样本、预计 recompile/reindex 数、profile diff | audit 后创建新 term snapshot/profile；完整 rebuild 和 release gate | +| Dedup/stale | C1 计划中的 cluster / stale annotation | representative/member refs、理由、置信区间、Revision compatibility | 按 kind 预注册样本和误标阈值；不足只能 inconclusive | + +**必须拒绝的 mutable-row 语义** + +- `Paragraph.content/title/chunks/is_active/position` 不能在同一个 stable row 上既做 source truth、检索载体又做 active release；表示变化创建全新 Revision/Fragments。 +- `hit_num` 不能写回 Fragment 作为学习权威；聚合使用带 release/profile/run lineage 的派生 telemetry，且不得影响授权。 +- `Problem.hit_num`、Tag、Termbase 的当前行不能即时改变 recall、tokenization、ranking 或 assembly；它们只能生成不可变候选/profile。 +- Problem 与 Fragment 的 mapping 不能赋予可见性；跨 Article/Resource 的任何 expansion 都要重新授权。 +- 模型生成 `Problem/Tag/Term` 只处于 `proposed`，不能写 production index；失败、低置信度或无 citation 只能拒绝/待审。 + +**Room-A 行为规格** + +```text +annotation: proposed -> evidence_validated -> human_accepted | rejected | expired +snapshot: assembling -> compatibility_checked -> evaluated -> release_candidate +profile: draft -> built -> evaluated -> promoted | superseded +``` + +- 输入：exact Organization、target Revision/Fragment refs、kind-specific payload、proposer identity/model ref、source ContextRun/feedback refs、citations、profile base digest、proposed_at。 +- 输出：content-addressed `CurationAnnotationRef`；accepted 集合组成 immutable `CurationSnapshotRef`，固定 compatible Revision set、member ordering、evaluation digest。 +- `evidence_validated` 要求引用真实且属于目标 Revision；模型 assertion 无 citation、citation 跨目标或 lineage stale 一律拒绝。 +- reviewer 的 accept/reject 必须记 reason category/digest；任何 edit 生成新 annotation candidate。 +- snapshot 构建时 target Revision 已 superseded、缺失、跨 Organization、重复/冲突 annotation、profile base stale 时 fail closed。 +- Runtime 只能在一个数据库 snapshot 中读取 promoted ReleaseManifest 指定的 active Revision 和 compatible CurationSnapshot。snapshot 缺失/失败时 curation-off，不回退到“最新 mutable annotation”。 +- annotation 只能影响 ranking/assembly，不能改变 AuthorizationKernel 的 allow/deny、field projection 或 SourceAclEvidence。 + +**operator-visible signals** + +- annotation inbox 列表保持 content-free：kind、target lineage digest、proposer kind、citation count、base profile/release refs、state、confidence/uncertainty、created/expiry time；不显示正文、denied Evidence 或跨 Organization existence。 +- 单条受权 inspection 才显示 Fragment 正文、proposed query/tag/term、citations、before/after profile diff、conflict 和 deterministic validation；accept/reject 必须要求 reason category/digest。 +- snapshot/release 面显示 accepted/rejected/expired counts、compatible Revision count/digest、per-kind sample/threshold/uncertainty、rebuild impact 和 evaluation digest；`proposed`、无 citation 或 inconclusive 均不得渲染为 active。 + +**接口形状草图** + +```python +@dataclass(frozen=True) +class CurationAnnotationCandidate: + organization_ref: OrganizationRef + target: FragmentLineage + kind: Literal["alternate_query", "facet_tag", "term", "dedup", "stale"] + payload_digest: Sha256 + evidence_refs: tuple[EvidenceRef, ...] + proposer_ref: ActorOrModelRef + base_profile_digest: Sha256 + +class CurationReviewPort(Protocol): + def propose(self, call: AuthorizedCurationCall, candidate: CurationAnnotationCandidate) -> CurationAnnotationRef: ... + def inspect(self, call: AuthorizedCurationCall, ref: CurationAnnotationRef) -> AnnotationInspection: ... + def decide(self, call: AuthorizedCurationCall, ref: CurationAnnotationRef, + expected_digest: Sha256, decision: AuditDecision) -> AnnotationAuditReceipt: ... + +class CurationSnapshotBuilder(Protocol): + def assemble(self, call: LearningBuildCall, + annotations: tuple[CurationAnnotationRef, ...], + compatible_revisions: tuple[ContextRevisionRef, ...]) -> CurationSnapshotRef: ... + +class ProfileCompiler(Protocol): + def compile_terms(self, snapshot: CurationSnapshotRef, + base: CompilationProfileRef) -> CompilationProfileRef: ... + def compose_assembly(self, snapshot: CurationSnapshotRef, + base: AssemblyProfileRef) -> AssemblyProfileRef: ... +``` + +最后两种 profile 必须落到 ReleaseManifest 已拥有的 `ContentProfileRef`/`IndexProfileRef`/`RuntimeProfileRef`/`CurationProfileRef` 兼容关系中；不得新增第二个 active pointer 或让 `ContextControl` 发布。 + +**测试 oracle 清单** + +- 修改 Fragment 内容只能产生新 Revision；旧 Revision/Fragment digest 和 citations 永不变化。 +- annotation target 跨 Organization、跨 Revision、不存在或 stale：accepted=0、snapshot member=0、index effect=0。 +- LLM alternate query/tag/term 缺 citation、引用不在 authorized feedback binding 内：保持 proposed/rejected，生产 index effect=0。 +- 同一组 accepted annotations 不论输入顺序都得到同一 canonical snapshot digest；duplicate/conflict 规则确定性。 +- term 增删导致新 Compilation/Index profile digest；旧 active profile 与 vectors/search_vector 不原地变化；全量 readiness 前不能 promote。 +- curation-on 缺 snapshot ref、compatible Revision set 或 evaluation digest：evaluate/promote 拒绝，active manifest 不变（`tests/unit/test_release_owner_architecture.py` 对应结构 oracle）。 +- Runtime 只把 annotation 应用于 AuthorizedProjection；denied bytes 进入 curation/rerank/assembly 计数为 0。 +- snapshot 不兼容时 Runtime 使用 curation-off 的正常检索或 release candidate 被拒绝，不读取 latest mutable row。 +- ContextControl、review UI、snapshot builder 均无 active pointer DML；只有 release-operator-authorized promote 生效。 + +**验证命令** + +```bash +make lint +make typecheck +make test # release contracts、curation workflow、feedback authority +make ui-test # UI 只用 public seam；feedback 无 publication authority +make catalog +make integration # FORCE RLS、cross-org、release compatibility +make security-gate +``` + +**工作量与依赖** + +- 估算：**24–36 engineer-days**（annotation contract/audit 7–10，Fragment/alternate-query UX 5–7，tag/term profile build 6–10，snapshot/release/tests 6–9）。若只做 `alternate_query` 最小切片：**10–15 days**。 +- 依赖：M3 frozen retrieval/eval baseline、authorized feedback binding、CurationSnapshot persistence、profile compatibility、C1 预注册样本与阈值、受权内容 inspection。Termbase 激活还依赖 tokenizer/index rebuild cost evidence。 + +### 3.3 Hit Test / direct return / feedback-improve：人工质量运营但非门禁 + +**上游路径 + permalink** + +- Hit Test 输入 `query_text/top_number/similarity/search_mode`，返回 Paragraph 与 similarity/comprehensive score：[`apps/knowledge/serializers/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/knowledge.py)。 +- direct return 由 document method/threshold 决定，并把匹配内容直接作为回答：[`apps/application/flow/step_node/search_knowledge_node/impl/base_search_knowledge_node.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/flow/step_node/search_knowledge_node/impl/base_search_knowledge_node.py)、[`apps/application/chat_pipeline/step/chat_step/impl/base_chat_step.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/chat_pipeline/step/chat_step/impl/base_chat_step.py)。 +- 点赞/点踩、原因：[`apps/chat/serializers/chat_record.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/chat/serializers/chat_record.py)。 +- improve 会把 chat answer 复制为新 Paragraph、关联 Problem、再 embedding：[`apps/application/serializers/application_chat_record.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/application/serializers/application_chat_record.py)。 + +**映射的本仓 seam / ADR** + +- 手动 explorer 必须调用最高已激活的 `ContextRuntime.resolve`/HTTP-generated SDK seam；不能直查 index。 +- 权威 evaluation 只能由 ADR-0080 私有 run executor 自行组合 `dogfood-loopback-resolve-acquire-v1` 并从它亲自获取的 ContextPackage 构造 security observation：[ADR-0080](../decisions/0080-refuse-authoritative-evaluation-without-an-executor.md)。 +- golden expectation 绑定 exact Evidence lineage，stale lineage 整份报告 REFUSED 而非记 recall miss：[ADR-0082](../decisions/0082-recover-the-golden-corpus-and-refuse-stale-lineage.md)。 +- canary/rollback 都是新的 generation-bound ReleaseCandidate，通过 [ADR-0033](../decisions/0033-promote-organization-releases-through-one-learning-owner.md) 同一 promote。 + +**Room-A 行为规格** + +#### Manual Hit Explorer + +- 输入：authenticated invocation、trusted delivery context、query、PackageBudget 和可选的**只收窄**请求；operator 不能覆盖 server active profile、Organization、principal、audience、threshold 或 security counter。 +- 输出：一个真实 `ResolutionOutcome` 的 content-bearing inspection：authorized Evidence title/snippet（限 projection）、citation/lineage、排名与可公开 score、budget usage、coverage/refusal、active release/profile refs；denied candidate 内容/score/count 不显示。 +- 可以把本次 query 提交为 `GoldenCaseCandidate`，但必须由 maintainer 后续补 expectation、partition/slice、hard negatives 和 exact lineage，再走 lock/intake。 +- manual result 状态仅为 `exploratory_success | exploratory_refusal | unavailable`。UI 禁止展示 `release_pass` 或把单次命中标成 regression gate。 + +#### Direct return 独立实现 + +- 行为名称建议改为 `verbatim_authorized_block`，明确它仍是交付一种已经 exact-authorized 的 ContextPackage 内容，不是 index 直出。 +- 策略属于 immutable `AssemblyProfile`/`RuntimeProfile`：eligible Evidence kind、minimum calibrated score、tie behavior、最大 blocks/tokens、citation rendering、abstention、audience/egress policy。document/Fragment mutable flag 无权覆盖。 +- 输出必须保留 Package/Evidence provenance、purpose、TTL、audience 和 citations；没有匹配 Evidence 时只给 canonical refusal，不允许 designated fallback 生成无 Evidence 内容。 +- 新策略先成为 ReleaseCandidate，在 frozen golden slices 上比较 baseline/candidate。Security/Reliability/Quality/Budget 四门独立；security 一票否决。 + +#### Feedback → improvement + +```text +feedback_received -> bound_to_authorized_run -> triaged + -> golden_case_candidate | curation_annotation_candidate | no_action + -> separately evaluated -> release candidate -> promoted +``` + +- feedback 必须绑定 exact Organization、ContextRun、Package digest、active release ref/generation、Evidence citations、actor、category、created_at；只有 authorized-only ContextRun 内容可以进入 Learning。 +- thumbs-up/down、closed reason 和自由文本是信号，不是真值。自由文本按敏感内容处理；不得携带 denied candidate 或 DecisionAudit detail。 +- “把回答加入知识”只生成 candidate。模型回答不是 source truth；没有 source Evidence 的 claim 不得自动成为 Fragment。需要新 source content 时，走 3.1 preview/confirm；需要 alternate query/tag/term 时，走 3.2 annotation audit。 +- candidate admission、golden lock、executor evaluation、release promotion 四个权限分离。ContextControl 不发布，feedback handler/triager/evaluator 不发布。 + +#### Golden slices、报告、canary 与 rollback + +- 最小 slices：精确指称、问答不对称、单文档/跨文档、时序、粒度完整性、冗余、负例拒答、安全、direct-return eligible/ineligible、feedback-derived candidate。 +- retrieval judge 用 deterministic recall@k/MRR/claim support；LLM blind judge 只提供 blind score/critical contradiction/produced claims，不能携 observed/security 字段。 +- executor seam unreachable、response malformed、没有 executor observation、lineage stale → `REFUSED`；观测到任一 unauthorized evidence/missing-context fallback/wrong-audience binding → `FAIL`；绝不靠 quality score 抵消。 +- canary manifest 固定 corpus/profile/snapshot 和 traffic/audience boundary，报告 generation 与 baseline digest。canary 不新增 publication owner。 +- rollback 选择兼容历史 immutable profiles 创建**新 candidate**，重新授权 promote，generation 继续递增；不回写历史 pointer，不绕过当前 authority/freshness。 + +**operator-visible signals** + +- Explorer：content-bearing authorized Evidence、query、release/profile refs、budget、coverage/refusal；普通导出默认 content-free，显式授权才能导出正文。 +- Golden report：case/slice counts、set/pilot/lineage map digests、release ref、executor seam、PASS/FAIL/REFUSED、四门、阈值来源、uncertainty、stale count、commands；不打印 corpus path或具体 stale refs。 +- Feedback inbox：category、run/package/release digests、citation count、triage state；内容 inspection 单独授权。任何“已改进”只表示 candidate recorded，不表示 production changed。 +- Canary：baseline/candidate generation、exposure count、slice delta、security events、budget/latency、stop/rollback decision；不得展示 aggregate win 替代各门。 + +**接口形状草图** + +```python +class ManualHitExplorer: + def run(self, invocation: AuthenticatedInvocation, + delivery: TrustedDeliveryContext, + request: Acquire) -> ResolutionOutcome: ... + def propose_case(self, call: AuthorizedGoldenIntakeCall, + run_ref: ContextRunRef, reason: TriageCategory) -> GoldenCaseCandidateRef: ... + +@dataclass(frozen=True) +class FeedbackBinding: + organization_ref: OrganizationRef + run_ref: ContextRunRef + package_ref: ContextPackageRef + package_digest: Sha256 + release_ref: ReleaseManifestRef + release_generation: int + citations: tuple[EvidenceRef, ...] + +class FeedbackIntake: + def record(self, call: AuthorizedFeedbackCall, + binding: FeedbackBinding, signal: FeedbackSignal) -> FeedbackRef: ... + def triage(self, call: AuthorizedTriageCall, + feedback: FeedbackRef, decision: TriageDecision) -> CandidateRef | NoAction: ... + +class EvaluationRunExecutor: + def execute(self, golden_set: GoldenSet, blind_judgments: BlindJudgeDocument, + tracked_thresholds: TrackedThresholds, report_at: datetime) -> GoldenReport: ... + # 不接受 callback/client/transport/counters/security_result +``` + +**测试 oracle 清单** + +- Manual Hit Explorer 与 canonical HTTP/generated SDK 对同请求给出相同 Package security fields；直查 index 的 UI/import 依赖测试失败。 +- 注入 hostile denied CandidateRef：denied bytes 进入 explorer/rerank/assembly/direct-return 计数为 0；public UI 不泄漏 denied score/count。 +- 单次/百次 manual test、手工标“通过”均不能构造 `ReleaseEvaluation` 或调用 promote。 +- file-only run 或 caller 自报 zero counters：报告必须 `REFUSED(no_run_executor_security_observation)`；executor 不接受 callback/client/transport。 +- blind judge 含 observed/security/refusal 字段：整份运行拒绝。 +- stale expected Revision/Fragment：case 不进入 judge、报告 REFUSED，不计 retrieval miss；不打印 ref/path。 +- direct-return candidate 对 authorized Evidence 保留 Package lineage/citations；无 Evidence、过阈值但 lineage 不完整、wrong audience 时返回 canonical refusal。 +- feedback 缺 run/package/release/citation binding、跨 Organization 或 generation stale：candidate=0、golden change=0、index effect=0。 +- triaged feedback 只能构造 candidate；`tests/unit/test_feedback_has_no_publication_authority.py` 与 `tests/integration/test_feedback_has_no_publication_authority.py` 必须证明 pointer DML=0。 +- canary 任一 security event 立即 FAIL/stop；rollback 产生新 generation 和 success audit，历史不可变。 + +**验证命令** + +```bash +make test +make ui-test +make smoke +make eval-v1-execute # 需配置 durable golden/backup roots、lock、judgments、lineage map +make integration +make security-gate +``` + +`make eval-v1` 的 file-only report 可以计算分层指标，但按 ADR-0080 不能产出权威非 REFUSED 结论；release gate 必须使用 executor-owned `make eval-v1-execute`。 + +**工作量与依赖** + +- 估算：**20–31 engineer-days**（manual explorer 收口 3–5，direct-return profile/ablation 6–9，feedback UX/triage 4–6，golden slices/executor/canary/rollback tests 7–11）。不含收集足够真实 golden cases 的日历时间。 +- 依赖：bounded Runtime HTTP seam、private golden/backup roots 与 lineage recapture、tracked thresholds、release operator、authorized feedback intake。direct return 只有在 frozen dataset 上证明收益且不破坏四门后才可激活。 + +### 3.4 Embedding / document-save chain：改成 lease-bound embed-before-publication + +**上游路径 + permalink** + +- document save 在事务中写 Document/Paragraph/Problem/mapping，`@post` 返回后调用 refresh 触发 embedding：[`apps/knowledge/serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py)、[`apps/common/utils/common.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/common/utils/common.py)。 +- Celery task 以 document/paragraph ID 调 listener：[`apps/knowledge/task/embedding.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/task/embedding.py)。 +- listener 修改 mutable status，删除旧 embeddings、batch save、finally 汇总状态：[`apps/common/event/listener_manage.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/common/event/listener_manage.py)。 +- vector store 对 normalized text 调模型并直接写 Embedding rows：[`apps/knowledge/vector/pg_vector.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/vector/pg_vector.py)。 + +静态观察只证明调用和状态形状；事务提交与 Celery enqueue 的 crash window、delete/recreate 期间在线可见性、exception 后状态正确性和并发覆盖均为 `[未取证]`。 + +**映射的本仓 seam / ADR** + +- [ADR-0066](../decisions/0066-embed-fragments-before-publication.md)：新 Fragment 在 activation 前必须具有完整、固定维度、有限、非零且顺序正确的 embedding；provider failure 不留可发布 Revision/Fragment。 +- ADR-0037/0040/0041、Implementation Design §6.1：durable job + exact WorkerLease、outbox/检查点、`acquired → prepared → ready → active` 恢复。 +- ADR-0018：新旧 Revision 用 pointer swap 隔离；index 只做候选发现，不是 authorization。 + +**Room-A 行为规格** + +```text +accepted job + -> acquired(canonical bytes + exact profile identity) + -> embedding_requested(exact Fragment document) + -> prepared(Fragments + validated vectors atomically persisted) + -> indexed(all candidates complete) + -> ready(authority/readiness revalidated) + -> active(single Resource pointer CAS) +``` + +- `EmbeddingProvider.embed_batch` 输入 source-ordered contextual Fragment texts 和显式 provider/model/dimension/batch profile；provider endpoint/key/timeout 只来自 worker environment，不进入 record/repr/log。 +- Worker 每个 durable effect 前核对 WorkerLease 的 Organization/job/source/resource/revision/workload/operation/generation/expiry/nonce 与当前 durable row；不能仅凭 job ID 或 Celery process identity。 +- dimension 在 schema profile 中唯一固定。transport/status/parse/count/order/dimension/non-finite/float32-zero-vector 任一失败折叠为 content-free unavailability。 +- 只有新或 replacement acquisition 调 provider；unchanged classification embedding calls=0。恢复自 `prepared/ready` 只复用已存 vectors，不重复收费调用。 +- complete embedding document 与 immutable Fragment rows 在同一 transaction 写入并推进到 `prepared`。任一向量缺失时全部不写；不允许逐 Paragraph success 就在线可见。 +- indexing 和 activation 都再次拒绝 missing/wrong-dimension vector。旧 Revision 在整个准备期间保持 active。 +- external provider 和 deterministic CI twin 都需显式配置；生产无 twin fallback。test twin 只证明协议/确定性，不声称语义质量。 +- 任何 LLM enrichment（related query/tag/term/summary）与 embeddings 分开：embedding 是已批准 representation 的派生值；LLM enrichment 必须先走 proposed/audit/snapshot/profile，绝不借 embedding job 直入 production index。 +- active 后 candidate 仍是 content-free `CandidateRef`；Runtime 必须走 `AuthorizationKernel → AuthorizedProjection`，vector score 不授权。 + +**operator-visible signals** + +- content-free：JobRef、Resource/Revision refs、lease generation、state、Fragment/vector counts、profile/provider/model refs（非 secret）、dimension、batch count、attempt count、generic failure category、checkpoint times、old/new active refs。 +- content-bearing：只有受权 candidate inspection 能查看 contextual Fragment text；worker logs、queue、metrics、failure report 不打印 source text/provider response/secret。 +- readiness 明确区分 `provider_unavailable`、`invalid_embedding_document`、`lease_stale`、`authority_revoked`、`profile_incompatible`、`ready`，但 public refusal 不泄露对象是否存在。 + +**接口形状草图** + +```python +class EmbeddingProvider(Protocol): + def embed_batch(self, request: EmbeddingBatchRequest) -> EmbeddingBatchDocument: ... + +@dataclass(frozen=True) +class EmbeddingBatchRequest: + profile_ref: EmbeddingProfileRef + revision_ref: ContextRevisionRef + fragments: tuple[ContextualFragmentInput, ...] # source order + +class SupplyPublicationPort(Protocol): + def prepare(self, lease: WorkerLease, acquired: AcquiredCompilation, + embeddings: EmbeddingBatchDocument) -> PreparedRevisionRef: ... + def index(self, lease: WorkerLease, prepared: PreparedRevisionRef) -> ReadyRevisionRef: ... + def activate(self, lease: WorkerLease, ready: ReadyRevisionRef, + expected_active: ContextRevisionRef | None) -> PublicationReceipt: ... +``` + +**测试 oracle 清单** + +- unchanged acquisition → provider calls=0，active lineage 和 index rows 不变。 +- N Fragments 必须返回 N 个同序、固定维度、float32-normalized、finite、nonzero vectors；count/order/dimension/NaN/Inf/zero 任一异常 → Fragment/Revision/pointer publication effects=0。 +- provider timeout/status/invalid JSON 不泄漏 endpoint/key/body；记录统一 content-free category；旧 Revision 可读。 +- crash after provider、during prepare、after prepared、after ready、before/after CAS：reclaim 仅发更高 generation；旧 lease/nonce mutation=0；prepared/ready recovery provider calls=0。 +- revoked Membership/source、disabled Source、tombstoned Resource、expired lease 在 activation 前重检后 pointer=0。 +- production missing explicit provider mode、选择 CI twin 或 dimension drift：启动/工作开始前拒绝。 +- new active Revision 的每个 Fragment 恰有一个有效 vector；historical nullable rows不能被新发布路径复用为 active。 +- vector hit 注入 cross-org/denied candidate：正文进入 rerank/assembly=0；index filters 不成为授权证据。 +- LLM generated Problem/Tag/Term 未 audited/promoted 时，embedding/index writes=0。 + +**验证命令** + +```bash +make lint +make typecheck +make test # provider validation、deterministic twin、worker lease +make smoke +make integration # tests/integration/test_fragment_embeddings.py + publication recovery +make security-gate +``` + +**工作量与依赖** + +- 现有 ADR-0066 主链已实现时，补齐 preview/curation carrier 的增量估算：**7–12 engineer-days**（candidate embedding document 2–3，lease/recovery integration 2–4，status/UX 1–2，fault/security tests 2–3）。若从零实现完整链：**18–26 days**。 +- 依赖：固定 schema dimension、explicit external provider config、deterministic CI twin、File job/checkpoint/reclaim、real PG17 + pgvector harness、profile/release compatibility。 + +### 3.5 产品形状反蓝图：serializer policy、mutable truth 与空测试桩 + +**上游路径 + permalink** + +- 大 serializer 同时写业务行、做事务、状态、task dispatch 和 embedding：[`apps/knowledge/serializers/document.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/serializers/document.py)。 +- mutable `Document/Paragraph/Problem/Tag/Termbase/Embedding`：[`apps/knowledge/models/knowledge.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/models/knowledge.py)。 +- backend test stub：[`apps/knowledge/tests.py`](https://github.com/1Panel-dev/MaxKB/blob/32b2d885e47ad04639abd7a18490bf5937f9c072/apps/knowledge/tests.py)；固定 checkout 中 11 个 Django app 的 `tests.py` 均为三行桩。 + +**映射的本仓 seam / ADR** + +- Deep Modules：ContextControl / ContextRuntime / ContextLearning；Runtime sealed，发布权单一。 +- ADR-0018 immutable publication、ADR-0033 release owner、ADR-0080 executor-owned evaluation、Implementation Design §8.3 highest test seams。 + +**Room-A 行为规格** + +- 状态：`untrusted_request → trusted_command → module_outcome → presented_response`；任何一步失败都终止，不能以 serializer fallback、默认 Organization、latest mutable row 或 direct DML 继续。 +- 输入：router 只收 untrusted DTO 与 trusted ingress 已构造的 invocation ref；Module 自己加载当前 Organization、authority、profile、job/release generation。输出只为 typed outcome/receipt/refusal，不返回可供 caller 再写 pointer 的内部 store 对象。 +- failure modes：认证/租户/authority 缺失、DTO 非 canonical、base generation stale、Module unavailable、transaction rollback、evidence selector 未执行，分别折叠为 closed refusal 或 gate FAIL；不得由 view/decorator catch 后标 success。 +- HTTP serializer/router 只能：解析非受信 DTO、调用 trusted ingress 构造的 authority、映射 Module outcome；不得直接读写 release pointer、授权表、index、CurationSnapshot 或 security audit。 +- 每个 effect 只有一个 owner 和一个 durable transaction boundary：Supply publish Revision；Learning promote ReleaseManifest；Control mutate source/access/policy；Runtime resolve；UI/feedback/evaluator 都无替代路径。 +- 所有 active capability 必须有 catalog applicability、activation coverage 和 executed PASS/FAIL 三维证据。存在测试文件、manual click、mock、green process 或 serializer validation 不能代替。 +- Room-B 不按 MaxKB class/table/API 逐一翻译；先实现本文 contract/property，再选本仓术语与结构。 + +**operator-visible signals** + +- 正常操作面只显示 operation ref、Module、状态、generic refusal category、base/new generation、digest、commit/audit receipt 与时间；内容页面继续服从各自单独的 content-bearing inspection authority。 +- release/evidence 面显示实际 collected selectors、skip/xfail/xpass/failed counts、capability tier、真实 seam 和 hard-oracle counters；“tests.py exists”“manual passed”“mock green”不能渲染为 PASS。 +- restricted security detail 只进 DecisionAudit；public error、ordinary log 和 Learning report 不显示 raw denied content、SQL/provider exception、secret、内部 row existence 或跨 Organization count。 + +**接口形状草图** + +```python +@router.post("/revision-previews") +def create_preview(body: UntrustedPreviewBody, + invocation: TrustedOperatorInvocation) -> PreviewResponse: + command = preview_ingress.authorize_and_map(invocation, body) + return presenter.render(revision_preview.create(**command)) + +# 路由层没有 store/session/vector/provider/release-pointer 参数。 +``` + +**测试 oracle 清单** + +- architecture/import test 禁止 UI/router/serializer 引用 release store、Kernel internals、vector adapter 或 direct DML。 +- privilege test 证明 Control/Runtime/Supply/UI/evaluator credentials 无 release pointer DML；Learning login 也只能经专用 definer promote。 +- 每个 active invariant selector 必须实际 collected 且无 skip/xfail/xpass；active-but-unexecuted=FAIL。 +- deterministic fake 只提升 domain/contract 层，不能标 sandbox/live；真实 PG17、wire、provider 分层报告。 +- candidate/feedback/manual test/LLM result 的 active pointer delta 恒为 0，直到 valid TrustedPromotionCall commit。 + +**验证命令** + +```bash +make lint +make typecheck +make test +make catalog +make smoke +make integration +make security-gate +make check # release 前最终合集；先启动真实 DB harness +``` + +**工作量与依赖** + +- 估算：**6–10 engineer-days**，用于 architecture guards、privilege tests、catalog mappings 和 empty-stub replacement；不含前四项业务实现。 +- 依赖：schema security manifest、evidence registry、真实非 owner roles、明确的 module import boundaries。 + +## 4. 不可借鉴清单与必须杀死的隐含前提 + +| 学习行为 | ContextEngine 独立实现 | 必须杀死的隐含前提 | +|---|---|---| +| preview 后 confirm | digest-bound immutable candidate；confirm 只创建 WorkerLease-bound job；ADR-0018 完整 pointer swap；ADR-0033 再 promote release | mutable Document row 同时是 source truth、current index 和 active release；用户点导入就立刻改变在线行为 | +| preview 暂存 source file | Organization/actor/purpose/TTL-bound preview blob + immutable audit + explicit expiry cleanup | 拥有临时 file ID 就有读取、confirm 或发布权；临时路径可以进日志 | +| Paragraph 可编辑 | 新 Revision + immutable Fragments；`tests/integration/test_zz_file_publication_recovery.py` 证明 old-or-new | 原地编辑内容还能保留可靠 provenance/citation；Paragraph 可独立授权 | +| Problem/related query | proposed `alternate_query` annotation，citation validation、human audit、CurationSnapshot、on/off eval | LLM 生成的问题天然正确，创建 row 后可立即提高 recall；命中次数是真值 | +| Tag | source metadata 与 curated annotation 分型、分别版本化 | 一个 mutable key/value row 可同时代表来源事实、人工治理、权限和 release behavior | +| Termbase | accepted term snapshot → 新 Compilation/Index profile → rebuild/eval/promote | 改一个术语 row 可以无版本、无重建地改变 tokenizer 与生产检索；索引派生状态是 source truth | +| Hit Test | canonical Runtime/HTTP seam 的 exploration；可提交 golden candidate | 手工 query 命中等于 regression/release gate；直查 index 与 Runtime 行为等价 | +| direct return | versioned Assembly/Runtime profile，只直出 ContextPackage 内完整 authorized Evidence，保留 citation | similarity threshold 是授权；document flag 可绕过 Kernel/egress；无命中时可输出无 Evidence fallback | +| 点赞/点踩 | binding-complete feedback signal → triage candidate | 用户反馈是真值、可跨 release 复用；自由文本可以直接进 Learning 或日志 | +| feedback improve | source candidate 或 CurationAnnotation candidate；`test_feedback_has_no_publication_authority` 证明零发布权 | 把模型 answer 复制为 Paragraph 并 embedding 就完成知识改进；模型内容等于 source evidence | +| document save 后 async embedding | ADR-0066 embedding document 与 Fragment 同事务进入 prepared；exact WorkerLease；旧 Revision 保持 active | DB commit 后 task dispatch 永不丢；job ID/Celery worker 是权限；部分 embeddings 可在线 | +| delete old embedding then rebuild | build-new → validate → index → CAS active；superseded artifact retained | 删除/重建窗口对读者天然原子；finally 状态等于真实成功 | +| mutable status string | append-only job events + durable checkpoints + separate publish watermark | 一个 row 的多位状态同时是操作进度、readiness、release truth 和审计历史 | +| index/filter | content-free CandidateRef → AuthorizationKernel → AuthorizedProjection；`test_candidate_security_regressions.py` 等 hostile oracle | knowledge/workspace filter、is_active 或 vector hit 已完成最终授权 | +| serializer/view 权限与 policy | trusted ingress + Deep Module interface + DB privileges/definer | transport decorator/serializer 校验就是不可绕过 security boundary；换 transport 可复制一套策略 | +| ContextControl 产品配置 | Control 只管 source/access/policy；profile 只由 Learning promotion | 配置后台天然拥有 profile publication；初始化/migration 可 seed active pointer | +| evaluator PASS | ADR-0080 executor-owned observation；报告无 publication authority | caller counters、空 events、callback/no-op client 可证明 clean；高 quality 可抵消 security | +| golden retrieval miss | ADR-0082 lineage map check；stale case 排除且整报 REFUSED | unresolvable expectation 等于模型 miss；可以用部分非 stale cases 出 release 分数 | +| canary/rollback | generation-bound candidate + same `ContextLearning.promote` + append-only audit | rollback 可直接改历史 pointer；A→B→A 可重用旧 candidate/base digest | +| 空 tests.py 与 UI 手测 | highest public seam + real PG17/non-owner/FORCE RLS + explicit hard-oracle counters | 测试文件存在、功能可点或 mock 通过就是 backend/security evidence | +| GPLv3 实现形状 | Room-A 本文 + Room-B 独立命名/代码/schema/tests；ADR-0074 | “只改一点”“只复制接口/SQL/UI”不算复制；依赖许可证能覆盖 MaxKB 粘合实现 | + +落实时，以下 repo tests/ADR 是最小“杀前提”集合：ADR-0014/0018/0033/0066/0080/0082；`tests/unit/test_release_owner_architecture.py`、`tests/unit/test_feedback_has_no_publication_authority.py`、`tests/unit/test_eval_run_executor.py`、`tests/unit/test_stale_lineage_detector.py`、`tests/integration/test_fragment_embeddings.py`、`tests/integration/test_release_promotion.py`、`tests/integration/test_worker_lease.py`、`tests/integration/test_zz_file_publication_recovery.py`。新增实现应扩展这些最高 seam，而不是另建只测 serializer 的平行套件。 + +## 5. 推荐实现顺序 + 给 coordinator 的开放问题 + +### 推荐顺序 + +1. **先固化 Room-B 输入边界。** 将本文作为唯一 MaxKB 深挖输入；Room-B 人员不读 checkout。先写 nominal contracts、state machine、closed refusals、architecture/privilege tests，确认没有 MaxKB 名称/schema/API/SQL/UI 逐字迁移。 +2. **先做最小 preview 候选，不接 publication。** 完成 deterministic `RevisionPreviewRef`、content-free 列表、受权 content inspection、edit-as-new-digest、expire/abandon。以 `active Revision delta=0`、`active Release delta=0` 验收。 +3. **接现有 Supply publication。** confirm 只建 durable job；复用 exact WorkerLease、ADR-0066 embedding、publication recovery、old-or-new CAS。不要创建第二个 queue/process 或另一套 publication state machine。 +4. **把 current corpus 观察与 release gate 接通。** Revision resource-active 后生成 ADR-0073 candidate；使用 ADR-0080 executor 和 ADR-0082 lineage map；由 release operator 调唯一 promote。此时才可声称候选影响在线行为。 +5. **上线 manual Hit Explorer，但显式标“探索”。** 只走 canonical Runtime HTTP/generated SDK；支持一键生成 golden candidate，不支持手工 PASS。它既给后续 UX 反馈，也扩大真实 golden corpus。 +6. **先做一个 `alternate_query` annotation 的 C1 薄切片。** feedback binding → propose → citation validation → human audit → immutable CurationSnapshot → frozen on/off eval → curation-on ReleaseCandidate。先证明 authority/compatibility，再做 Tag/Termbase。 +7. **Tag 分型后再做 Termbase。** 先禁止 source_fact/curated 混用；Termbase 由于改变 tokenizer/index，必须最后接 profile rebuild、cost evidence 与 release gate。 +8. **direct return 最后、默认关闭。** 使用 `verbatim_authorized_block` profile，通过专门 golden slices、ablation、canary 和 rollback 才激活；失败保持普通 ContextPackage 或 canonical refusal。 +9. **最终以全门禁收口。** `make check`、executor-owned golden run、raw evidence/report、真实 PG17 security gate；任何 `[未取证]` 不升级 capability tier。 + +按最小可交付波次估算：Wave A（preview-only）8–12 days；Wave B（confirm + Supply/release）10–15 days；Wave C（explorer + feedback candidate）8–12 days；Wave D（alternate-query C1）10–15 days；Wave E（tag/term/direct-return）16–25 days。单人串行总量约 **52–79 engineer-days**，可在 contract 固定后由 UI、Supply、Learning 测试工作并行，但 release ownership 和 shared schema 仍需单一集成人负责。 + +### 给 coordinator 的开放问题 + +1. **谁拥有 preview draft 的 public application seam？** 建议它是 API process 内的窄 Curation/Intake application service，而不是扩大 `ContextControl` 的 canonical Interface；需 coordinator 确认是否写新 ADR，还是只作为内部 seam。 +2. **confirm 与 promote 是否由不同角色执行？** 建议 curation reviewer 可 confirm import，只有 release operator 可 promote；若产品要求一人完成，仍应产生两份 nominal authority 和两次审计，不能合并能力。 +3. **content-bearing preview 的保留与导出策略是什么？** 需要确定 TTL、owner-only storage、是否允许下载、离职/撤权后的清理、普通 operate log 是否只留 digest；在答案前不能实现持久 preview 默认值。 +4. **`CompilationProfile` / `AssemblyProfile` 是新增一等合同，还是现有 profile 的内部组成？** 建议 Compilation 落入 Content/Index compatibility，Assembly 落入 Runtime/Curation compatibility；不要新增 active pointer。需要架构 owner确认命名与迁移边界。 +5. **direct return 的产品语义是否允许绕过生成但不绕过 BotDelivery？** 建议仍由 BotDelivery/egress 消费 audience-bound ContextPackage，并带 citation；若要求引擎直接生成回答，则与“ContextPackage 是唯一输出、答案生成不进引擎”的边界冲突，应拒绝或新 ADR。 +6. **第一种 curation kind 是否锁定 `alternate_query`？** 它最贴近 Problem/feedback 且 rebuild 成本低于 Termbase；若先做 tag/term，需要先补 profile rebuild 和 tokenizer impact evidence。 +7. **canary 的可用 carrier 当前是什么？** 若没有受信 traffic allocation/exposure observation，报告必须保持 `[未取证]`，先做 offline release comparison + operator-controlled promote/rollback，不虚构在线 canary。 +8. **preview 前后的内容评测是否需要独立 evaluator？** 本报告建议 deterministic `RevisionReadinessEvaluator` 只做编译/readiness，权威 release evaluation 仍唯一由 ADR-0080 executor 完成；若要让 preview evaluator判“质量通过”，必须先定义 tracked seam、threat model、golden lineage 与无 publication authority。 + +在这些问题未决时，Room-B 仍可安全实施 Wave A 的 preview-only contract 和零发布权 oracles；不得提前激活 curation-on、direct return 或新的 canary claim。 diff --git a/docs/research/2026-07-31-onyx-blueprint-evaluation.md b/docs/research/2026-07-31-onyx-blueprint-evaluation.md new file mode 100644 index 00000000..ca3137de --- /dev/null +++ b/docs/research/2026-07-31-onyx-blueprint-evaluation.md @@ -0,0 +1,409 @@ +# Onyx → ContextEngine 可复刻蓝图评估（ADR-0075 lift plan） + +> **决策状态**：本文开放问题已由维护者于 2026-07-31 全部决定（D2/D3/D4），结果见 [`five-repository-implementation-blueprint.md`](./2026-07-31-five-repository-implementation-blueprint.md) §5；正文推荐项为评估时刻的状态。 + +# 1. 固定 commit 与许可证核验 + +本评估只针对 Onyx commit [`2fb3dd10493b3883870fa8adced5b1a0e114feff`](https://github.com/onyx-dot-app/onyx/commit/2fb3dd10493b3883870fa8adced5b1a0e114feff)。已在 `/tmp/onyx-bp` 以 detached HEAD 核验 `git rev-parse HEAD` 等于该值；以下 permalink 均固定到该 commit，而不是浮动分支。 + +根 [`LICENSE`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/LICENSE) 明确规定：所有 `ee` 目录内内容受 Onyx Enterprise License 约束，目录外内容适用 MIT Expat。固定 checkout 中存在的 `ee` 根为 `backend/ee`、`backend/tests/external_dependency_unit/ee`、`backend/tests/unit/ee`、`web/src/app/ee`、`web/src/ee`；六个 lift 的拟议源文件均不位于这些目录。对拟议目录做嵌套 `LICENSE`/`NOTICE`/`COPYING` 静态扫描，未发现会改变这些文件许可证边界的嵌套文本。[一手静态] 这证明固定 checkout 的路径边界；不把它扩大为对未来 commit 或未扫描依赖的许可证结论。 + +| 拟议源路径 | 本次用途 | 固定 checkout SHA-256 | 边界结论 | +|---|---|---|---| +| `backend/onyx/tools/tool_implementations/search/search_utils.py` | weighted RRF、邻接/全量扩展、重叠合并 | `393b5909f1b40784eac1babd6a8836a003d18f272bf0a35d4bb4970211b28943` | 非 `ee/`，MIT；只有切割后的函数可考虑 copy+patch | +| `backend/onyx/tools/tool_implementations/search/search_tool.py` | query 去重、budget trim、检索编排 | `00d486a15705f1855f2d06ea898bb22c6d5eae16f09b2433b2376c5c3d329c43` | 非 `ee/`，MIT；整文件不得复制，依赖与正文面过宽 | +| `backend/onyx/context/search/retrieval/search_runner.py` | hybrid runner 接口行为 | `846e4f16e142355ae3f37544de33196e5f8e20fc3dfbee6eebf2e54587534587` | 非 `ee/`，MIT；实现绑定 Onyx index，不能移植 | +| `backend/onyx/document_index/interfaces_new.py` | `HybridCapable` 接口形状 | `2285d0cbedf91b109f9484325a769872dc520529c81ab50f65d8630bc1339576` | 非 `ee/`，MIT；可只复制并收窄 ABC 形状 | +| `backend/onyx/context/search/models.py` | hybrid request/expansion enum 形状 | `5732486e4e44a6b337cf5b732934666b946b96d2cf2cc8ffa81d992dc2dab4c0` | 非 `ee/`，MIT；含 `bypass_acl` 等禁用语义，不整文件复制 | +| `backend/onyx/context/search/pipeline.py` | 相邻 section 合并的行为 oracle | `d662115ed3c3fcc6256a58ea8e03f3bf692342d3ca0e7f88953dfcdc9adad584` | 非 `ee/`，MIT；函数会动态调用企业后处理，故只取无 `ee` 的局部行为 oracle | +| `backend/onyx/secondary_llm_flows/query_expansion.py` | semantic/keyword rewrite | `3639371ee18301ab209fa73682ddc227466a20ee4ceaa84f10a3a49c4b828a06` | 非 `ee/`，MIT；模型调用必须改接治理端口 | +| `backend/onyx/secondary_llm_flows/document_filter.py` | relevance selection/expansion classification | `9634888840574ac8d0c264dcf308b7d73ef0c489730b3090008526da27a2961d` | 非 `ee/`，MIT；正文消费者只能位于 Kernel 后 | +| `backend/onyx/natural_language_processing/search_nlp_models.py` | cross-encoder/API rerank 行为 | `539441523f90e0c8c5be2a7be833c8c9d0f2c65273548002640dea99914e19c9` | 非 `ee/`，MIT；provider/credential/client 实现不复制 | +| `backend/onyx/natural_language_processing/utils.py` | token count/trim helpers | `8df3950e61c41023992d9c149220c69b918454771588b29b13293fb6cbcc2bb4` | 非 `ee/`，MIT；best-effort token 语义不能成为硬预算实现 | + +企业权限同步编排不进入任何 `source_paths`。它只允许经 ADR-0074 的两室流程形成行为规格和测试 oracle；本报告不提供、复述或建议复制 `backend/ee/**` 实现。现有 `third_party/onyx/UPSTREAM.toml` 已注册 connector framework 的四个文件，且 `excluded_paths` 已含 `backend/ee`、`web/src/app/ee`、`web/src/ee`；本报告不重新提议该框架，也不把它误计为六个 retrieval lift 的新增工作。 + +# 2. 六个 lift 的状态盘点 + +三条 keystone seam 不是绿地：Supply execution/checkpoint bridge 已由 `engine/persistence/supply_execution.py`、`adapters/connectors/file.py` 和 connector-runner 落地；content-free candidate/authorized-fragment access port 已由 `CandidateDiscoverySession`、`CandidateIndex`、`AuthorizationKernel`、`FragmentWindowSession` 落地；governed model-inference port 已由 `engine/runtime/model_inference.py` 与共享 `PackageBudgetMeter` 落地。第三条 seam 只有端口和单元/HTTP-PG 证明，`STATUS.md` 明确所有 Runtime `rewrite/rerank/select` carrier 仍为 `NOT_ACTIVE`。 + +| 固定顺序 | lift | 仓内状态 | 已有证据/实现 | 仍缺的可交付工作 | +|---:|---|---|---|---| +| 1 | weighted RRF/dedupe | **核心已完成** | `fuse_candidate_evidence` 做 exact-ref 去重并携带 content-free rank evidence；`join_authorized_ranking` 在授权后压紧位置并应用 server-owned weights；ADR-0083 已禁止 pre-Kernel weighting | 用固定 Onyx oracle 做差分用例；不要再复制一个第二 fusion authority | +| 2 | hybrid retrieval adapter | **实现与真实 PG/HTTP 测试已完成，通用 served activation 仍受限** | `PostgreSQLFtsCandidateIndex` + `PostgreSQLVectorCandidateIndex` + `PostgreSQLHybridCandidateIndex`；FTS/pgvector SQL 原生执行于 retained UserActor transaction；`tests/integration/test_rank_blind_kernel_hybrid.py`、`test_hybrid_path_sealed_http.py` | 只需补接口 provenance、基准与明确的 production activation；外部 query embedding 仍需计量/不可隐式启用 | +| 3 | same-Article expansion | **安全 seam 已完成，resolve 编排未接通** | `AuthorizationKernel.expand_fragment_window`、`PostgreSQLFragmentWindowReader`、同 Article/current Revision 检查、跨 Article `reauthorization_refs`；unit + real PG tests 已有 | 将 expansion plan 接入 sealed Runtime 顺序、把跨 Article refs 回送 Kernel、再做预算与 HTTP 最高 seam 验证 | +| 4 | query rewrite | **端口能力已完成，carrier 编排绿地** | `RewriteModelRequest`/`ModelInferencePort.rewrite` 已有闭合 output、grant、timeout、usage；无 Runtime 调用方 | 固定 profile、输入来源、rewrite 数量/长度、失败策略、共享 meter、grant issuance 与最终 package usage | +| 5 | authorized rerank | **类型/治理端口已完成，正式 carrier 绿地且有 ADR 类型冲突待决** | `RerankModelRequest` 只接 `AuthorizedProjection`，输出必须是精确 permutation；HTTP/PG seam 已证明 raw CandidateRef 不到达 port | 接入 resolve、决定 ADR-0052 `AuthorizedModelInput` 与 Runtime `AuthorizedProjection` 的合法桥接、共享 meter、最终 package usage 和安全 gate | +| 6 | budgeting helpers | **基础 meter 已完成，端到端累计发布未完成** | `PackageBudget`/`PackageBudgetMeter` 原子 reserve/commit/cancel；当前 Package 构造仍直接计算 block UTF-8 bytes，并把 provider/cost/elapsed 写成 0 | 建立每 resolve 单一 meter，rewrite/rerank/expansion/assembly 共用并把累计 usage 写入最终 Package；替换 stage-local/ad-hoc 计数 | + +因此“照 lift 顺序实现”在当前分支上的含义不是重写 1→6，而是按同一顺序关闭每个 lift 的剩余 gate。特别是 lift 1、2 不应因为评估发现 Onyx 函数而退回上游数据模型或索引。 + +# 3. 逐 lift 蓝图 + +## 3.1 Lift 1 — weighted RRF/dedupe + +### 上游函数路径与 permalink + +- [`weighted_reciprocal_rank_fusion`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_utils.py#L28-L111)：按 `id_extractor` 去重，计算 `weight / (k + rank)`，以 score、首次 source rank、source index 稳定排序。 +- [`deduplicate_queries`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_tool.py#L159-L180)：大小写不敏感合并 query，保留首次 casing 并累加 weight；这是 lift 4 的输入 oracle，也能固定 lift 1 的重复列表语义。 +- [`test_search_utils.py`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/tests/unit/onyx/tools/test_search_utils.py#L25-L337)：空列表、重复、权重、tie-break 等上游 oracle。 + +### Kernel 切割线 + +上游函数 28–96 的 exact ID 去重、rank position 采集是 content-free，可映射到 pre-Kernel `CandidateRef` + `RankerEvidence`；不得携带上游 `InferenceChunk` 本体、title、content、metadata 或 score explanation。上游 83–111 的 **weighted** fused order 不能成为 pre-Kernel 有效决策：ADR-0083 已固定权重只在 Kernel 后按已授权集合压紧 rank position 后计算。Runtime 在 `fuse_candidate_evidence` 后必须把 refs 按 canonical `_candidate_sort_key` 排序送入 Kernel；只有 `join_authorized_ranking(AuthorizedProjection[], rank_evidence, server_weights)` 能形成下游顺序。denied ref 的 evidence 在 join 时丢弃，且不得影响 position compact、budget selection 或可见 trace。 + +### copy+patch vs 原生重写决定与理由 + +**决定：不新增 vendored runtime code；保留本仓原生实现，并把上游测试作为差分 oracle。** 上游 generic function 本身是 MIT、可复制，但它把 weight 放在授权前集合上，且 tie-break 受 denied candidates 的初始位置影响；复制后再把权重移到 Kernel 后会只剩算法名字相同。当前 `fuse_candidate_evidence` + `join_authorized_ranking` 已直接编码 ADR-0076/0083，新增 vendored helper 只会制造第二 fusion authority。 + +### 复刻配方 + +1. 保留 `CandidateQuery(ranked_lists)`，在 seam 入口重新运行 `require_bounded_candidate_submission`，拒绝超过 server-owned candidate/list ceiling 的 hostile object。 +2. 对每个 ranker 内以 exact `CandidateRef` 去重，保存第一次 position 与 optional finite score；输出 `FusedCandidates(candidate_refs, rank_evidence)`，不输出正文或 ACL。 +3. 将 `candidate_refs` canonical sort 后送入 sealed Kernel；rank evidence 不进 Kernel。 +4. Kernel 返回 `AuthorizationDecision.projections` 后，按 exact CandidateRef join；只对 admitted refs 为每个 ranker 重排连续 `1..N` position。 +5. 从 server-owned `RankerWeights` 读取权重，绝不接受 request 字段；计算 authorized-only fusion，稳定 tie-break 使用 `_candidate_sort_key`。 +6. 后续 selector、rerank、budget pack、assembler 只消费 `AuthorizedRerankItem` 顺序。没有 evidence 的授权 projection 使用 neutral rank；denied evidence 不进入 tenant-visible output。 + +### 测试 oracle + +- 功能：同 ref 跨 ranker 只输出一次；同 ranker 重复只计首次；输入 permutation 在同 rank evidence 下给稳定 exact-ref tie-break；缺 rank evidence 的 projection 得 neutral rank；权重缺项、非有限、非正数拒绝。 +- 安全：向两个合法 ranker 列表插入任意数量/位置的 denied 与 cross-Organization refs，授权后的合法顺序、选择和 Package bytes 必须与未插入时一致；Kernel 输入不含 rank；denied content 到 rerank/assembler 为 0。 +- 资源：在第一次 `locate()` 前拒绝超 bound submission 和伪造 `object.__new__` DTO。 + +### third_party 注册计划 + +无新增 `UPSTREAM.toml` 条目。若仅导入上游 oracle fixture，也必须作为测试派生物登记 `backend/tests/unit/onyx/tools/test_search_utils.py` 的 source hash、vendored fixture hash 与修改说明；推荐直接重写等价 fixture，避免把 Onyx `InferenceChunk` test model 引入本仓。 + +### 工作量与依赖 + +增量 **1–2 engineer-days**（oracle 补齐、provenance 注释与回归），原始绿地约 4–6 天。依赖 content-free ranked-candidate/authorized-fragment port；不依赖 Supply bridge 或 governed model port。 + +## 3.2 Lift 2 — hybrid retrieval adapter + +### 上游函数路径与 permalink + +- [`HybridCapable.hybrid_retrieval` / `keyword_retrieval`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/document_index/interfaces_new.py#L365-L425)：hybrid/keyword adapter 的输入、bounded top-N 与 score-ranked 输出形状。 +- [`_embed_and_hybrid_search`、`_keyword_search`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/context/search/retrieval/search_runner.py#L52-L87)：query embedding、hybrid/keyword dispatch 的编排行为。 +- [`search_chunks`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/context/search/retrieval/search_runner.py#L90-L166) 与 [`combine_retrieval_results`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/context/search/retrieval/search_runner.py#L28-L49)：多 retrieval function 的并发/合并形状。 +- [`BasicChunkRequest` / `ChunkIndexRequest`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/context/search/models.py#L142-L169)：query、alpha、limit、filters 的上游 request 形状；其中 `bypass_acl` 语义明确禁止带入。 + +### Kernel 切割线 + +只保留以下 pre-Kernel 数据：bounded query plan、query embedding、trusted removal-only scope、每个 native ranker 返回的 `CandidateRef` 和 rank position/score。`InferenceChunk` 的 content/title/metadata、Onyx ACL list、tenant ID、highlight 和 score explanation 均不得跨 candidate seam。Runtime 自己在 retained UserActor transaction 上执行 prepared request，把 primitive refs 放进一次性 `CandidateDiscoverySession`；replaceable index 永远拿不到 connection、projection session、locator 或 projector。Kernel 后才 exact join rank evidence 与 `AuthorizedProjection`。 + +### copy+patch vs 原生重写决定与理由 + +**copy+patch：**只抽取 `HybridCapable` 的接口形状，压缩成 dependency-free、content-free 的 runner-side ABC/DTO；删除 `InferenceChunk`、ACL filter、tenant、score-bearing content 和 index management 方法。 + +**原生重写：**FTS、pgvector、同 transaction execution、scope filter、HNSW settings、ranker composition 全部留在 ContextEngine。绝不复制 Onyx Vespa/OpenSearch 查询、schema、SQL、client 或 deployment。当前 `PostgreSQLFtsCandidateIndex`、`PostgreSQLVectorCandidateIndex`、`PostgreSQLHybridCandidateIndex` 已实现正确方向：FTS 使用 PostgreSQL `websearch_to_tsquery/ts_rank_cd`，vector 使用 pgvector `<=>`，两者只返回 lineage refs。 + +### 复刻配方 + +1. 在 `third_party/onyx/retrieval/interfaces.py` 放入经裁剪的 `HybridRetrievalShape.prepare(query, limit, filters)` / `shape_results(content_free_rows)`；其 wire DTO 只允许 opaque refs、position 和有限 score。 +2. CE adapter 将 `Acquire` + `CandidateDiscoveryScope` 转为 `HybridDiscoveryRequest(fts, vector)`；每个子 limit 与总 limit 必须小于等于 server submission bound。 +3. Runtime 对 request 做 hostile revalidation，然后在 retained `MaterializedProjectionSession` 内执行。FTS SQL只选择 organization/source/resource/revision/fragment；vector SQL同样只选择 lineage，query embedding profile 必须与已发布向量 profile 匹配。 +4. FTS/ANN 都先应用 Kernel-computed EffectiveScope 的 removal-only projection及 caller narrowing，再 `LIMIT`；这不是授权，返回 ref 仍逐个过 Kernel。 +5. 把两个结果集标成独立 `fts`/`vector` ranked lists，交 lift 1 携带 evidence；禁止在 adapter 内返回融合正文或做授权判断。 +6. external query embedding 只有在同一个 resolve-owned budget meter 上计 provider call/cost/elapsed 且 profile identity 被 ReleaseManifest 固定后才能 served；否则只允许 network-free twin 并报告 `NOT_ACTIVE`。 + +原生实现 skeleton： + +```text +prepare_discovery(Acquire, CandidateDiscoveryScope) -> HybridDiscoveryRequest +Runtime.execute(retained_user_actor_tx, request) -> CandidateDiscoverySession +discover(Acquire, data_only_session, scope) -> CandidateQuery[ + RankedCandidateList("fts", CandidateRef...), + RankedCandidateList("vector", CandidateRef...) +] +canonical refs -> AuthorizationKernel -> AuthorizedProjection +``` + +### 测试 oracle + +- 功能：FTS lexical 命中与 pgvector neighbor 各自稳定；同 ref 双路命中只在 lift 1 合并；limit/total bound 生效；embedding dimension/profile mismatch 在 SQL 前拒绝；deterministic tie-break。 +- real dependency：PostgreSQL 17 + pgvector 下验证 FORCE RLS、active Revision、tombstone、source/resource narrowing、selective filter 下 iterative scan；记录 `EXPLAIN ANALYZE`、underfill、exact-vs-ANN recall，不将静态分析写成 benchmark 通过。 +- 最高 seam：HTTP `resolve(Acquire)` 证明 `CandidateRef → Kernel → AuthorizedProjection`；在 authorized corpus 混入 same-Org denied 与 cross-Org candidates，denied content 到 rerank/assembler=0，响应与不混入时的合法内容一致。 +- capability：关闭 external embedding 时零 network bytes；provider failure 映射统一 content-free unavailable，不回显 query/vector/locator。 + +### third_party 注册计划 + +若采用接口形状 copy+patch，在现有 `UPSTREAM.toml.source_paths` 增加： + +- `backend/onyx/document_index/interfaces_new.py`，upstream SHA-256 `2285d0cbedf91b109f9484325a769872dc520529c81ab50f65d8630bc1339576`；vendored target `third_party/onyx/retrieval/interfaces.py`。 +- 如 DTO 确需来源追踪，再加 `backend/onyx/context/search/models.py`，upstream SHA-256 `5732486e4e44a6b337cf5b732934666b946b96d2cf2cc8ffa81d992dc2dab4c0`；target `third_party/onyx/retrieval/models.py`。 + +`files[].sha256` 必须记录 **post-patch vendored hash**，原始 hash 写入 `MODIFICATIONS.md`；修改说明列明删除的 index/ACL/content/tenant 方法。`excluded_paths` 保持全部 `ee` 根，并新增明确“不包含 `backend/onyx/document_index/vespa/**`、`opensearch/**`”。若最终继续使用当前 CE-native Protocol，不为“灵感”伪造 vendored 注册。 + +### 工作量与依赖 + +现有实现上的增量 **3–5 engineer-days**（接口 provenance 1、benchmark 1–2、activation gate 1–2）；外部 embedding served 另计 3–5 天。依赖 ranked-candidate/authorized-fragment port；governed model port只在 external query embedding 被归入模型推理时依赖；Supply bridge不依赖。 + +## 3.3 Lift 3 — same-Article expansion + +### 上游函数路径与 permalink + +- [`_retrieve_adjacent_chunks`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_utils.py#L129-L205)：按 document ID 和 chunk range 取邻居。第 149–151 行显式假设初始权限已检查、扩展无需再检查，是必须删除的 anti-pattern。 +- [`merge_overlapping_sections`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_utils.py#L208-L348)：同 document 相邻/重叠 section 合并并保留首次顺序。 +- [`expand_section_with_context`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_utils.py#L351-L494)：main/adjacent/full-document 四类 expansion。 +- [`merge_individual_chunks`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/context/search/pipeline.py#L130-L244)：按 document/chunk ordinal 合并相邻结果的补充 oracle。 + +### Kernel 切割线 + +expansion plan（anchor exact ref、before/after、候选邻居 refs）可 content-free；任何邻居正文读取都必须在 Kernel 内或 Kernel 授予的窄 `FragmentWindowSession` 后。已授权 Article（`ContextResource`）的 **current active Revision** 内，邻居经 organization/source/resource/revision lineage 和 source ACL projection freshness 验证后可继承 Article decision并构造 `AuthorizedProjection`。一旦候选跨 source/resource（即跨 Article），只能输出 content-free `CandidateRef`，回到完整 Kernel 重授权；Fragment 永远没有独立 ACL。revision 不匹配不是跨 Article fallback，而是 stale lineage，拒绝。 + +### copy+patch vs 原生重写决定与理由 + +**决定：原生实现；上游只作 behavior oracle，不 vendoring。** 上游函数直接从 `DocumentIndex` 取 content，显式关闭 ACL filter，并以 `document_id` 作为长期权限继承依据；这与 ADR-0077 的 Article/current Revision atom、retained transaction 和 hostile port snapshot verification不兼容。纯 merge 算法虽可复制，但当前 CE window 是 ordinal-sorted exact projections，原生十余行 merge 更易审计且不需要 Onyx content model。 + +### 复刻配方 + +1. post-Kernel ranking/selection 先选择 anchor `AuthorizedProjection`；构造 `FragmentWindowRequest(anchor, before<=32, after<=32, expansion_candidates<=64)`。 +2. `AuthorizationKernel.expand_fragment_window` 用 anchor locator 在同一 retained UserActor transaction 读取权威 window；SQL join `ContextResource.active_revision_id`、`tombstoned=false`，按 Fragment ordinal 取窗口。 +3. 对每一 item 验证 organization/source/resource/revision 全等、source ACL projection ref/as-of 全等、field ceiling 不扩张；从 anchor decision 继承并构造 active `AuthorizedProjection`。 +4. replaceable `FragmentWindowReader` 只见一次性 data session；Kernel 对其返回与 authoritative snapshot 做 primitive exact comparison，防内容/lineage mutation。 +5. `expansion_candidates` 中 source/resource 不同的 refs 去重后进入 `reauthorization_refs`；同 Article ref 若被错误送去 reauthorize，直接拒绝 seam contract。 +6. Runtime 对 `reauthorization_refs` 调用完整 `authorizeAndProject`；成功 projection 才能与 inherited projection 合并。跨 Article denied 详情只进 restricted DecisionAudit category/digest。 +7. 最终按 Article+Revision+ordinal 合并重叠窗口，保持首次 authorized rank；然后交共享 budget meter 和 assembler。 + +### 测试 oracle + +- 功能：window 上下界、anchor 在首尾、重叠窗口去重、ordinal 排序、current Revision replacement 后旧 anchor 拒绝、tombstone/disabled source 拒绝。 +- lineage：同 Article+same Revision 继承；same Article+old Revision 不继承；cross Article 一定进入 Kernel；伪造 source ACL ref/as-of、field ceiling、body 或 locator 时 authoritative comparison 拒绝。 +- 安全：authorized anchor 周围放 denied other-Article candidate，窗口 reader/merge/assembler 看到的 denied body=0；跨 Article reauthorization 拒绝后不能用 anchor Article decision补位。 +- 最高 seam：real PG + HTTP resolve 覆盖 expansion 后 Package Evidence 一一闭合，revocation/Revision activation 与 window read 并发时只见完整旧或完整新，绝不混版。 + +### third_party 注册计划 + +无新增 vendored 条目。若未来复制纯 `merge_overlapping_sections`，必须单独登记 `search_utils.py` 原始 hash `393b...943`，target 只能是 `third_party/onyx/retrieval/merge_sections.py`，并在 `MODIFICATIONS.md` 证明输入类型已限制为已授权 projection view；但当前建议是不复制。 + +### 工作量与依赖 + +剩余 **4–6 engineer-days**：resolve 编排 2、cross-Article loop 1、budget/merge 1、real PG+HTTP/security gate 1–2。依赖 ranked-candidate/authorized-fragment port；若 expansion classification 使用模型则再依赖 governed model port；不依赖 Supply bridge。 + +## 3.4 Lift 4 — query rewrite + +### 上游函数路径与 permalink + +- [`_build_additional_context` / `_build_message_history`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/secondary_llm_flows/query_expansion.py#L23-L65)：上下文和 history 规范化。 +- [`semantic_query_rephrase`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/secondary_llm_flows/query_expansion.py#L68-L147)：生成 standalone semantic query。 +- [`keyword_query_expansion`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/secondary_llm_flows/query_expansion.py#L150-L227)：生成最多三条 keyword query 的上游意图（实现本身未强制三条，CE 必须强制）。 +- [`SearchTool._expand_queries_and_decide_scope`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_tool.py#L588-L633) 及 [`query mix/dedupe`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_tool.py#L865-L910)：并行 rewrite、缓存、query 权重与去重行为。 + +### Kernel 切割线 + +rewrite 输出是 query text，不是 source content，可在 Kernel 前作为 retrieval input；但它仍是模型 egress，不能绕过 profile、EgressGrant、timeout、trace 和 shared `PackageBudgetMeter`。输入只允许当前 closed `Acquire.need.query` 及已明确授权进入该 hop 的对话上下文；不得注入 candidate body、Provider metadata、denied details、raw `TrustedDeliveryContext` 或 arbitrary memory。rewrite 只产生 bounded strings，随后作为多个 `CandidateQuery` 的 query plan；它不能扩大 `EffectiveScope` 或选择 source authorization。 + +### copy+patch vs 原生重写决定与理由 + +**copy+patch：**可抽取“semantic + keyword 两类输出、case-insensitive query dedupe、保留 original query”的纯编排形状。 + +**原生重写：**prompt、history DTO、模型调用、trace、cache、日期注入全部由 CE profile/port 实现。Onyx 函数接受通用 `LLM`、任意 user_info/memories，并在空输出时采用不一致 fallback；直接复制会形成第二未治理模型口。现有 `ModelInferencePort.rewrite` 已是合法 gateway，应成为唯一调用点。 + +### 复刻配方 + +1. 定义 server-owned `RewriteProfile`：exact model/provider/region/retention、1 input、最多 1 semantic + 3 keyword outputs、每条字符/UTF-8/token bound、one-shot/no retry、timeout/cost ceiling。 +2. Runtime 创建每 resolve 唯一 `PackageBudgetMeter`，在任何模型 bytes 前 reserve；构造 `RewriteModelRequest(profile, original_query)`，禁止传 content-bearing object。 +3. 通过 exact model EgressGrant 和 `ModelInferenceEgressBinding` 调 `ModelInferencePort.rewrite`；grant mismatch/replay、budget 不足、timeout、malformed output 均返回同一 content-free unavailable。 +4. 对输出做 Unicode/whitespace canonicalization，拒绝空串、超长、超数量、duplicate JSON key；casefold exact 去重，保留 first spelling；将 original query 作为固定 ranker 输入，不允许模型删掉唯一 retrieval path。 +5. 为每个 rewrite 生成独立 bounded FTS/vector request；总 candidate submission仍受 ADR-0083 seam-local bound，模型不能控制 limit、weights 或 scope。 +6. 不做跨请求/跨 audience cache。若做 request-local reuse，key 必须包含 profile digest、original query digest、purpose/audience/epoch，并只保存 content-free output。 +7. final Package 发布同一个 meter 的累计 usage；不得像当前构造那样把 provider calls/cost/elapsed固定为 0。 + +### 测试 oracle + +- 功能：semantic/keyword/original query 去重；大小写重复合并；输出顺序稳定；最多三条 keyword；malformed/empty/oversize model output拒绝；rewrite 不能改变 source/resource narrowing。 +- 模型治理：wrong model/provider/region/audience/purpose/package digest、expired/replayed grant、timeout、第二 provider call均为 provider bytes=0或不再增加；trace只含 digest/category/usage。 +- 安全：尝试把 CandidateRef、AuthorizedProjection、denied detail 或 caller-authored tenant/audience传入 rewrite DTO均在 gateway 前拒绝；rewrite output不得进入 audit 作为 raw query。 +- budget：rewrite + hybrid query embedding +后续 rerank共用同一 meter；任一 reserve超限不允许 stage-local meter重试，最终 Package usage等于累计值。 + +### third_party 注册计划 + +推荐不复制上游 prompt/LLM code。若仅复制 query dedupe helper，则登记 `search_tool.py` 原始 hash `00d486...29c43`，target `third_party/onyx/retrieval/query_dedupe.py`，`MODIFICATIONS.md` 写明删除 `SearchTool`、LLM、scope、logging、cache 与 content DTO。`query_expansion.py` 仅在确实复制解析/编排代码时加入 source path，hash `363937...8a06`；不能为了引用 permalink提前登记空文件。 + +### 工作量与依赖 + +**5–8 engineer-days**：profile/grant composition 2、Runtime plan 1–2、shared meter/final usage 1–2、tests/gate 1–2。依赖 governed model-inference port和 ranked-candidate port；不依赖 Supply bridge。 + +## 3.5 Lift 5 — authorized rerank + +### 上游函数路径与 permalink + +- [`cohere_rerank_api`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/search_nlp_models.py#L680-L699)、[`cohere_rerank_aws`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/search_nlp_models.py#L702-L738)、[`litellm_rerank`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/search_nlp_models.py#L741-L760)：query+passages → per-input relevance score 的 provider shape。 +- [`RerankingModel._make_direct_rerank_call` / `predict`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/search_nlp_models.py#L1207-L1312)：provider/local-server 路由和 score order。 +- [`select_sections_for_expansion`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/secondary_llm_flows/document_filter.py#L187-L360)：content-bearing relevance selection 的 LLM形状；不是可直接复用的授权实现。 + +### Kernel 切割线 + +绝对切线是 `AuthorizationKernel` 输出。query、CandidateRef、rank evidence 可在前；passage/body/title/metadata/relevance prompt 一律只能由 `AuthorizedProjection` 或一个从 **单一、当前、audience-bound ContextPackage** 构造的 nominal `AuthorizedModelInput` 提供。denied candidates 的 body、field、rank evidence不得进入 rerank request，模型输出只能重排或选择输入集合的精确成员，不能发明 ref。rerank 后仍由 PackageBudget、provenance、final egress 和 audit gate封口。 + +### copy+patch vs 原生重写决定与理由 + +**决定：provider client与模型路由全部不复制；原生接到 governed model port。** Onyx functions直接持有 API key、URL、provider fallback和普通 `list[str]`，没有 audience/package/grant/budget nominal boundary。可借鉴的只有“输入顺序与 score index一一对应”oracle。当前 CE `RerankModelRequest` 已强制 `AuthorizedProjection`、结果是 exact permutation，比复制 provider wrapper更接近目标。 + +存在一个必须先决议的类型冲突：ADR-0052 把 `AuthorizedModelInput` constructor固定在 BotDelivery，并从一个完整 ContextPackage构造；当前 Runtime rerank port则固定接 `AuthorizedProjection`，发生在最终 Package之前。任务约束要求 lift 5 使用前者。实现不得暗中创造第二种同名 nominal type；推荐在新 ADR 中选择并证明以下桥接：Kernel先形成一个内部、不交付、audience-bound的 pre-rerank ContextPackage，使用独立 model EgressGrant构造唯一 `AuthorizedModelInput`；模型只能返回其 Evidence的 permutation/subset；随后 Runtime用同一 shared meter形成最终 Package与独立 final-hop grant。若 maintainers 不接受内部 Package，该 lift 必须保持 `NOT_ACTIVE`，并由 ADR 明确 Runtime专用 projection model input，而不是声称已满足 ADR-0052。 + +### 复刻配方 + +1. `join_authorized_ranking` 先丢弃 denied evidence，只保留 active `AuthorizedProjection`；在此之前禁止任何 tokenizer/model访问正文。 +2. 对已授权集合做 deterministic pre-pack，形成单一当前 audience/purpose/epoch绑定的 pre-rerank Package；Block/Evidence一一闭合、expiry和digest完整，不向 caller交付。 +3. 用 ADR-0052 constructor从该 Package + exact model EgressGrant + closed query envelope + versioned rerank profile生成 nominal `AuthorizedModelInput`；同 resolve 的 `PackageBudgetMeter`先 reserve最大 input/output/call/cost/elapsed。 +4. governed gateway只序列化该 input 的 authorized Block text/Evidence refs；不含 raw identity、grant、denied detail、arbitrary context。one-shot grant redemption成功后才发 provider bytes。 +5. 输出解析为输入 Evidence index的 exact permutation（或 profile允许的 unique subset）；长度、indices、duplicates、invented citation/ref任一异常都拒绝。不得接受上游 float scores直接作为授权或公开字段。 +6. reranked projections/blocks进入 deterministic PackageBudget assembly；final Package只能是 pre-rerank Package Evidence的子集/重排，且 `budgetUsage`发布 shared meter累计值。 +7. final egress重新使用与最终 Package digest匹配的独立 grant；model grant与final channel/model hop grant不可复用。 +8. 模型 unavailable 的策略必须在 profile中显式冻结。安全默认是该 lift unavailable而非静默调用未治理 fallback；若允许 deterministic retrieval-order fallback，必须零 provider bytes且记录 closed category，不能隐藏预算或授权失败。 + +### 测试 oracle + +- 核心安全 oracle：authorized set中混入 same-Org denied、cross-Org和不存在 candidates，实际进入 rerank gateway和assembler的 denied content bytes **= 0**；合法 request payload与未混入时相同。 +- type：raw CandidateRef、duck-typed projection、expired projection、两个 Packages、wrong audience/purpose/epoch Package均不能构造 `AuthorizedModelInput`。 +- egress：wrong/replayed/expired grant、wrong provider/model/region/retention、database failure均 provider bytes=0；audit failure不释放最终 Package。 +- output：exact permutation、stable tie、no invented refs；duplicate/out-of-range/missing index、NaN score、oversize output拒绝。 +- budget：reservation发生在 redemption/provider之前；timeout或parse失败按固定最大用量charge；最终 Package usage等于 meter累计，不能重置为 0。 +- highest seam：real PG + HTTP/generated SDK完成 `CandidateRef → Kernel → AuthorizedProjection → AuthorizedModelInput → final ContextPackage` tracer，security-gate注册上述混入场景。 + +### third_party 注册计划 + +无新增 provider代码。若只复制 score/index normalization oracle，登记 `search_nlp_models.py` 原始 hash `539441...e19c9` 并将极窄纯函数放入 `third_party/onyx/retrieval/rerank_normalization.py`；明确删除 Cohere/AWS/LiteLLM credentials、HTTP clients、fallback与logging。推荐继续使用当前 CE exact-permutation parser，不产生该 vendored文件。 + +### 工作量与依赖 + +**7–10 engineer-days**，另加类型桥接 ADR/评审 **1–2 天**。依赖 governed model-inference port、ranked-candidate/authorized-fragment port；不依赖 Supply bridge。若内部 Package方案被否决，此工作量不可视为可开工承诺，carrier保持 `NOT_ACTIVE`。 + +## 3.6 Lift 6 — budgeting helpers + +### 上游函数路径与 permalink + +- [`_estimate_section_tokens`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_tool.py#L183-L210) 与 [`_trim_sections_by_tokens`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/tools/tool_implementations/search/search_tool.py#L213-L255)：按顺序装箱，遇首个不适配 section即停止；metadata用固定 75 token估算。 +- [`count_tokens`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/utils.py#L191-L210)：大文本分片计数并可 early exit。 +- [`split_text_by_tokens` / `tokenizer_trim_content`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/natural_language_processing/utils.py#L213-L247)：best-effort split/trim；上游注释承认重新 tokenize不保证硬上限和可能产生 replacement character。 +- [`test_projects_file_utils.py`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/tests/unit/onyx/server/test_projects_file_utils.py#L284-L362)：chunked count/early-exit oracle。 + +### Kernel 切割线 + +预算 ceiling和模型调用 reserve可在正文前；对某 projection的 token/byte计数、trim和packing是 content-bearing，只能消费 `AuthorizedProjection`/authorized Package Block。pre-Kernel不得通过 token length、oversize或trim result观察 denied content。最终 Package必须发布所有 active stages共用的一个 `PackageBudgetMeter.usage`，不是只发布 block bytes；调用方请求只能缩小 server ceiling。 + +### copy+patch vs 原生重写决定与理由 + +**决定：保留 CE-native PackageBudget/Meter，最多把上游 early-exit行为写成测试，不复制 helper。** 固定 `METADATA_TOKEN_ESTIMATE=75`、`max_tokens<=0`返回原 sections、BPE边界漂移和replacement character均不满足安全硬预算。CE当前 tokenizer profile是 versioned contract，必须精确计数；reserve/commit/cancel还要覆盖 provider call、cost、elapsed，Onyx helper没有这些维度。 + +### 复刻配方 + +1. Runtime在完成有效 budget intersection后创建唯一 `PackageBudgetMeter(effective_budget)`，由 resolve拥有，不暴露给 caller或replaceable candidate port。 +2. rewrite/query embedding/rerank/selection分别在外部调用前原子 reserve其profile maximum；成功commit actual，调用已发生但结果异常则按冻结策略charge maximum，调用前拒绝则cancel。 +3. content packing接 `AuthorizedProjection[]`，用 ReleaseManifest固定的 tokenizer/version精确计算每个 block和provenance overhead；不得估算 private metadata或读取 denied item长度。 +4. deterministic first-fit策略必须明示是 skip-oversize继续还是首个不适配即stop；建议继续扫描，避免一个大 authorized block饿死后续小block，并以 canonical authorized rank保持确定性。 +5. trim只在结构允许的边界进行；不可切断 citation/provenance/UTF-8 scalar。不可切的 block超限则跳过并记录 authorized-only gap category，而非best-effort破坏文本。 +6. Package构造从 `meter.usage`取累计值，并验证usage不超过effective ceiling；ContextRun持久化同一值。当前 `construction.py` 直接重建 `BudgetUsage(tokens=block bytes, others=0)` 必须被替换。 +7. final package digest覆盖 usage；任何 stage-local meter、usage重置、provider call未计量均由construction/type test拒绝。 + +### 测试 oracle + +- 功能：精确边界 `== limit`通过、`+1`拒绝/跳过；Unicode、长文本chunk、结构不可切 block；deterministic packing；caller cap只缩小。 +- concurrency：两个 stage并发 reserve总和超限时只有一个成功；cancel不泄漏reservation；double commit/cancel拒绝。 +- security：denied大正文与不存在ref不能改变usage、coverage、packing顺序或timing claim；content consumer bytes=0。 +- failure accounting：grant拒绝前零usage，provider已调用后parse/timeout按策略charge；最终 Package/ContextRun usage一致，digest mutation被拒绝。 +- real seam：HTTP Package `budgetUsage`与模型 trace receipts、assembly bytes可独立重算；security-gate把budget绕过作为veto，不以quality抵消。 + +### third_party 注册计划 + +无新增条目。若将 `count_tokens` 的大输入 early-exit纯算法复制为非权威优化，登记 `natural_language_processing/utils.py` 原始 hash `8df395...cc2bb4`，target `third_party/onyx/retrieval/token_count.py`，并由权威 tokenizer在边界附近重新精确计数；它不能决定硬预算。当前建议完全原生。 + +### 工作量与依赖 + +剩余 **4–6 engineer-days**：resolve-owned meter wiring 2、assembler/Package contract 1–2、failure/concurrency/HTTP tests 1–2。依赖 governed model port和 authorized-fragment access port；Supply bridge不依赖。 + +# 4. 次要评估 + +## 4.1 `tests/README.md` layering 到本仓 test pyramid + +Onyx [`backend/tests/README.md`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/tests/README.md#L3-L59) 定义四层：pure unit、real external-dependency unit、full deployment integration、Playwright E2E。映射如下： + +| Onyx 层 | ContextEngine 合法对应 | 命令归属 | 六 lift 用法 | +|---|---|---|---| +| Unit，无外部服务 | `tests/unit/` domain、DTO、算法、hostile object、import/type边界 | `make test` | RRF、bounds、exact permutation、meter concurrency、no-content capability graph | +| External dependency unit，真实 PG/Redis/MinIO/Vespa、无 app process | `tests/integration/` 中真实 PostgreSQL 17 + pgvector、non-owner roles、FORCE RLS；可直接调用最高公开 Python seam | `make integration`（需先 `make db-up`） | FTS/pgvector、same-transaction discovery、window lineage、release CAS、WorkerLease | +| Full deployment integration，优先 HTTP、无 mock | 本仓 real PG + API/worker process/HTTP/generated SDK vertical slice；数据层仍在 `tests/integration/`，process boot在 `tests/process/` | `make integration` + `make smoke`；完整组合由 `make check` | hybrid resolve、rewrite/rerank final Package、worker runner、revocation并发 | +| Web E2E | 当前没有独立通用 `make e2e`；UI/SDK/Bot/Action tests与未来浏览器E2E分别管理 | `make ui-test`、`make sdk-test`、`make bot-test`、`make action-test`；未来应加显式E2E target | Evidence Console/browser只验证产品流，不替代 Kernel/PG oracle | + +`make security-gate` **不是第五个scope层**，而是跨层的release veto：只执行catalog注册的精确security evidence并产出独立报告，通常需要真实数据库已经up。一个unit/integration/E2E绿灯只有被catalog注册且hard oracle被观测时才构成对应安全证据；skip、未执行和`NOT_ACTIVE`不能算PASS。`make test`不应暗含数据库，`make integration`不应被fake替代，`make smoke`只证明boot/readiness。 + +## 4.2 `db/swap_index.py` future/present readiness → ReleaseManifest 行为规格 + +Onyx静态行为：`SearchSettings.status`有 `FUTURE/PRESENT/PAST`；[`_port_swap_ready`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/db/swap_index.py#L196-L219) 等待required ports全部成功且metadata backlog清零；[`check_and_perform_index_swap`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/db/swap_index.py#L222-L336)按INSTANT/REINDEX/ACTIVE_ONLY决定切换；[`_perform_index_swap`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/db/swap_index.py#L40-L171)更新旧/新状态。其 `update_search_settings_status` 每次调用都会独立 `commit`，所以旧→PAST与新→PRESENT不是一个事务；而index verify还发生在状态切换之后。这只能作为clean-room行为输入，不能复制。 + +ContextEngine readiness/promotion行为规格： + +1. Candidate必须引用一个immutable `ReleaseManifest`，完整绑定content/index/runtime/curation profile refs+digests、tokenizer/package schema、compatible/active Revision refs；没有“默认 PRESENT”。 +2. `ContextLearning.evaluate`只生成immutable evaluation，检查四个独立gate、所有active Revision的FTS/vector/profile readiness、无pending publication gap、schema compatibility和security veto；它无pointer权限。 +3. `promote`输入绑定exact Organization、manifest/candidate/evaluation digests、expected base digest与expected active generation；fresh first activation期待generation 0和不存在pointer。 +4. non-owner Learning transaction锁定Organization release state，重新读取并重算immutable lineage与readiness，验证current operator grant、expiry、signature、四gate和compatibility。 +5. **一个数据库事务**内做generation-bound CAS active pointer、generation+1、success audit append；任一失败提交0 pointer change与0 success audit。不存在先把旧版设PAST再激活新版的窗口。 +6. 旧Manifest/activation event保持immutable；rollback创建选择历史manifest的新candidate，重新evaluate/promote，不能反向改status。`A→B→A`仍有新generation，旧candidate不能ABA复活。 +7. Runtime只读exact active pointer与完整manifest；missing/mixed/stale lineage fail closed，绝不fallback到上一版或“future”。cleanup与readiness不是authorization。 + +必须测试：fresh absence；not-ready不切换；wrong Org/operator/direct DML=0；并发两个promote只有一个成功；事务故障pointer/audit同回滚；A→B→A拒绝stale generation；Runtime并发只见完整old或new；rollback走同一promote；任何安全gate失败不可被quality分抵消。 + +## 4.3 missing-tenant fallback anti-pattern checklist + +Onyx [`TenantAwareTask.__call__`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/background/celery/apps/app_base.py#L100-L118) 在`tenant_id`缺失/falsey时回落 `POSTGRES_DEFAULT_SCHEMA`；[`on_task_postrun`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/background/celery/apps/app_base.py#L150-L204) 在kwargs缺失时也回落默认schema；[`on_task_revoked`](https://github.com/onyx-dot-app/onyx/blob/2fb3dd10493b3883870fa8adced5b1a0e114feff/backend/onyx/background/celery/apps/app_base.py#L255-L276) 同样从task kwargs回落。ContextEngine suite要断言的不是“默认值不同”，而是以下 exact anti-behavior 全部不可能： + +- 缺失/空/错误类型Organization、job、operation、source、registered service workload、Policy Epoch、lease generation、nonce、issued-at/expiry中的任一项，**在source root、connector、database business function、index、checkpoint store前拒绝**。 +- task payload或环境中的 tenant/Organization不能替代server-minted、签名且exact job绑定的 `WorkerLease`；不能用触发用户身份、默认Organization、默认schema或上一次context。 +- wrong workload/job/source/operation/Organization、expired/stale/superseded/replayed lease业务效果=0：无文件scan、无Provider call、无DB row变更、无checkpoint/publish watermark推进、无job完成、无Redis/default-schema cleanup。 +- postrun/revoke/cleanup也必须携带可验证exact binding；缺kwargs不得对默认tenant taskset做`srem`或其他“善后”。cleanup failure不能创造另一个authority。 +- worker复用进程和connection pool前后都清空context；下一任务缺context时拒绝，不能继承前一Organization。并发A/B任务与pool checkout/checkin stress必须证明cross-Organization read/write=0。 +- rejection audit只保留lease/job digest与closed category，不记录raw token、path、denied object或tenant枚举细节；caller看到统一`WorkNotAvailable`。 +- runner不得独立持久化checkpoint/index/cache；checkpoint只是proposal，只有engine在exact lease transaction中durably accept整页后才推进。 + +现有 `test_supply_bridge_lease.py`、`test_connector_runner_lease.py`、`test_connector_runner_isolation.py` 已覆盖其中大部；新增lift或connector必须复用同一suite，而非把“有 tenant_id 参数”当合格。 + +# 5. 不可借鉴清单与必须杀死的隐含前提 + +| Onyx形状/隐含前提 | ContextEngine必须采取的相反约束 | +|---|---| +| 部署Onyx service、DB、Redis、Vespa/OpenSearch作为第二产品 | 不部署；ContextEngine PostgreSQL 17 + pgvector/FTS是唯一corpus/index/policy/revocation truth | +| 非 `ee` 就可以整目录复制 | 仍须逐路径license/nested notice/依赖扫描、固定commit、hash、MODIFICATIONS、SBOM；依赖纠缠和安全不兼容仍可否决复制 | +| `ee/` permission-sync可因行为有价值而复制 | 绝不复制；只允许两室clean-room行为规格/test oracle | +| `IndexFilters.access_control_list`、`tenant_id`、`bypass_acl`是最终授权 | index filter只做removal/defense-in-depth；caller永远不能bypass；每个ref过sealed Kernel | +| `InferenceChunk`可作为pre-Kernel search result | pre-Kernel只有opaque `CandidateRef`与content-free rank evidence；正文/title/path/metadata/score explanation禁止 | +| first hit做过权限检查，所以同document expansion无需再检查 | 只有same Article+current Revision+fresh lineage可继承；跨Article一律新CandidateRef重授权，stale Revision拒绝 | +| document/chunk ID本身足以继承权限 | Article/ContextResource是唯一atom；Fragment无ACL，ID/index presence不授予任何权限 | +| pre-Kernel weighted RRF order可直接交付 | Kernel rank-blind；授权后对admitted positions压紧并用server weights重新fusion；denied positions不影响输出 | +| hybrid alpha、limit、filter、weight可由request/model任意给出 | server-owned profile和seam bounds；request只能收窄scope/budget，不能选择authority或放大work | +| Vespa/OpenSearch hybrid实现或SQL可换壳移植 | 只复用interface shape；FTS/pgvector query、HNSW、scope filter原生实现并在retained transaction执行 | +| 模型wrapper持API key/URL并可直接调用或fallback | 唯一governed port；exact profile、one-shot EgressGrant、timeout、shared meter、digest-only trace；失败不降级到未治理模型 | +| arbitrary user_info/memory/history可进入rewrite | closed input envelope；无candidate/provider/denied content，无caller-authored trusted audience/tenant | +| rerank普通`list[str]`输入且float score可信 | 只从单一audience-bound Package构造`AuthorizedModelInput`；输出只能是输入Evidence exact permutation/subset；score不授权也不公开 | +| 固定75 metadata token、BPE best-effort split足以做hard budget | Release-bound tokenizer精确计数；结构边界trim；provider/cost/elapsed与tokens共用atomic meter | +| 每stage各自budget或最后只数block bytes | 每resolve一个meter，所有stage累计，最终Package和ContextRun发布同一usage | +| FUTURE/PRESENT用两次commit切换，切换后再verify index | readiness先验证；一个Learning-owned transaction做generation CAS pointer+audit；失败0可见变化 | +| missing tenant落默认schema，postrun/revoke也可默认 | 任一exact WorkerLease/ActorContext字段缺失即fail closed，source/DB/effect=0；绝无默认Organization/schema | +| index/cursor/checkpoint推进等于内容已发布/授权 | acquisition checkpoint、publish watermark、active Revision、Policy Epoch完全分离；任何一个都不单独授权 | +| 日志可打印query、filter、document IDs、provider异常 | denied/object/query内容不进入ordinary trace；只保留closed category/digest和bounded usage | +| 并行query和重试天然安全 | 总candidate/model call有server bound；one-shot grant无透明retry；并发reserve与job replay由durable authority裁决 | +| 上游unit/integration数量等于能力已验证 | 每项claim绑定本仓unit/real PG/HTTP/E2E层和catalog security oracle；[未取证]保持[未取证] | +| Onyx search tool可负责最终assembly/answer | ContextPackage assembly保持engine-native、sealed budget/provenance/audit；answer generation在BotDelivery且另过ADR-0052边界 | + +# 6. 推荐实现顺序 + 给 coordinator 的开放问题 + +顺序保持ADR-0075不变，并按当前完成度执行： + +1. **weighted RRF/dedupe**：不重写，先用Onyx固定oracle补足差分与provenance，冻结“pre-Kernel只携带、post-Kernel才weight”。退出条件是denied candidate位置无法改变合法顺序。 +2. **hybrid retrieval adapter**：登记或明确拒绝接口形状vendoring，运行native PG FTS+pgvector真实依赖/HTTP/security验证与benchmark；不碰Vespa/OpenSearch。退出条件是bounded data-only session和mixed-denied oracle通过。 +3. **same-Article expansion**：把现有Kernel window接入resolve，完成cross-Article reauthorization loop、merge和budget。退出条件是same Article/current Revision继承、cross Article重授权、old Revision/tombstone拒绝均由real PG+HTTP证明。 +4. **query rewrite**：在governed port上固定profile/grant/共享meter与bounded multi-query plan；保持`NOT_ACTIVE`直到final Package累计usage可验证。 +5. **authorized rerank**：先解决ADR-0052 `AuthorizedModelInput`与Runtime projection port的类型/时序冲突，再接carrier；混入denied candidate→rerank/assembler content=0是security veto。 +6. **budgeting helpers**：最后统一替换ad-hoc Package usage，令1–5所有调用共享一个resolve meter，并把累计usage写入final Package/ContextRun；完成后才能声称model-backed lift active。 + +按现有分支的增量估算合计 **24–37 engineer-days**（不含真实provider认证、生产credential、长期benchmark/corpus标注）；三条seam的已投入成本不重复计算。每一lift独立PR/ADR gate，且只有完成`make test`、适用的`make integration`、最高HTTP/generated SDK proof与catalog `make security-gate`后才能改变`STATUS.md`。 + +给 coordinator 的开放问题： + +1. **rerank类型桥接（阻塞lift 5 activation）：**是否接受“内部、不交付的audience-bound pre-rerank ContextPackage → ADR-0052 AuthorizedModelInput → final Package”的两Package时序？若不接受，需要新ADR明确Runtime专用nominal input，且必须解释为何任务约束中的AuthorizedModelInput不适用；不能维持当前两种叙述同时声称active。 +2. **lift 2 provenance：**要不要实际vendor极窄`HybridCapable` ABC形状，还是认可当前CE-native Protocol加permalink/hash为“behind copied interface shape”的等价实现？前者增加升级/attribution成本但更贴ADR字面，后者代码更浅。 +3. **budget tokenizer：**最终 Package的`tokens`是否继续定义为`utf8-byte-token-v1`，还是在model-backed lift前引入ReleaseManifest绑定的真实tokenizer？这决定lift 6是简单统一meter还是包含schema/OpenAPI migration。 +4. **rewrite failure策略：**governed rewrite unavailable时，是整次resolve closed unavailable，还是允许profile显式声明“零provider bytes、original-query-only”的deterministic fallback？两者都可安全实现，但必须在activation前冻结并进入golden slices。 +5. **same-Article扩展位置：**先做deterministic fixed window（复用已完成seam）还是同时引入governed relevance classification？建议先fixed window，避免lift 3暗中依赖lift 5并破坏固定顺序。 + diff --git a/docs/research/2026-07-31-openviking-blueprint-evaluation.md b/docs/research/2026-07-31-openviking-blueprint-evaluation.md new file mode 100644 index 00000000..b2367249 --- /dev/null +++ b/docs/research/2026-07-31-openviking-blueprint-evaluation.md @@ -0,0 +1,626 @@ +> **Room-A 研究产物 — 维护者本地研究，非公开 provenance；第五仓准入已由维护者于 2026-07-31 有条件批准（D9），公开基线修订待 legal 复核与 permalink 固定完成。其余开放问题决定（D8/D10）见 [`five-repository-implementation-blueprint.md`](./2026-07-31-five-repository-implementation-blueprint.md) §5；正文推荐项为评估时刻的状态。** + +# 1. 固定 commit、许可证核验与证据基线状态 + +## 研究固定点 + +- 上游仓库：`volcengine/OpenViking`。 +- 固定 commit：[`49b182045b42d34ad530948ad77d9d0226897da8`](https://github.com/volcengine/OpenViking/commit/49b182045b42d34ad530948ad77d9d0226897da8)。本地以 `git show -s --format='%H%n%cI%n%s'` 核对的提交时间为 **2026-07-31T11:38:57+08:00**，提交标题为 `refactor(parser): Refactor code summaries to fixed skeleton-first routing (#3568)`；研究日期为 **2026-07-31**。 +- 本文所有 OpenViking 结构性判断只以该 commit 的固定链接为证据。后续上游行为、文档或许可证变化不自动更新本文结论；`main` 链接不构成本文证据。 +- 研究方法：Room A 只观察文档、接口形状、行为与测试 oracle，不复制 AGPL 实现；没有运行 OpenViking 的联网服务，也没有把上游依赖或源码带入 ContextEngine。 + +## 路径级许可证核验 + +| 上游区域 | 固定证据 | 2026-07-31 核验结论 | ContextEngine 处置 | +|---|---|---|---| +| 仓库根与 Python 主项目 | 根 [`LICENSE`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/LICENSE) 是 GNU AGPL v3；[`pyproject.toml`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/pyproject.toml) 自报 `AGPL-3.0`；[`README.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/README.md) 也称 Main Project 为 AGPLv3 | **AGPLv3 区域**。除经单独许可证覆盖的路径外，`openviking/`、`openviking_cli/`、`web-studio/`、`bot/`、主项目测试与文档所描述的实现均按 AGPL 边界处理 | 严格 clean-room：仅行为规格、接口形状与测试 oracle；零源码复制、零派生实现、零主项目运行时依赖 | +| `crates/` 父目录 | [`crates/LICENSE`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/crates/LICENSE) 是 Apache License 2.0；README 称 `crates/ov_cli` 为 Apache-2.0 | 父目录给出 Apache-2.0 证据，但不能覆盖子项目 manifest 的相反声明 | 逐文件、逐 manifest 核验后才可能 copy+patch；不能把整个 `crates/` 当作已清洁素材池 | +| `crates/ov_cli` | [`crates/ov_cli/Cargo.toml`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/crates/ov_cli/Cargo.toml) 自报 `MIT`，与 [`crates/LICENSE`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/crates/LICENSE) 和 [`README.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/README.md) 的 Apache-2.0 声明冲突 | **许可证边界未消歧，当前不可复制**。不能任选对项目更方便的一种许可证解释 | 需要上游/maintainer/legal 明确适用许可证、精确 source path 和所需 notice；在此之前只作 strategic-reference-only | +| `examples/` 父目录 | [`examples/LICENSE`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/LICENSE) 是 Apache License 2.0；README 称 examples 为 Apache-2.0 | 没有下层相反声明的具体文件，才有 Apache-2.0 初步依据；这不是对子树所有文件的无条件覆盖 | 每个候选需核对 manifest、SPDX、嵌套 LICENSE、依赖锁和实际 source path 后再登记 | +| `examples/openwebui-plugin` | [`pyproject.toml`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/openwebui-plugin/pyproject.toml) 自报 AGPL-3.0；[`tools.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/openwebui-plugin/openviking_openwebui/tools.py) 带 AGPL-3.0 SPDX | 子项目明确为 **AGPL-3.0**，不得因父目录 `examples/LICENSE` 而误判为 Apache | clean-room only，禁止 copy+patch | +| 其他 examples | 例如 [`opencode-plugin/package.json`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/opencode-plugin/package.json) 自报 Apache-2.0，而 [`openclaw-plugin/package.json`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/openclaw-plugin/package.json) 自报 MIT；[`claude-code-memory-plugin/package.json`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/claude-code-memory-plugin/package.json) 没有项目 license 字段 | examples 内存在多种或缺省声明，必须缩小到确切文件集合。依赖的 license 不是项目源码的 license | 当前无 copy+patch 提案；需要复用时另开 path-level 审批，不从本报告推定许可 | + +AGPL 网络服务影响必须按工程红线理解：AGPLv3 第 13 节要求，若修改后的 AGPL 程序通过网络与用户交互，运营者须向这些用户提供获得该版本 Corresponding Source 的明确机会。ContextEngine 的 API/worker 是网络服务；复制、修改或依赖 OpenViking 主项目服务代码会把源码提供义务带到我们的网络交付面，并产生与 ADR-0074 受控复用政策不相容的风险。因此所有 AGPL 区域一律 clean-room，不能以“没有分发二进制”为由放宽。此结论是仓库工程治理边界，不代替法律意见。 + +## 证据基线状态与准入条件 + +OpenViking **不是** [`2026-07-19-four-public-repositories-evidence.md`](./2026-07-19-four-public-repositories-evidence.md) 的四仓之一。本文只能影响 maintainer-local 推理；在准入前，公开设计、README、ADR 或产品说明不得以 OpenViking 作为 provenance，也不能把本文链接当作公开证据。 + +若要成为第五仓，至少需要一次显式的 maintainer admission，且在同一变更中完成： + +1. 固定仓库、完整 SHA、研究日期和路径级许可证矩阵，并由 maintainer/legal 处理 `ov_cli` 与 examples 的冲突或排除这些路径； +2. 将每一项拟公开 claim 改写为可核验、有限的结构性陈述，并逐项链接到该 SHA 的文件/测试 permalink；无法固定取证的内容保留 **[未取证]**，不得以推断补齐； +3. 修订四仓证据报告为明确版本化的五仓基线，更新设计 authority、PLAN/STATUS 中的公开来源范围和 attribution；说明 OpenViking 只证明 context filesystem、memory workflow 与 exposure UX，不证明 ContextEngine 的多租户授权、安全或合规结论； +4. 登记 clean-room 边界与两室流程；如有任何 permissive copy+patch，再独立提交 `third_party//UPSTREAM.toml`、上游许可证/NOTICE、修改日志、逐文件哈希和 SBOM 覆盖； +5. 维护者复核所有公开文本不再引用浮动分支、博客截图或未固定 benchmark，并明确未来上游更新不自动继承准入。 + +# 2. 能力盘点 → ContextEngine 区域映射表 + +| OpenViking 能力 | 上游固定证据 | ContextEngine 映射区域 | 复刻分类 | 结论 | +|---|---|---|---|---| +| L0 abstract / L1 overview / L2 detail | [`context-layers.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/03-context-layers.md)、[`hierarchical_retriever.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/retrieve/hierarchical_retriever.py) | Supply 编译的 Fragment/heading ancestry；Runtime AssemblyProfile、PackageBudget、Assembler | **clean-room Room-A spec** | 采用“信息密度逐级揭示”和逐 hop 预算；删除“层级/深度产生权限”的暗示 | +| `viking://` 虚拟文件系统、`ls/tree/glob/read` | [`viking-uri.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/04-viking-uri.md)、[`filesystem.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/api/03-filesystem.md) | EffectiveScope、AuthorizationKernel、AuthorizedProjection、OpenCitation、Evidence Console | **clean-room Room-A spec** | 仅可做 post-authorization 的 browse UX 投影；不能暴露源 URI、目录存在性或 prefix 递归权限 | +| Session commit 后异步提取 memory | [`session.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/08-session.md)、[`session-memory-extraction-flow.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/design/session-memory-extraction-flow.md) | authorized-only Learning input、CurationCandidate/Annotation、CurationSnapshot、ReleaseCandidate/evaluate/promote | **clean-room Room-A spec** | 采用归档 intent、幂等队列、候选 diff；禁止模型直接 create/merge/delete active memory | +| 检索 trajectory / provenance | [`retrieval.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/07-retrieval.md)、[`test_provenance.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/tests/retrieve/test_provenance.py) | ContextRun、DecisionAudit、Package digest、Evidence Console | **clean-room Room-A spec** | 只保留 authorized-only lineage、预算与版本化 digest；拒绝 pre-auth path/query/score/count/thinking trace | +| Studio playground / Helper | [`observability.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/guides/05-observability.md)、[`openviking-helper.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/14-openviking-helper.md) | ADR-0090 co-resident local Evidence Console | **clean-room Room-A spec** | 采用同进程、timeline、health、preview/confirm UX；必须经 HTTP seam、显式 browser auth 和 one-Control-operation-per-call | +| VikingBot 的 context/model/tools/channel 一体化 | [`vikingbot.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/15-vikingbot.md) | 外部 BotDelivery、ActionPlane；Engine Runtime | **do-not-take** | 与 ADR-0006 冲突；Engine 不生成答案、不拥有渠道或工具执行，只交付 ContextPackage | +| 同进程 `/mcp`、13 tools、coding-agent hooks | [`mcp_endpoint.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/server/mcp_endpoint.py)、[`mcp-integration.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/guides/06-mcp-integration.md) | ADR-0017 sealed Runtime、OpenAPI/generated SDK、可选 thin MCP adapter | **clean-room Room-A spec** | 采用同认证、薄协议、hook 生命周期；不采用宽写面、目录直读、remember/forget 或旁路授权 | +| `crates/ov_cli` 和 examples 的客户端实现 | [`crates/ov_cli`](https://github.com/volcengine/OpenViking/tree/49b182045b42d34ad530948ad77d9d0226897da8/crates/ov_cli)、[`examples`](https://github.com/volcengine/OpenViking/tree/49b182045b42d34ad530948ad77d9d0226897da8/examples) | TS generated SDK、MCP/API exposure spikes、`third_party/` registry | **strategic-reference-only**（当前） | 许可证冲突且本仓已有 generated SDK authority；当前结论为 none，不提出 copy+patch | +| 完整 context layer / agent memory filesystem thesis | [`README.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/README.md) | ADR-0006、ADR-0061、Supply/Runtime/Learning 三环 | **strategic-reference-only** | 产品叙事相邻；我们以多租户授权真相和 audience-bound ContextPackage 为差异化核心 | + +# 3. 逐能力蓝图 + +## 3.1 Tiered context loading → 预算约束的 progressive disclosure + +### 上游路径与可观察行为 + +OpenViking 把内容写成 L0 摘要、L1 overview、L2 完整详情，并建议先读低成本层、按需深入；其固定描述与 token 量级见 [`docs/en/concepts/03-context-layers.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/03-context-layers.md)。层次检索先做全局 L0/L1 搜索，再以 priority queue 递归子目录；实现中有收敛轮次与并发边界，见 [`openviking/retrieve/hierarchical_retriever.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/retrieve/hierarchical_retriever.py)，rerank/回退行为由 [`tests/retrieve/test_hierarchical_retriever_rerank.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/tests/retrieve/test_hierarchical_retriever_rerank.py) 固定。上游的目录与层级只是检索结构，不提供可移植的多租户授权证明。 + +### 本仓 seam / authority + +- [ADR-0038](../decisions/0038-compile-and-publish-structural-markdown.md)：heading ancestry 在编译时进入同一 Fragment；Runtime 不得为了“补父标题”读取另一个未授权 Fragment。 +- [ADR-0012](../decisions/0012-sealed-authorization-projection-pipeline.md) 与 [ADR-0077](../decisions/0077-fix-the-article-as-the-content-authorization-atom.md)：`CandidateRef → AuthorizationKernel → AuthorizedProjection`；Article 是内容授权原子。同 Article + 同 current Revision 的扩展在 lineage 校验后继承该 Article 决定，跨 Article 扩展必须重新产生 CandidateRef 并授权。 +- [ADR-0006](../decisions/0006-engine-delivers-context-not-answers.md)：最终仍只形成 ContextPackage，不形成答案。 +- 实现 seam：`engine/runtime/budget.py::PackageBudgetMeter`、`engine/runtime/evidence.py::{CandidateRef, AuthorizedProjection}`、Runtime Assembler/PackageBudget gate。 + +### Room-A 行为规格 + +1. 在 `RuntimeProfile` 引用的不可变 `AssemblyProfile` 中定义三种**信息密度目标**，不把它们命名为存储层或 ACL：`abstract`（可判相关的最短投影）、`overview`（含结构/heading ancestry 的任务导航投影）、`detail`（获授权 Fragment 的原文投影）。默认从 `abstract` 开始；profile 可以限定 `max_density`、每 Article 的最多 expansion hops、每 hop token 上限、总 expansion 数和停止阈值。 +2. 每个初始候选先经 Kernel 形成 `AuthorizedProjection`。Assembler 的任何 content-bearing relevance、rerank、hydration 与 expansion 只接受该类型；不能接受 URI、Fragment id 或 `CandidateRef` 原始内容。 +3. 同 Article 扩展的输入必须携带 `article_ref + revision_ref + authorization_decision_ref + lineage_digest`。扩展前验证 current Revision 与 lineage；验证失败视作没有可扩展内容，不回退旧决定。跨 Article 链接转换为新的无内容 `CandidateRef`，从 Kernel 重走授权。 +4. 每个 hop 在读取正文前向同一 `PackageBudgetMeter` 预留候选数、token、block、evidence 和 latency 预算，完成后以实际值结算；超额则确定性停止并输出已形成的合法 Package，不使用未计量回退。heading ancestry 已在当前 Fragment 内，按同一 block token 计量。 +5. 停止条件只来自 profile + budget + 已授权相关性：达到 `max_density`、每 Article hop 数、总预算、无授权 child 或收益低于阈值。上游的 `MAX_CONVERGENCE_ROUNDS` 可作为性能 oracle，但不能成为授权或“目录已搜尽”的声明。 +6. 上游 rerank 失败回退 vector score 的可用性行为不得照搬到任何会改变授权/投影的阶段。授权后 ranker 不可用时，只能走 profile 中显式、经过评估且不接触 denied 数据的 deterministic policy；否则 fail closed/unavailable。 + +### 我方接口形状草图 + +```python +class DisclosureDensity(StrEnum): + ABSTRACT = "abstract" + OVERVIEW = "overview" + DETAIL = "detail" + +@dataclass(frozen=True) +class AssemblyProfile: + profile_ref: str + start_density: DisclosureDensity + max_density: DisclosureDensity + max_expansion_hops_per_article: int + max_total_expansions: int + max_tokens_per_hop: int + min_authorized_gain_micros: int + +@dataclass(frozen=True) +class AuthorizedExpansionRequest: + parent: AuthorizedProjection + requested_density: DisclosureDensity + article_ref: str + revision_ref: str + lineage_digest: str + +class ProgressiveAssembler(Protocol): + def assemble( + self, + projections: tuple[AuthorizedProjection, ...], + profile: AssemblyProfile, + budget: PackageBudgetMeter, + ) -> ContextPackage: ... +``` + +该接口不暴露 `depth_is_authorized`、raw URI 或 directory prefix；`DisclosureDensity` 只决定同一已授权内容的表达密度。 + +### 测试 oracle + +- 同一 query/profile/budget/Revision 得到确定的 density/hop 顺序和 Package digest；预算降一单位时在同一边界停止。 +- 任一 content consumer 传入 `CandidateRef`、Fragment record 或 string body 而非 `AuthorizedProjection` 都在类型/运行时双重拒绝。 +- 同 Article + 同 Revision 扩展成功；Revision 变化、lineage digest 错误、跨 Article 未重授权都返回不含内容的安全结果。 +- denied child 与不存在 child 形成同一公开 Package/错误形状，且没有 child count、path 或 score 差异。 +- heading ancestry 从当前 Fragment block 取得；测试 DB 中放置不可访问的“父 Fragment”也不得读取。 +- PackageBudget 在 abstract→overview、overview→detail 的每 hop 预留/结算；并发扩展不能超卖预算。 +- ranker 故障不得将 pre-auth vector score 或 candidate 内容送入 assembler；安全 veto 不能降级。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`；涉及真实 RLS/跨 Article fixture 时 `make db-up && make integration && make security-gate`；最终 `make check`。 +- 工作量：**6–9 engineer-days**。 +- 依赖：AssemblyProfile/RuntimeProfile contract 版本决策；ADR-0038 Fragment lineage；ADR-0077 Article atom；PackageBudget 并发预留语义；授权后 ranker 的已接受激活边界。 + +## 3.2 Virtual context filesystem → AuthorizedProjection 之上的安全 browse UX + +### 上游路径与可观察行为 + +OpenViking 使用 `viking:///`，区分 resources/user/agent 等 namespace，见 [`docs/en/concepts/04-viking-uri.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/04-viking-uri.md) 与 [`openviking/core/namespace.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/core/namespace.py)。文件系统 API 暴露 `ls`、recursive listing、tree、glob、read 等行为，见 [`docs/en/api/03-filesystem.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/api/03-filesystem.md)、[`openviking/storage/viking_fs.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/storage/viking_fs.py) 和 [`openviking/service/fs_service.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/service/fs_service.py)。`node_limit`/`level_limit` 约束资源使用，但不等价于 ContextEngine 的 finite-target authorization。 + +### 本仓 seam / authority + +- [ADR-0024](../decisions/0024-model-effective-scope-as-finite-target-intersection.md)：EffectiveScope 是来源、租户、purpose 与请求目标的精确有限交集，不接受 caller prefix 代表无限后代。 +- [ADR-0051](../decisions/0051-reauthorize-opaque-citation-opens.md)：OpenCitation 只接受 opaque `CitationOpenRef`；每次以新 current UserActor、DeliveryEvidenceRef 和 policy epoch 重授权，不能把旧决定或 source URI 当权限。 +- [ADR-0077](../decisions/0077-fix-the-article-as-the-content-authorization-atom.md)：同 Article/current Revision 的 Fragment 展开需 lineage 校验，跨 Article 重新授权。 +- [ADR-0090](../decisions/0090-admit-a-co-resident-local-evidence-console.md)：browse 只能通过 local Evidence Console 的 authenticated Runtime/Control HTTP seam 组合。 + +### Room-A 行为规格 + +结论：**browse-like-files 可以干净地作为 AuthorizedProjection 之后的 UX，但不能成为检索或授权抽象。** 实现规则如下： + +1. UI 的“文件夹”是一个 Package 内已授权 Evidence/Block 的临时分组，不是可遍历的源 namespace。可显示的 label、heading 与 child 只来自当前 ContextPackage 或一次新的 resolve；服务绝不回答“这个目录真实存在吗”。 +2. 初始 browse 请求仍是 `Acquire`，target 必须在 server 解析为 ADR-0024 的 exact finite set；不得传 `viking://org/**`、path prefix、glob 或 recursive flag 作为授权目标。 +3. 可展开节点只携带 opaque `citationOpenRef` 或 server-minted `BrowseContinuationRef`。后者绑定 Organization、audience、purpose、current Revision/Policy Epoch、父 Package digest、exact authorized Article set、下一 hop 上限和短 TTL；它不是 bearer ACL，redeem 后仍重建 trusted context 并过 Kernel。 +4. OpenCitation 返回新的 ContextPackage。UI 用新 Package 替换/追加视图，但不能在 wire 上得到 source URI、database id、未授权 sibling 数量、总 child 数、被裁剪数量、pre-auth score 或“有更多但无权限”的标志。 +5. 不存在、跨 Organization、same-Organization denied、过期 ref 和 policy epoch 变化必须收敛为同一 generic not-available 结果。空目录与全 denied 目录不得通过 timing、分页总数或占位符区分；如需 pagination，只返回 opaque next ref，不能返回 total。 +6. PackageBudget 对每一次 browse hop 生效；`level_limit`/`node_limit` 只能成为 meter 的 caller-independent server cap，不能增加 scope。跨 Article 的每个 child 都重新授权；同 Article expansion 也必须验证 current Revision lineage。 + +### 我方接口形状草图 + +```python +@dataclass(frozen=True) +class AuthorizedBrowseNode: + label: str # only from AuthorizedProjection + density: DisclosureDensity + citation_open_ref: str | None # opaque, short-lived + children: tuple["AuthorizedBrowseNode", ...] + +@dataclass(frozen=True) +class BrowseProjection: + package_digest: str + nodes: tuple[AuthorizedBrowseNode, ...] + next_ref: str | None # no total/count-of-hidden + +def project_browse(package: ContextPackage) -> BrowseProjection: ... + +ContextRuntime.resolve(invocation, trusted_delivery, Acquire | OpenCitation) + -> ResolutionOutcome +``` + +`project_browse` 是 presentation-only 纯函数，不能访问 database/index。若产品需要新的 `BrowseContinuationRef`，必须先用 ADR 激活相应 carrier；在此之前只使用已声明的 Acquire/OpenCitation，并诚实显示 unavailable。 + +### 测试 oracle + +- 构造同名的 nonexistent、cross-org、denied 和 expired-ref case，断言 HTTP status/domain code、body shape、是否有 next ref 均一致；对 timing 做宽容的侧信道上界测试。 +- RLS fixture 中只授权目录的一个 Article；Package/HTML 仅出现该 Article 衍生 label，不出现 sibling path、count、gap、score 或 source URI。 +- caller 提交 prefix/glob/recursive target 时在 content I/O 前拒绝；server 只能使用 exact finite targets。 +- current Membership、audience、purpose、Revision 或 Policy Epoch 改变后，旧 OpenCitation/Browse ref 不能打开内容。 +- 同 Article expansion 验证 lineage；跨 Article 一定产生新的 Kernel decision；直接 `project_browse(CandidateRef)` 类型拒绝。 +- 每 hop 预算可复算，分页 ref 绑定父 Package digest 且一次篡改/跨 org 重放失败。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`、`make smoke`；真实隔离用 `make db-up && make integration && make security-gate`；最终 `make check`。 +- 工作量：**8–12 engineer-days**。 +- 依赖：OpenCitation 的持久化/激活 issue（当前不可把未来 carrier 说成 active）；EffectiveScope target contract；Evidence Console browse job；opaque ref issuer/redeemer；PackageBudget。 + +## 3.3 Session → persistent memory extraction → 只产生 Learning 候选 + +### 上游路径与可观察行为 + +OpenViking 的 `session.commit()` 同步归档消息，再异步生成摘要并提取 memory，最终写 `memory_diff.json`；阶段与输出见 [`docs/en/concepts/08-session.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/08-session.md) 和 [`docs/design/session-memory-extraction-flow.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/design/session-memory-extraction-flow.md)。实现的 session/archive/queue 路径见 [`openviking/session/session.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/session/session.py) 与 [`openviking/session/compressor_v2.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/session/compressor_v2.py)，`memory_diff` 的 add/update/delete oracle 见 [`tests/session/memory/test_memory_diff.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/tests/session/memory/test_memory_diff.py)。上游会把抽取结果 create/merge/delete 到用户 memory；这个激活路径不能移植。 + +### 本仓 seam / authority + +- [ADR-0031](../decisions/0031-persist-authorized-context-run-lineage.md)：ContextRun 只持久化 authorized Evidence refs、版本化 query HMAC、Package digest 与预算等 digest-only lineage；不保存原始 query、Package body 或 denied trace。 +- [ADR-0014](../decisions/0014-curation-snapshot-and-release-ownership.md)：curation 输出不可变 CurationSnapshot，pipeline 无直接 activation operation。 +- [ADR-0033](../decisions/0033-promote-organization-releases-through-one-learning-owner.md)：`ContextLearning.evaluate(ReleaseCandidateRef)` 产生 evaluation；只有 release-operator-authorized promote 可激活/回滚 ReleaseManifest。 +- [ADR-0073](../decisions/0073-compose-explicit-release-candidates-from-current-corpus.md)：候选从 current corpus/profile/snapshot 显式组合，不能由模型输出跳到 active pointer。 + +### Room-A 行为规格 + +1. **先解决输入 authority，不从 ContextRun 反推正文。** ContextRun 的 digest/evidence refs 只能证明 lineage，不能重建 session。若要使用对话正文，先接受单独 ADR：定义用户/Organization consent、最小字段、加密/retention、删除/export、purpose、Learning 读取角色和 RLS。没有该 carrier 时，候选生成只能消费现有 authorized Package evidence 与显式 feedback，原始 session extraction 保持 `NOT_ACTIVE`。 +2. 对获准的 session material 创建 immutable `LearningInputRef`：绑定 Organization、actor/audience、source ContextRun refs、exact authorized Evidence refs、Package digest、consent/retention profile、as-of 和 input digest。它不能引用 denied candidate，也不授予对原始 Source 的新读权。 +3. “commit”分两相：事务内只写 recoverable candidate-generation intent/outbox；worker 用 exact durable-job-bound WorkerLease 消费。重试以 `(organization_id, input_digest, generator_profile_ref)` 幂等，`.failed` 等价物只能是受限状态，不泄露正文。 +4. 生成器输出 `CurationCandidate`，而不是 memory record：操作为 `propose_add | propose_update | propose_suppress`，携带 target Revision compatibility、candidate body/digest、来源 Evidence refs、模型/规则 profile、confidence/evidence category。`memory_diff` 只作为 operator-friendly candidate diff 形状，绝不执行 create/merge/delete。 +5. Curation 审核将 accepted candidate 变为 audited `CurationAnnotation`；一批兼容 annotations 固化为 immutable `CurationSnapshot`。失败或缺失 curation 的 Runtime 行为是 curation-off，不允许半完成 candidate 生效。 +6. release composer 以 current Content/Index/Runtime/Curation profiles 和 compatible Revision set 形成 `ReleaseCandidateRef`；`ContextLearning.evaluate` 运行已登记 slices、安全 veto、sample threshold 与 uncertainty。只有携带 release-operator authority 的 `promote` 可原子切换 ReleaseManifest；Control、candidate worker、model 和 Evidence Console 均没有这个 capability。 +7. Candidate body 不是 ContextRun，也不是 tenant-visible delivery；进入后续训练/eval 前再次保证 authorized-only，并继承明确 retention。撤销 Membership/ACL 后必须有 tombstone/revalidation policy；旧 Evidence refs 不能隐式续权。 + +### 我方接口形状草图 + +```python +@dataclass(frozen=True) +class LearningInputRef: + organization_id: UUID + source_run_refs: tuple[str, ...] + authorized_evidence_refs: tuple[str, ...] + package_digests: tuple[str, ...] + consent_profile_ref: str + input_digest: str + as_of: datetime + +@dataclass(frozen=True) +class CurationCandidate: + candidate_ref: str + operation: Literal["propose_add", "propose_update", "propose_suppress"] + compatible_revision_refs: tuple[str, ...] + proposed_annotation: bytes + provenance_digest: str + generator_profile_ref: str + +class SessionCandidateGenerator(Protocol): + def propose(self, input_ref: LearningInputRef, lease: WorkerLease) \ + -> tuple[CurationCandidate, ...]: ... + +CurationSnapshotBuilder.accept(audited_annotations) -> CurationSnapshot +ContextLearning.evaluate(release_candidate_ref) -> ReleaseEvaluation +ContextLearning.promote(evaluation_ref, release_operator_authority) -> ReleaseManifest +``` + +### 测试 oracle + +- 未接受 session-retention contract、缺 consent、缺 tenant/current actor、Evidence 非 authorized 或 WorkerLease job 不匹配时，在读取正文前 fail closed。 +- 相同 input/profile 重投不产生重复候选；archive 成功后 worker crash 可恢复；失败不会修改 active ReleaseManifest。 +- 模型输出 add/update/delete 只能形成 diff；直接调用 corpus/index/profile activation 的能力不存在。 +- candidate 的 Organization、Revision compatibility、Evidence refs 全做同 org 强约束；Org A candidate 不能进入 Org B snapshot/evaluation。 +- CurationSnapshot 不可变，且 Runtime 只从 active ReleaseManifest 读取 compatible snapshot；missing/failed snapshot 正常 curation-off。 +- evaluate 未通过 security veto、sample threshold 或 compatibility 时 promote 拒绝；Control credential、worker lease、candidate ref 均不能 promote。 +- ACL/Membership 撤销 fixture 验证旧 candidate 不因历史 delivery 自动获得当前授权；删除/export/retention fixture 与新 ADR 一起激活。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`；outbox/RLS/release 原子性用 `make db-up && make integration && make security-gate`；进程恢复用 `make smoke`；最终 `make check`。 +- 工作量：**10–15 engineer-days**，不含新的 session-body 加密/retention 系统；后者需另估。 +- 依赖：session/Learning input retention ADR；CurationCandidate 与 annotation schema；WorkerLease；CurationSnapshot builder；ReleaseCandidate composer；evaluation executor 与 release-operator auth。 + +## 3.4 Observable retrieval trajectories → authorized-only、digest-only 轨迹 + +### 上游路径与可观察行为 + +OpenViking 的 retrieval result/provenance 可带 query、匹配 URI、level、score、searched directories、match reason 等，结构见 [`openviking_cli/retrieve/types.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking_cli/retrieve/types.py) 与 [`tests/retrieve/test_provenance.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/tests/retrieve/test_provenance.py)；递归轨迹由 [`openviking/retrieve/hierarchical_retriever.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/retrieve/hierarchical_retriever.py) 形成，汇总指标见 [`openviking/retrieve/retrieval_stats.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/retrieve/retrieval_stats.py)。这些字段适合单机调试，但在多租户系统会泄露被拒绝资源的存在与相似度。 + +### 本仓 seam / authority + +- [ADR-0031](../decisions/0031-persist-authorized-context-run-lineage.md)：成功 Acquire 在 commit-before-response 的 ContextRun 保存 authorized-only Evidence refs、effective/used budget、Policy/epoch、版本化 query HMAC 与 Package digest；完整 Package/body 不持久化。delivered-empty 的 restricted DecisionAudit 只有闭合类别 `no_authorized_evidence`，没有 query/candidate/id/name/score/count。 +- [ADR-0076](../decisions/0076-rejoin-rank-evidence-after-authorization.md)：rank/rejoin 发生在授权后，但 public ContextPackage 不因此成为 score debug contract。 +- [ADR-0090](../decisions/0090-admit-a-co-resident-local-evidence-console.md)：Hit Test 只显示 Package Blocks/Evidence，明确不显示 candidate rank/score。 + +### Room-A 行为规格 + +轨迹分三层，禁止用一个“debug trace”对象跨层复用： + +| 接收者 | 允许信号 | 禁止信号 | +|---|---|---| +| Runtime consumer / tenant Evidence Console | `runRef`、`decisionRef`、Package digest/profile、release/policy epoch refs、effective/used PackageBudget、terminal `delivered_authorized|delivered_empty`、Package 已携带的 authorized Evidence refs/Blocks；可显示 Package 顺序但无 numeric rank | 原始 query、source URI、Candidate/denied ids、pre-auth path、score、rank gap、hidden/denied count、thinking trace | +| 独立授权的同 Organization operator read | 上述 digest-only ContextRun projection；可见 Organization-bound versioned query HMAC 用于有限相关性比较；只允许 Kernel **之后**且经独立 ADR 审查的粗粒度阶段耗时 | Package body副本、可逆 query、任何 denied trace；当前 ADR 未激活逐候选 post-auth score/rank 持久化，因此默认也不可见 | +| restricted security DecisionAudit | exact org/run/decision/policy/epoch、闭合 category `no_authorized_evidence`、recorded time | 除上述七字段外全部内容，尤其 query digest、candidate/ref/count/reason/score | + +实现上只从最终 `ContextPackage` 与 Kernel decision receipt 投影 ContextRun。pre-auth retriever 可以在请求内用 transient counters 做预算，但不能把 URI/score/路径送入 logs/metrics/debug/Learning；若要持久化阶段耗时，必须先证明数值不会把 denied candidate 数量编码出来，优先使用 Kernel 后阶段和固定桶。空结果与 denied 结果共享 terminal/category，不发布“searched N directories”。 + +### 我方接口形状草图 + +```python +@dataclass(frozen=True) +class AuthorizedTrajectoryProjection: + run_ref: str + decision_ref: str + policy_snapshot_ref: str + policy_epoch: int + query_digest_profile: str + query_digest: str + package_digest_profile: str + package_digest: str + effective_budget: PackageBudget + used_budget: PackageBudget + authorized_evidence_refs: tuple[str, ...] + outcome: Literal["delivered_authorized", "delivered_empty"] + +@dataclass(frozen=True) +class RestrictedDecisionAudit: + organization_id: UUID + run_ref: str + decision_ref: str + policy_snapshot_ref: str + policy_epoch: int + category: Literal["no_authorized_evidence"] + recorded_at: datetime +``` + +不存在通用 `RetrievalTrace(uri, score, query, thinking)` 类型；operator UI 读取的是 one-shot、same-org、digest-only projection，而非 Runtime 表直读。 + +### 测试 oracle + +- 对 delivered-authorized、nonexistent、cross-org、same-org-denied、empty-index 建 golden serialization，扫描 ContextRun/DecisionAudit/HTTP/log/metrics 中无 raw query、URI、candidate/ref、name、score、count、body。 +- query digest 使用版本化、Organization-bound HMAC；同 org 相同 bytes 可比较，跨 org 不相关，key rotation 改变 domain，密钥不序列化。 +- Package digest 可按固定 canonical profile 重算；数据库只保存 digest 与 authorized Evidence refs，不保存 Package body。 +- Runtime role 只能 INSERT、不能 SELECT；Control/worker 无表权；one-shot operator read 强制 same-org、exact decision、短 TTL 和提交后消费。 +- delivered-empty 只有 generic audit category；authorized delivery 不产生第二份 denied audit。 +- 任意 telemetry adapter 接到 CandidateRef/pre-auth score 时测试失败；阶段耗时只允许已登记字段与桶。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`；FORCE-RLS/roles/operator ticket 用 `make db-up && make integration && make security-gate`；HTTP projection 用 `make smoke`；最终 `make check`。 +- 工作量：**6–10 engineer-days**。 +- 依赖：ADR-0031 当前载体；日志/metrics redaction catalog；operator auth；若增加阶段耗时或 post-auth ranking，需新 ADR 与 retention policy，不能由本蓝图静默激活。 + +## 3.5 Studio / Helper / VikingBot → co-resident local Evidence Console + +### 上游路径与可观察行为 + +OpenViking 在同一服务的 `/studio` 提供 Home、Resources、Retrieval、Sessions、Request Logs，见 [`docs/en/guides/05-observability.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/guides/05-observability.md) 和 [`openviking/server/routers/console.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/server/routers/console.py)。Helper 检测 Claude Code/Codex/Cursor 等集成并展示 recall、prompt injection、capture、commit timeline，见 [`docs/en/agent-integrations/14-openviking-helper.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/14-openviking-helper.md)。VikingBot 把 context、model、tool 和 delivery 组合为 agent，见 [`docs/en/concepts/15-vikingbot.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/15-vikingbot.md) 与 [`docs/en/guides/17-vikingbot.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/guides/17-vikingbot.md)。最后一种一体化边界不适用于 ContextEngine。 + +### 本仓 seam / authority + +- [ADR-0090](../decisions/0090-admit-a-co-resident-local-evidence-console.md) 已固定 Jinja2 + static CSS、现有 API 进程、显式 browser auth、Runtime 经 in-process ASGI HTTP、Control 每次独立 credential/one operation、无 promote UI、无 score。 +- [ADR-0006](../decisions/0006-engine-delivers-context-not-answers.md)：Ask 的干净答案属于外部/consumer presentation；Engine 在线产物仍是 ContextPackage。 +- ActionPlane 必须 `prepare → ticket → exact effect`；Evidence Console 不得因同进程而直调 effect 或 release authority。 + +### Room-A 行为规格 + +建议把上游 UX 模式映射为 ADR-0090 已准入的七类 operator job，而不是复制 Web Studio： + +1. **Home/health**：显示 API/worker readiness、active release/policy refs、最近 authorized/empty run 数的安全聚合；不显示 tenant-wide query 或 denied candidate 统计。 +2. **Source progress**：Control read，一次请求一个 `SourceProgressRead`；浏览器每次提交 Control credential，server 不存储、不回显。 +3. **File import**：先 preview，token 绑定 current Membership、exact bytes digest、compiler version、Fragment set；confirm 是新的单一 `FileImportConfirm` 操作并重新校验。 +4. **Article policy**：read/change 分开；change preview token 绑定 expected policy version/epoch/proposed policy，confirm 只提交 exact effect。 +5. **Hit Test / Retrieval playground**：调用 `/v0/resolve` 的 Acquire，只渲染 ContextPackage Blocks/Evidence/预算/digest；numeric score/rank 明示 unavailable。 +6. **Ask**：同样先经 Runtime；每个 citationOpenRef 以 OpenCitation 获得 replacement Package，核对 exact Article/Revision/Fragment/Policy Epoch 后才由独立 presentation/model 形成文本。OpenCitation 未 active 时不伪造成功。 +7. **Timeline/feedback**：只展示 ADR-0031 authorized-only digest lineage；反馈是 evidence candidate，不是 Control 操作，不触达 promote。 + +同进程只减少部署拓扑，不合并权限。匿名 browser 不继承 dogfood principal；browser session proof 不包含 credential/identity claim；Control credential 与 Runtime browser proof 相互不能替代。VikingBot 的 generation、tool loop 与 channel adapter 必须留在 BotDelivery/ActionPlane/consumer，不移入 console 或 engine。 + +### 我方接口形状草图 + +```python +@router.post("/console/runtime/hit-test") +async def hit_test(browser_proof: BrowserProof, form: HitTestForm) -> HTMLResponse: + package = await in_process_http.post("/v0/resolve", acquire_wire(form)) + return render_package_without_scores(package) + +@router.post("/console/control") +async def control_call( + browser_proof: BrowserProof, + control_credential: TransportSecret, + operation: ClosedControlOperation, # exactly one union member +) -> HTMLResponse: + trusted = authenticate_one_control_call(control_credential, operation) + return render(await control.consume_once(trusted, operation)) +``` + +`ClosedControlOperation` 仅含 ADR-0090 明列的 source-progress read、preview-bound File import、Article-policy read/change；接口不接受 operation list、generic method name、release credential 或 engine object reference。 + +### 测试 oracle + +- 匿名、expired browser proof、public authenticator reject 都在 Runtime/Control I/O 前拒绝；cookie 为 short-lived/HttpOnly/SameSite=Strict 且不含原 credential。 +- monkeypatch 证明 Hit Test/Ask 经过 HTTP `/v0/resolve`，没有 direct retriever/assembler/DB 调用；citation 显示前确实 OpenCitation 重授权并校验 replacement Package lineage。 +- 每个 Control request 恰好一个 operation；批量 body、credential reuse/omission、cross-org Membership、stale preview token 全拒绝。 +- HTML snapshot 无 credential、raw query、source URI、denied count、score/rank；CSRF/session fixation/HTML escaping 有负面测试。 +- feedback path 只有 Runtime-role function 权限；UI 无 release operator field/route，Control role 不能 promote。 +- 不启动第二进程、Node runtime 或外部 CDN；`make smoke` 证明 API/worker 现有 topology。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`、`make smoke`；browser auth、RLS、Control credential 用 `make db-up && make integration && make security-gate`；最终 `make check`。 +- 工作量：**7–11 engineer-days**。 +- 依赖：ADR-0090 carriers、server templates/static assets、dogfood browser auth、local Control composition、OpenCitation activation（Ask 完整 citation path）、feedback evidence contract。 + +## 3.6 MCP client integrations + SDK → ONE sealed Runtime 的薄 parity + +### 上游路径与可观察行为 + +OpenViking 把 `/mcp` 与 REST 同进程，以 streamable HTTP 暴露 13 个工具，包括 find/search/recall/read/list/remember/add_resource/watch/grep/glob/forget/health；工具表见 [`docs/en/guides/06-mcp-integration.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/guides/06-mcp-integration.md)，实现注册见 [`openviking/server/mcp_endpoint.py`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/openviking/server/mcp_endpoint.py)。Claude Code、Codex、Cursor 组合 hooks 与 MCP：prompt 前 recall/inject，response 后 capture，PreCompact/session end commit，见 [`claude-code.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/02-claude-code.md)、[`codex.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/04-codex.md)、[`cursor.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/12-cursor.md)；通用 client 连接形状见 [`mcp-clients.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/agent-integrations/06-mcp-clients.md)。 + +### 本仓 seam / authority + +- [ADR-0017](../decisions/0017-trusted-invocation-and-closed-runtime-access.md)：所有 carrier 调同一非插件化 `ContextRuntime.resolve(AuthenticatedInvocation, TrustedDeliveryContext, Acquire|Continue|OpenCitation)`。 +- [ADR-0047](../decisions/0047-freeze-openapi-v0-through-one-runtime-path.md) 与 [ADR-0048](../decisions/0048-generate-typescript-sdk-behind-a-closed-facade.md)：HTTP/OpenAPI/generated SDK 是 contract authority；MCP 不能形成第二套模型或 capability。 +- [`2026-07-28-exposure-spike-mcp-vs-api.md`](./2026-07-28-exposure-spike-mcp-vs-api.md) 与 [`2026-07-28-pi-agent-consumer-spike.md`](./2026-07-28-pi-agent-consumer-spike.md)：MCP 只在真实 consumer gap 被证明时增加，pi/agent 也只能消费 fresh evidence-bearing ContextPackage。 + +### Room-A 行为规格 + +1. 第一版 MCP 只有 `resolve_context`，参数是与 OpenAPI `Acquire` 等价的 purpose/query/已激活 target shape；authenticated MCP transport 由 ingress adapter 兑换 per-resolve opaque `DeliveryEvidenceRef`，caller 不能在 tool body 制造 Organization、audience、trusted identity 或 raw delivery claims。 +2. tool handler 只做 schema translation → 调用 generated/typed HTTP facade → 原样映射 `ResolutionOutcome`。返回完整 ContextPackage（含 digest、expiry、Evidence/Blocks），不摘取“答案”、不缓存、不自行 vector search、不自行目录 read、不自行授权。 +3. `open_citation` 只有在同一 OpenCitation carrier 被正式激活后才加入，输入仅为 opaque citationOpenRef；`Continue` 同理。未激活能力返回现有 generic `request_not_available`，不能返回假 empty success。 +4. 不复制上游的 `read/list/glob/grep`：它们会把 filesystem 当 authority 并暴露 existence。不复制 `remember/forget/add_resource`：Learning candidate、Control import、ActionPlane effect 必须各走自己的 authority，不能混入 Runtime MCP。 +5. coding-agent hook 可以学习生命周期，但行为收窄为：`UserPromptSubmit/beforeSubmitPrompt` 发起 fresh Acquire 并把**完整、未过期、audience-bound** Package 交给 agent adapter；PreCompact/Stop/SessionStart 不直接写 memory。若未来记录反馈或 Learning input，必须调用已接受的独立 evidence/candidate seam，不能把 transcript 通过 MCP `remember` 激活。 +6. credentials 只来自平台的 MCP authenticated transport/secret store；禁止把 DeliveryEvidenceRef、API key 或 trusted audience 写入 prompt、tool body、logs。每次 resolve 新建 evidence ref，不在 long-lived MCP session 中复用。 +7. 更新两份 2026-07-28 spike 时记录：OpenViking 证明 Hooks + MCP proxy 能覆盖 Claude Code/Codex/Cursor 的安装与 lifecycle UX，但不证明宽 tool surface 是必要的；ContextEngine 的 parity 标准是“相同 sealed Runtime 语义与错误”，不是工具数量。 + +### 我方接口形状草图 + +```json +{ + "name": "resolve_context", + "inputSchema": { + "type": "object", + "additionalProperties": false, + "required": ["purpose", "query"], + "properties": { + "purpose": {"type": "string"}, + "query": {"type": "string"}, + "targetRefs": {"type": "array", "items": {"type": "string"}} + } + } +} +``` + +```python +async def resolve_context(tool_input: AcquireToolInput, transport: McpTransport) -> dict: + invocation, evidence_ref = await trusted_ingress.authenticate_and_issue(transport) + wire = acquire_to_closed_resolve_wire(tool_input) + outcome = await generated_runtime_sdk.resolve( + wire, + delivery_evidence_ref=evidence_ref, + ) + return exact_context_package_or_closed_refusal(outcome) +``` + +若 `targetRefs` 还不是已激活 public contract 就从 v0 schema 删除，不能为 MCP 单独发明。MCP response 的语义 golden 必须与 HTTP SDK response 一致。 + +### 测试 oracle + +- 对同一 authenticated invocation，HTTP、generated SDK、MCP 得到字段等价、digest 可复算的 ContextPackage 或同一 closed refusal；MCP 不改变排序/预算/expiry。 +- MCP handler 的依赖图只能指向 typed HTTP/generated facade；静态 catalog 禁止 import retriever、Kernel plugin、DB repository、assembler、Control/Learning。 +- tool body 注入 org/audience/identity/DeliveryEvidenceRef、跨 session replay、expired ref、缺 tenant context 均在 content work 前 generic fail closed。 +- tool list snapshot 初版恰为 `resolve_context`；不存在 remember/forget/list/read/glob/grep/add_resource。未来 OpenCitation 需 capability activation test 才改变 snapshot。 +- Hook 每个 prompt fresh Acquire；过期/不同 audience Package 不注入；Runtime unavailable/missing context 时停止 context-dependent consumer 行为，而非继续生成并声称有上下文。 +- logs/telemetry 不含 prompt、Package body、credentials、DeliveryEvidenceRef；MCP auth error 不回显 secret。 + +### 验证命令、工作量、依赖 + +- 命令：`make lint`、`make typecheck`、`make test`、`make catalog`、`make smoke`；认证/RLS/DeliveryEvidenceRef 用 `make db-up && make integration && make security-gate`；最终 `make check`。 +- 工作量：**6–9 engineer-days**，不含每个 consumer 的独立安装器。 +- 依赖：真实 consumer gap/maintainer go decision；MCP transport dependency 审核；OpenAPI/generated SDK parity fixtures；MCP authenticated ingress；DeliveryEvidenceRef issuer；首个目标 consumer（建议 Codex 或 Claude Code 二选一）。 + +## 3.7 Apache-2.0 subtrees → 逐路径 copy+patch 决策 + +### 上游路径与核验结果 + +候选表面是 [`crates/ov_cli`](https://github.com/volcengine/OpenViking/tree/49b182045b42d34ad530948ad77d9d0226897da8/crates/ov_cli) 与 [`examples`](https://github.com/volcengine/OpenViking/tree/49b182045b42d34ad530948ad77d9d0226897da8/examples)。但是 `ov_cli` 的 [`Cargo.toml`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/crates/ov_cli/Cargo.toml) 自报 MIT，和 [`crates/LICENSE`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/crates/LICENSE)/README 的 Apache 声明冲突；examples 内又有 [`openwebui-plugin/pyproject.toml`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/openwebui-plugin/pyproject.toml) 明确 AGPL、[`opencode-plugin/package.json`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/opencode-plugin/package.json) Apache、[`openclaw-plugin/package.json`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/examples/openclaw-plugin/package.json) MIT 等路径差异。 + +### 本仓 seam / authority + +- [ADR-0074](../decisions/0074-adopt-controlled-third-party-code-reuse.md)：只复制 pinned、license-verified permissive region，并在 `third_party/` 登记 attribution、modification 和 SBOM。 +- [ADR-0048](../decisions/0048-generate-typescript-sdk-behind-a-closed-facade.md)：generated SDK 是客户端 contract authority；手抄 CLI/client 会产生第二份 wire 模型。 +- 静态安全 catalog 与 `third_party/ARTIFACT_EXEMPTIONS.toml`： shipped artifacts 必须可追溯。 + +### Copy+patch 复刻配方与本次结论 + +**本次结论：none — generated SDK + 自有薄 MCP adapter 路径更好。** 没有任何上游文件应在本 spike 后进入 `third_party/`。理由不是 permissive 代码不可用，而是：`ov_cli` 许可证尚未消歧；examples 逐路径许可证不一致；上游 wire 与 filesystem/memory authority 不符合 sealed Runtime；复制客户端 glue 的维护收益低于 generated SDK。 + +未来若出现 generated SDK 无法覆盖的、可量化的 consumer gap，只能按以下配方开独立 PR： + +1. 将候选收窄到逐文件列表，核对每个文件的 SPDX、最近的 LICENSE/NOTICE、manifest、生成物和嵌套依赖；冲突即停止并请求 maintainer/legal 书面消歧。 +2. 证明候选不 import/link AGPL 主项目，也不包含由 AGPL 代码生成或搬运的实现；固定同一 SHA 并记录原始逐文件 SHA-256。 +3. 维护者批准具体 reuse issue 后，复制到 `third_party/openviking//`，保留上游 license/notice，新建 `MODIFICATIONS.md` 与 SBOM；只从该 vendored path import。 +4. 删除/重写任何 filesystem authority、memory write、raw URI、caller-authored tenant 或 direct REST model；适配到 generated closed facade。若重写量接近全部逻辑，取消复用，回到 clean-room。 +5. `make catalog` 必须拒绝未登记文件、AGPL SPDX、未固定 commit、hash drift、越界 import 和 shipped SBOM 缺项。 + +如将来获准，`UPSTREAM.toml` 至少采用以下形状（占位符不能在审批前填成推定事实）： + +```toml +repository = "https://github.com/volcengine/OpenViking.git" +commit = "49b182045b42d34ad530948ad77d9d0226897da8" +source_paths = [""] +excluded_paths = ["openviking", "openviking_cli", "web-studio", "bot", ""] +reuse_mode = "copy-patch" +approval = "" +license = "" + +[[files]] +upstream_path = "" +vendored_path = "third_party/openviking//" +sha256 = "" +``` + +### 我方接口形状草图 + +本区域不新增 runtime 接口。若未来 copy+patch，只能实现已经由 generated SDK 封闭的 adapter protocol： + +```python +class ContextPackageConsumerAdapter(Protocol): + async def acquire(self, request: GeneratedAcquireWire) -> GeneratedResolutionOutcome: ... +``` + +adapter 不得自己定义 tenant、authorization、filesystem 或 memory contracts。 + +### 测试 oracle + +- `UPSTREAM.toml` 的每个 `source_path` 存在于 pinned commit，vendored bytes/修改日志/许可证/SBOM 可追溯；未列文件不得 ship。 +- catalog 注入 AGPL SPDX、manifest/license 冲突、父许可证覆盖子 AGPL、浮动 branch、hash drift，均必须失败。 +- dependency graph 无 AGPL package/link；adapter contract golden 与 generated SDK 完全一致。 +- vendored adapter 删除后 generated SDK 主路径仍可构建/运行，证明它不是 Runtime foundation。 + +### 验证命令、工作量、依赖 + +- 当前文档决策命令：`make catalog`、`make lint`、`make test`；有 vendored artifact 时再运行 `make build` 并检查 wheel/sdist SBOM；最终 `make check`。 +- 工作量：**2–3 engineer-days** 用于一次正式 path-level/legal/provenance 决策；若实际 vendoring 另估。 +- 依赖：maintainer/legal 处理许可证冲突；明确 consumer gap；ADR-0074 approval；artifact/SBOM catalog。当前依赖未满足，因此 copy+patch 保持关闭。 + +## 3.8 战略判断：相邻竞品，也是有限互补方；不是 runtime foundation + +### 上游路径与本仓 thesis + +OpenViking 自称 agent 的 context database，把 memories/resources/skills 组织为可浏览的 virtual filesystem，以 tiered loading、recursive retrieval、session memory 和 observable trajectory 支撑 agent，见 [`README.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/README.md)；VikingBot 进一步组合 context/model/tools/channel，见 [`docs/en/concepts/15-vikingbot.md`](https://github.com/volcengine/OpenViking/blob/49b182045b42d34ad530948ad77d9d0226897da8/docs/en/concepts/15-vikingbot.md)。从这些固定材料可观察到 account/user/agent-local 的使用模型，但“缺乏与 ContextEngine 等价的 FORCE-RLS、多租户 audience authorization”只能表述为**在已核验路径中未见同等证据**，不能扩张为对上游全部安全能力的断言。 + +### 本仓 seam / authority + +ContextEngine 的边界由 [ADR-0006](../decisions/0006-engine-delivers-context-not-answers.md) 和 [ADR-0061](../decisions/0061-commit-to-the-complete-context-layer-thesis.md) 固定：我们建设完整 context layer，但在线只交付 current-audience-bound、可审计、可预算的 ContextPackage；generation/channel/effect 分离，多租户授权真相是 Kernel 的不可替代核心。 + +### Room-A 行为规格（产品边界）与明确答案 + +- **竞争重叠**：两者都争夺“agent 不应直接面对黑盒向量库”的 context layer 位置，都重视结构化摄取、分层披露、session-derived learning、可观察检索和 Claude Code/Codex/Cursor 等消费者。对单人/本地 agent memory 需求，它是相邻竞品。 +- **关键分歧**：OpenViking 的主对象是可读写 memory filesystem 和 agent integration；ContextEngine 的主对象是经当前身份、Organization、audience、purpose、policy epoch、budget 和 provenance 封闭的 ContextPackage。目录深度/路径不是 authority，Learning 不能发布，ContextRun 不保存 denied trace，Engine 不生成答案或执行工具。 +- **互补可能**：OpenViking 可以作为外部 consumer，通过未来 thin MCP/SDK 获取 ContextPackage；也可被评估为 Supply source，但只有 connector 能取得并证明 Live/Mirrored/真正 Weak `SourceAclEvidence`、映射 exact Organization/Article 且运行在 WorkerLease 下时才准入。若只有单用户 bearer/本地路径，没有可证明 ACL，就不能作为多租户授权来源。 +- **禁止关系**：不能把 OpenViking server、filesystem、retriever、memory store、Studio 或 VikingBot 作为 ContextEngine Runtime foundation；不能让它绕过 Kernel、保留第二索引/持久化或把 ContextPackage 拆成 raw content transport。 + +因此最准确定位是：**产品层的相邻竞品，协议/来源层的条件式互补方，安全架构上仅作战略参考。** ADR-0061 的“完整”不意味着吞并 agent runtime；完整的是从 Supply 到 authorized delivery 再到 governed Learning 的 context truth 闭环。 + +### 我方接口形状草图 + +```python +class ContextConsumer(Protocol): + async def consume(self, package: ContextPackage) -> None: ... + +class OptionalOpenVikingSupplyConnector(Protocol): + async def discover(self, lease: WorkerLease) -> tuple[SourceObjectRef, ...]: ... + async def acl_evidence(self, ref: SourceObjectRef, lease: WorkerLease) \ + -> SourceAclEvidence: ... +``` + +consumer 只能收完整 current ContextPackage；Supply connector 只产出候选 source objects/ACL evidence，经 ContextEngine 编译与授权，不把 `viking://`、上游 index 或 memory decision带进 Runtime。若 `acl_evidence` 无法达到已登记 carrier 的语义，connector 保持 `NOT_ACTIVE`。 + +### 测试 oracle + +- 架构 import/call graph 证明任何 OpenViking adapter 都不能被 `engine/runtime` 当 retriever/Kernel/assembler 使用，不能持有第二 Runtime index。 +- consumer 收到过期、其他 audience、digest 不匹配的 Package 时拒绝；不能仅用 Blocks/raw content 绕过 package lineage。 +- 可选 Supply spike 对缺 org mapping、缺 WorkerLease、ACL evidence 失败、Weak 被当 fallback、source ACL 撤销全部 fail closed。 +- BotDelivery/ActionPlane 测试证明 generation 与 exact effect 不在 Engine；缺 ContextPackage 的 context-dependent answer/effect 为 missing-context refusal。 +- 任何战略/公开 claim 在准入第五仓前不引用本报告为 public provenance。 + +### 验证命令、工作量、依赖 + +- 命令：战略 boundary 的静态检查为 `make lint`、`make typecheck`、`make test`、`make catalog`；未来 connector/RLS 用 `make db-up && make integration && make security-gate`；consumer/API 用 `make smoke`；最终 `make check`。 +- 工作量：**2–4 engineer-days** 用于边界 ADR/产品文字与 dependency tests；实际 OpenViking Supply connector 不在此估算内。 +- 依赖：maintainer 的第五仓/产品定位决定；MCP exposure go/no-go；任何 connector 都需独立 SourceAclEvidence、license/network、threat-model 和 activation issue。 + +# 4. 不可借鉴清单与必须杀死的隐含前提 + +| 必须杀死的隐含前提 / 上游诱因 | 为什么危险 | ContextEngine 强制替代 | +|---|---|---| +| L0/L1/L2 或目录 depth 越深，权限越大 | 把数据结构误当授权，能读取未授权 parent/child | tier 仅是 AssemblyProfile 信息密度；每个初始/跨 Article CandidateRef 走 Kernel | +| 先查 raw candidate/正文，再做权限过滤 | content-bearing rank/hydration 已看见拒绝内容 | `CandidateRef → AuthorizationKernel → AuthorizedProjection`，后续 consumer 只接受 projection | +| path prefix、glob、recursive flag 等于 scope | 无限后代、未来新增对象和 sibling 被隐式纳入 | ADR-0024 exact finite EffectiveScope intersection；每 hop 重授权 | +| `viking://` URI、文件名、目录是否存在是无害 metadata | 名称、数量和路径本身泄露敏感资源存在 | opaque locator；denied/nonexistent/cross-org 同形；只投影 authorized labels | +| `node_limit`/`level_limit` 是安全控制 | 它们只限制资源量，不能证明 audience/purpose/tenant | PackageBudget + Kernel；limit 永不扩大 scope | +| 同一次历史授权可永久打开 citation | Membership、ACL、Revision、Policy Epoch 已变化 | ADR-0051 每次 OpenCitation 新身份/DeliveryEvidenceRef/当前重授权 | +| rerank 故障可无条件回退 vector score | 可能把 pre-auth score/content 带入安全路径，改变选择而无评估 | 仅授权后、profile-登记的 deterministic fallback；否则 unavailable/fail closed | +| session model 输出可直接 create/merge/delete active memory | 模型成为第二个 publication authority，绕过评估/回滚 | immutable candidate/diff → CurationSnapshot → evaluate → release-operator promote | +| ContextRun digest 能重建 session/query | digest-only lineage故意不可逆；反推会建立暗中内容存储 | 新的 consent/retention Learning input carrier，未接受则 session extraction NOT_ACTIVE | +| retrieval trajectory 越详细越可观察 | denied URI/score/count/query/thinking 形成枚举与新内容库 | authorized-only ContextRun；七字段 generic DecisionAudit；redaction catalog | +| numeric rank/score 属于 operator 基本权利 | score gap 可泄露 denied candidates，且 public Package 当前 rank-free | Console 只显示 Blocks/Evidence/预算/digest，score 明示 unavailable | +| 同进程 Studio 可直调 engine/DB 或继承 dogfood principal | presentation 变成旁路 authority，匿名浏览器获得环境权限 | 显式 browser proof；Runtime 经 ASGI HTTP；Control 独立 credential、一次一操作 | +| feedback/Helper 的“同步 memory”可以顺带上线 | feedback 变成发布或 Control 能力 | feedback 只产生 evidence/candidate；无 Console promote route | +| VikingBot 的 model/tool/channel 应进入 Engine | Engine 开始交付答案与效果，破坏 ADR-0006/ActionPlane | BotDelivery/consumer generation，ActionPlane prepare→ticket→perform，Engine 只出 Package | +| MCP 工具越多 parity 越好 | read/list/remember/forget 形成第二 Runtime/Control/Learning authority | 初版一个 `resolve_context`，以后只镜像已激活 closed Runtime capability | +| MCP 长连接可复用 tenant/audience/DeliveryEvidenceRef | trusted context 过期或跨 prompt/session replay | authenticated transport 每 resolve 兑换短期、请求绑定的 opaque evidence ref | +| recall/memory 失败后 agent 可继续并声称有上下文 | 违反 missing-context fail closed，产出不可追溯答案/效果 | context-dependent consumer 在无 current Package 时拒绝；无假 empty success | +| 单 agent/local account 语义可直接推广到多租户 | 没有 Organization、current Membership、group audience 与 RLS 证明 | SourceAclEvidence + UserActor/WorkerLease + FORCE RLS + Kernel/security gate | +| `examples/LICENSE` 可覆盖所有子项目 | `openwebui-plugin` 已明确 AGPL；其他 manifest 也有 MIT/Apache/缺省 | 每次逐文件/manifest/SPDX/NOTICE 核验，冲突即停止 | +| `ov_cli` 已确定是 Apache-2.0 | Cargo manifest 自报 MIT，与父 LICENSE/README 冲突 | legal/maintainer 消歧前不复制；当前 none | +| AGPL 网络部署不算分发，所以可复制服务代码 | AGPL 第 13 节专门覆盖修改程序的远程网络交互 | 主项目严格 clean-room，零 code/dependency copy；必要时寻求法律意见 | +| [未取证] 可由 README 宣传或实现直觉补齐 | 会把推断包装成公开 provenance | 保留 `[未取证]`；只用 pinned permalink 支撑有限 claim | + +# 5. 推荐实现顺序 + 给 coordinator 的开放问题 + +## 推荐顺序 + +1. **先做治理决策（2–3d）**：决定 OpenViking 是否有条件成为第五仓；记录 AGPL clean-room、`ov_cli`/examples 排除项和 public-claim 范围。在决定前不改四仓公开基线，也不创建 `third_party/openviking`。 +2. **冻结 Room-A contracts 与 oracle（2–4d，和第 1 步可并行评审）**：为 `AssemblyProfile` density/hop、Authorized browse、trajectory redaction、Learning candidate diff 写新 ADR/contract fixtures；明确哪些 future carrier 仍是 `NOT_ACTIVE`。 +3. **先实现 tiered Assembly（6–9d）**：它复用现有 AuthorizedProjection、Article lineage 和 PackageBudget，价值高且不要求新的外部协议。先通过 raw-candidate rejection 与跨 Article reauthorization tests。 +4. **收紧 trajectory projection（6–10d）**：在增加更丰富 UI/MCP 前让 logs/metrics/operator view 有结构性红线；扩展 catalog 对 URI/query/score/count 的扫描。 +5. **完成 ADR-0090 Evidence Console UX（7–11d）**：先 Hit Test/source progress/preview-confirm；browse 只投影当前 Package。OpenCitation 未激活前，citation/browse 深入诚实显示 unavailable。 +6. **激活 OpenCitation 后做 Authorized browse（8–12d）**：opaque ref、current reauthorization、no-total pagination、per-hop budget 和 non-enumeration 一起交付，不单独上线 filesystem endpoints。 +7. **另立 retention 决策后做 session candidate generation（10–15d）**：先 LearningInput consent/retention，再 intent/outbox/WorkerLease、candidate diff、CurationSnapshot、evaluate/promote；绝不从 ContextRun digest 反推正文。 +8. **最后按真实 consumer gap 决定 thin MCP（6–9d）**：用一个目标 consumer 做 `resolve_context` parity；成功标准是 ContextPackage/closed-refusal 等价，不是复制 13 tools。同步把本文结论提炼进 MCP-vs-API 和 pi-agent 两份 spike，但只有在第五仓准入后才能把固定链接升级为 public provenance。 +9. **保持 copy+patch 为 none**：只有 generated SDK 确认无法解决的 gap、路径许可证消歧和 ADR-0074 审批同时满足时，才重新评估 permissive subtree。 + +总量不是简单相加：contracts/redaction 与 Console/Runtime 可共享 fixtures；不含 OpenCitation carrier、生产 operator identity、session 加密 retention 或新 Supply connector 的基础建设。按当前边界，核心蓝图约 **47–73 engineer-days**，治理/接口评审可与部分测试设计并行，但安全 gate 不可并行绕过。 + +## 给 coordinator 的开放问题 + +1. **OpenViking 是否进入公开四仓证据基线，成为第五仓？** 建议：**有条件进入**，仅作为 context filesystem/tiering、session candidate UX、observable retrieval 和 agent exposure 的参考；明确排除它作为多租户授权、安全证明或 runtime foundation。先完成路径许可证/legal 复核与逐 claim permalink，再改公开基线。 +2. `AssemblyProfile` 是否接受 `DisclosureDensity + per-hop limits` 为新的 immutable RuntimeProfile constituent，还是只在现有 PackageBudget/profile 内编码？建议新建显式 profile contract，防止调用方把 tier/depth 当 request-time authority。 +3. Authorized browse 的首个 UX 是否必须等待 OpenCitation 正式激活？建议等待；在此之前 Console 只对当前 ContextPackage 做纯投影，不创建临时 filesystem API 或伪 continuation。 +4. 是否愿意为 session-derived Learning input 承担独立的正文 consent、加密、retention、delete/export contract？若否，应明确只从现有 authorized Package evidence + explicit feedback 产生 candidates，原始 transcript extraction 长期保持 NOT_ACTIVE。 +5. operator trajectory 是否确有逐阶段 latency 的需求？建议首版只显示 ADR-0031 已有 digest/budget/outcome；任何新 timing/rank 字段先做侧信道 threat analysis，numeric score 继续关闭。 +6. thin MCP 的首个真实 consumer 选 Codex 还是 Claude Code，且 consumer 缺口是否足以优先于 generated SDK/HTTP？建议只选一个做 parity spike，初始工具面固定为 `resolve_context`。 +7. 是否请 maintainer/legal 向上游确认 `crates/ov_cli` 的 MIT vs Apache-2.0 冲突？即使确认，也建议暂不复制 CLI，因为 generated SDK 更符合 ADR-0048；确认结果只用于未来治理准确性。 +8. 是否允许未来把 OpenViking 作为 Supply source 做独立 spike？建议只有在能取得强 SourceAclEvidence 与 exact Organization mapping 时开启；单用户 token/local path 只能证明连接性，不能激活多租户 carrier。 + +在上述问题被回答前，本报告的默认执行边界是：**AGPL 全部 clean-room、无 copy+patch、无新 public provenance、无 filesystem/MCP/Learning carrier 自动激活，所有实现继续穿过唯一 sealed Runtime 与 release-operator publication authority。** diff --git a/docs/research/2026-07-31-ragflow-blueprint-evaluation.md b/docs/research/2026-07-31-ragflow-blueprint-evaluation.md new file mode 100644 index 00000000..a80153c2 --- /dev/null +++ b/docs/research/2026-07-31-ragflow-blueprint-evaluation.md @@ -0,0 +1,512 @@ +# RAGFlow → ContextEngine 可复刻蓝图评估 + +> **决策状态**：本文开放问题已由维护者于 2026-07-31 全部决定（D1/D6/D12），结果见 [`five-repository-implementation-blueprint.md`](./2026-07-31-five-repository-implementation-blueprint.md) §5；正文推荐项为评估时刻的状态。 + +> 评估日期：2026-07-31 +> 证据边界：RAGFlow 固定 commit 的静态源码、许可证与依赖声明；未启动 RAGFlow，未运行动态 benchmark、OCR 质量测试或故障注入。 +> 结论口径：`copy+patch` 只表示许可证区域允许且下文给出了注册候选；只有逐文件哈希、嵌套依赖/资产许可证、补丁、SBOM 和批准全部闭合后才可进入产品制品。`clean-room` 在本文表示保留行为而按 ContextEngine seam 独立实现，不是因为 RAGFlow 根许可证禁止复制。 + +## 1. 固定 commit 与许可证核验 + +### 固定版本与仓库级结论 + +- **[一手静态]** 上游固定为 `https://github.com/infiniflow/ragflow.git` 的 [`4391e03886b996201f3b8818f671b19eb24d0f7b`](https://github.com/infiniflow/ragflow/commit/4391e03886b996201f3b8818f671b19eb24d0f7b)，根 [`LICENSE`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/LICENSE) 是 Apache License 2.0。 +- **[一手静态]** 固定树根没有 `NOTICE` 或第三方 notice 聚合文件；这只证明该 checkout 没有相应路径，不证明所有嵌套代码和模型均无额外义务，参见[固定树](https://github.com/infiniflow/ragflow/tree/4391e03886b996201f3b8818f671b19eb24d0f7b)。 +- ADR-0074 因而允许按**精确源码区域**复制 RAGFlow，但不允许以根许可证替代逐路径、依赖和模型资产核验。任何真正落盘的复制仍须位于 `third_party/ragflow/`，并具备 `LICENSE.upstream`、`UPSTREAM.toml`、`MODIFICATIONS.md`、`patches/`、SBOM 和制品内 notice。 + +### 逐路径许可证区域与准入状态 + +下表中的“Apache 源码区已核验”表示文件自身带 InfiniFlow Apache-2.0 header 且受根许可证覆盖；“可复制”不等于其依赖闭包或模型制品已获准分发。 + +| 精确上游路径 | 固定文件 SHA-256 | 静态许可证/依赖结果 | 本评估准入 | +|---|---:|---|---| +| [`deepdoc/parser/docx_parser.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/docx_parser.py) | `891ffc11d2a3ac32e5c0d8b25b35aa62ab8cda1033c9e0a93782e9d45e759586` | **[一手静态]** Apache 源码区；直接依赖 `python-docx`、`pandas`、RAGFlow tokenizer、`LazyImage` 和 `common.constants`。 | `copy+patch` 候选；只复制这一文件，补丁移除所有 RAGFlow 应用层依赖。第三方包许可证文本未在固定 checkout 聚合，注册前补齐。 | +| [`deepdoc/parser/utils.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/utils.py) | `7d1674fb7c92b2db24964575cb2290139a823a923da89a321cbdaea795452849` | **[一手静态]** Apache 源码区；`extract_pdf_outlines` 只需 `pypdf`，同文件 `get_text` 依赖 `rag.nlp.find_codec`。 | `copy+patch` 候选；删除 `get_text`，只保留 outline 提取，并把吞异常改为 typed refusal/warning policy。 | +| [`deepdoc/parser/pdf_parser.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py) | `edf235cff17e11eb7a541c11711055f1f57fcb3fdd0d130e4a65b9097d5282eb` | **[一手静态]** Apache 源码区；直接连接 OCR/layout/TSR、`pdfplumber`、`pypdf`、XGBoost、scikit-learn、Pillow、Hugging Face 下载、RAGFlow tokenizer/settings/prompts。构造时可联网下载模型，语言判断使用 `random.choices`，若干解析异常会被吞掉。 | 源码法律区域允许 `copy+patch`，但**当前不准入制品**；必须先闭合下述模型与依赖审计并完成去网络/确定性补丁。 | +| [`deepdoc/vision/__init__.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/__init__.py) | `b48fcf785f6373a891a89b4a22cfe01771653fce3ff87b11abea8c32bf812b16` | **[一手静态]** Apache 源码区；引入整个 vision 图并附带 CLI 文件遍历 helper。 | 仅提取最小 export；不复制 `init_in_out`。 | +| [`deepdoc/vision/ocr.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/ocr.py) | `a4b1c380046584124edacb1d14c2c11fe293e843c0ae32bbe3e3a8a26b8474e3` | **[一手静态]** Apache 源码区；加载 `det.onnx`、`rec.onnx` 和 `ocr.res`，并可调用 `snapshot_download`。 | 条件式 `copy+patch`；模型必须由父进程以 digest-bound bundle 传入，runner 不得下载。 | +| [`deepdoc/vision/operators.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/operators.py) | `0464aa347d171ca18cbf11b3ab7e000c8c69886d9da28c816baf75c6b80a2500` | **[一手静态]** Apache header；没有嵌套来源/notice 声明，并依赖 OpenCV、NumPy、Pillow 与 `rag.utils.lazy_image`。 | 暂缓准入；先做来源/notice 人工审阅，补丁移除 RAGFlow helper。 | +| [`deepdoc/vision/postprocess.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/postprocess.py) | `a63d34ef62721e9d7965c33827f408d576f829d61e47f90c493081c3f419e741` | **[一手静态]** Apache header；没有嵌套来源/notice 声明，依赖 OpenCV、NumPy、Shapely、pyclipper。 | 暂缓准入；先闭合源码来源与包 notice。 | +| [`deepdoc/vision/recognizer.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/recognizer.py) | `f7284fe38e2b88a720475a7d6e6e0214e7aa6654ebcab838244a3eb1c29809fa` | **[一手静态]** Apache 源码区；执行 ONNX 模型并提供稳定几何排序/重叠 helper。 | 条件式 `copy+patch`；推理与纯几何 helper 分文件，禁止隐式模型路径。 | +| [`deepdoc/vision/layout_recognizer.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/layout_recognizer.py) | `52033843da3564f41f7f9c13269862f512cfcf41bfae821a4e19b5cd6e36d33a` | **[一手静态]** Apache 源码区；ONNX 路径可下载 `InfiniFlow/deepdoc`，Ascend 路径引用 `.om` 与 checkout 中不存在的 client。 | 只评估 ONNX 路径；Ascend 分支 `do-not-take`。 | +| [`deepdoc/vision/table_structure_recognizer.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/table_structure_recognizer.py) | `387827e3c1299db7d2867a40eebf64b6765fc941d555c9e26e1f811c72cd3324` | **[一手静态]** Apache 源码区；混合 ONNX/Ascend 推理、RAGFlow tokenizer、typed table 重建和 HTML/string 渲染。 | 条件式 `copy+patch`；只保留 typed grid/span 重建，禁用 HTML 作为 canonical output，删除 Ascend 与 tokenizer 依赖。 | + +### 嵌套 notice/模型资产扫描与排除项 + +- **[一手静态]** 固定仓没有跟踪 `rag/res/deepdoc/**` 模型文件；[`deepdoc/server/download_deps.py`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/server/download_deps.py) 会从仓外下载 `layout.onnx`、`det.onnx`、`rec.onnx`、`tsr.onnx`，而 [`deepdoc/server/README.md`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/server/README.md#L176-L192) 仅以文档文字称这些资产为 Apache-2.0。资产本体、资产 commit/hash、训练数据/基础模型 notice 不在该固定 Git checkout 内，故实际资产许可为 **[未取证]**。 +- **[一手静态]** `pdf_parser.py` 还会下载 `InfiniFlow/text_concat_xgb_v1.0/updown_concat_xgb.model`，见[构造路径](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py#L92-L101)。该二进制也不在固定树内，许可与 hash 对本报告而言 **[未取证]**。 +- **[一手静态]** 固定 `uv.lock` 锁定了运行包版本，但不提供逐包许可证清单，见 [`uv.lock`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/uv.lock) 与 [`pyproject.toml`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/pyproject.toml)。因此 `python-docx 1.2.0`、`pandas 2.3.3`、`pypdf 6.13.1`、`pdfplumber 0.10.4`、`xgboost 1.6.0`、`scikit-learn 1.8.0`、`Pillow 12.2.0`、`NumPy 1.26.4`、`opencv-python 4.10.0.84`、`onnxruntime 1.23.2`、`huggingface-hub 1.3.1`、`Shapely 2.1.2`、`pyclipper 1.4.0` 必须在复制 PR 中从各自主来源保存许可证/notice，本文不把常识许可证当已核验证据。 +- `operators.py`、`postprocess.py` 的实现形状与常见 OCR 工具链相似，但固定仓未给嵌套来源 notice；是否含外部派生代码为 **[未取证]**。在人工逐块来源审查完成前，不批准这两个文件进入分发制品。 + +必须登记在 PDF/OCR 候选的 `excluded_paths`，且不得由 runner 自行取得的路径/资产如下： + +| 排除项 | 原因 | +|---|---| +| `rag/res/deepdoc/**`，包括 `layout*.onnx`、`det.onnx`、`rec.onnx`、`tsr.onnx`、`ocr.res`、`updown_concat_xgb.model` | 固定 Git commit 未携带本体，资产级许可证、hash 与训练/基础模型 notice 未闭合。 | +| `**/*.om`、`deepdoc/vision/dla_cli.py` 及 `AscendLayoutRecognizer`/`_run_ascend_tsr` 分支 | `.om` 本体与 client 不在固定树的可核验闭包，且 ContextEngine 没有 Ascend 运行需求。 | +| `deepdoc/server/**` | 是上游第二个服务边界及联网下载面；违反 ADR-0075 的 owned pure runner 形状。 | +| `deepdoc/parser/__init__.py` | 导入所有 parser，扩大依赖和许可证闭包；现有 Markdown 注册已证明应绕开 initializer。 | +| `rag/nlp/**`、`rag/prompts/**`、`rag/app/**`、`api/**`、`common/settings.py`、`common/file_utils.py`、`common/misc_utils.py`、`rag/utils/lazy_image.py` | RAGFlow 应用、模型、存储和环境配置耦合；用 ContextEngine-owned ports/config 替换。 | +| parser 产出的 HTML、PIL image 对象以及原始 loose dict | 不是许可证排除，而是契约排除：不能成为 canonical Revision/Fragment 或直达 UI。 | + +结论：本固定 checkout **没有发现一个已跟踪且已证实“非宽松许可证”的模型文件**，因为模型本体根本不在树内；同样也不能把“未发现”写成“模型可分发”。在资产审计关闭前，DOCX+outline 是唯一可立即准备注册 PR 的新 parser lift，PDF/OCR/layout/table 只可做隔离、不可分发的 evidence spike。 + +## 2. 能力盘点 → ContextEngine 区域映射表 + +| 能力/上游位置 | ContextEngine 区域与 seam | 结论 | Kernel/发布边界 | +|---|---|---|---| +| DOCX 段落、style、表格：`deepdoc/parser/docx_parser.py` | Supply / `adapters/parsers` → owned compiler-runner → typed `ParsedDocument` | **copy+patch**；最小源码区，移除 RAGFlow tokenizer/image/application coupling。 | 不改变 Runtime；未来发布仍为 immutable Revision/Fragment。 | +| PDF outline：`deepdoc/parser/utils.py::extract_pdf_outlines` | Supply / format-neutral document compiler | **copy+patch**；小而可隔离。 | Outline 是编译输入/结构，不是授权或 Runtime hydration。 | +| PDF OCR/layout/table：`pdf_parser.py` + `deepdoc/vision/*` | Supply / digest-bound offline model bundle + compiler-runner | **copy+patch（准入门未闭合）**；源码可用，资产/notice 未闭合前不得入制品。 | runner 无网络、DB、index、checkpoint；完整 typed document 成功后才能进入 `prepared`。 | +| PDF/DOCX loose tuple/dict、HTML、PIL 输出 | Supply domain constructor | **do-not-take**；只借识别行为，输出层重写。 | 自验证 constructor 重算顺序、结构、locator、digest 和 bound。 | +| `search.py::search/retrieval` 的后端 weighted-sum、稳定排序 | Runtime candidate discovery / `CandidateQuery`、`RankedCandidateList`、`prekernel_fusion` | **clean-room**；保留本仓原生 PostgreSQL FTS+pgvector 与现有 RRF，补 parity fixtures。 | pre-Kernel 仅 `CandidateRef` + rank evidence；index filter 永不授权。 | +| `rerank*`、`_rank_feature_scores` | Runtime authorized ranking/model inference port | **clean-room**；只在 `AuthorizedProjection` 上实现。 | 精确 cut：`AuthorizationKernel.authorizeAndProject` 之后。 | +| `retrieval_by_toc`、`retrieval_by_children`、`chunk_list` | `FragmentWindowRequest` + content-free expansion refs | **clean-room**；不可复制直接 datastore hydration。 | 同 Article/current Revision 经 lineage 验证继承；跨 Article 发新 `CandidateRef` 并重授权。 | +| `chunk_builder.py::get_parser/run_chunking/extract_outline` | Supply compiler selection/execution | **do-not-take**；registry、TaskContext、DB metadata write 与 v3 seam 冲突。只保留“显式 parser profile + outline 是结构”这一行为。 | 编译器 runner 纯变换，不能写 Doc metadata。 | +| comparator + recording context + write interceptor | Eval / Adapter parity executor | **clean-room**；采用 canonical typed artifact digest，不复制任意 Python object comparator。 | Eval 无发布权；不得把真实生产写返回值重放当安全等价。 | +| Redis unacked queue、task executor、heartbeat | Supply execution/checkpoint bridge | **do-not-take** 部署形状；**clean-room** 吸收 reclaim、阶段记录、取消、终态后 ack 行为。 | PostgreSQL durable job + exact signed WorkerLease 是唯一执行权；runner 无 ambient tenant 或持久化。 | + +特别纠偏：**[一手静态]** 固定 `rag/nlp/search.py` 并没有 weighted RRF；它在 [`search`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L134-L245) 中把全文与 dense 表达式交给后端 `FusionExpr("weighted_sum", weights="0.05,0.95")`，随后在 [`retrieval`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L549-L745) 中做内容参与的 rerank 和稳定排序。RRF、exact-ref dedupe 与授权后权重压缩应继续以 ContextEngine 当前 owned 实现为 authority，而不是错误归因并复制 `search.py`。 + +## 3. 逐能力蓝图 + +### 3.1 deepdoc PDF/DOCX/table/OCR-layout parser + +#### 层级与上游路径 + +- **[一手静态] DOCX：** [`RAGFlowDocxParser.get_picture/__extract_table_content/__compose_table_content/__call__`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/docx_parser.py#L33-L182) 返回段落文本+style 和表格自然语言字符串；它没有原始 DOCX member/XML locator，也不保留逐表格 cell 的 canonical typed structure。 +- **[一手静态] PDF outline：** [`extract_pdf_outlines`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/utils.py#L39-L54) 深度遍历 `pypdf` outline，给出 title/depth/page。 +- **[一手静态] PDF pipeline：** [`RAGFlowPdfParser.__call__`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py#L1674-L1699) 串接 page image/OCR、layout、table transformer、text merge、跨页 concat、filter 与 table/figure extraction；[`parse_into_bboxes`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py#L1701-L1763) 提供分批 page window 和 bbox 输出。 +- **[一手静态] provenance 素材：** [`_line_tag/extract_positions/get_position`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py#L1443-L2000) 可提供 page/bbox；[`_extract_table_figure`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/parser/pdf_parser.py#L1208-L1417) 组合跨页 table/figure、caption、crop 与 position。 +- **[一手静态] OCR/layout/table：** [`OCR`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/ocr.py#L493-L670)、[`LayoutRecognizer`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/layout_recognizer.py#L33-L167)、[`Recognizer`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/recognizer.py#L32-L409) 和 [`TableStructureRecognizer.construct_table`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/deepdoc/vision/table_structure_recognizer.py#L156-L579) 构成推理、阅读顺序、table cell/span 重建与字符串渲染链。 + +#### 映射的本仓 seam/模块/ADR + +- `adapters/parsers/.py` 只能实现 Supply-owned `DocumentCompiler` port；subprocess envelope 复用 ADR-0079 的 fixed deadline、typed `CompilationFailure`、no network/DB/state 约束，并在生产调用时由 ADR-0075 Supply bridge 绑定父 WorkerLease。 +- ADR-0036/0038/0079 的**确定性、版本显式、all-or-nothing、self-validation、provenance、hard bound**继续适用，但当前 `engine/supply/markdown.py::ParsedDocument` 的 `SectionKind` 和 UTF-8 byte span 是 Markdown representation，不足以诚实表达 DOCX zip member 或 PDF page+bbox。必须先写一个 ADR：是把 `ParsedDocument` 提升为 format-neutral family，还是新增同族 `ParsedOfficeDocument`/`ParsedPdfDocument` 后在统一 publication seam 汇合。不能把 PDF bbox 伪装成 Markdown byte span。 +- 建议的 typed source locator union： + - `TextByteSpan(source_digest, start, end)`：只用于真正可 round-trip 的文本； + - `DocxXmlLocator(artifact_digest, part_uri, block_ordinal, xml_digest)`； + - `PdfRegionLocator(artifact_digest, page_number, bbox_points, page_render_digest, extraction_method)`； + - table/figure 可携带多个 ordered locators，但仍只生成一个 structural Fragment。 +- structural mapping：outline/style heading → `HEADING`；连续 paragraph/text layout → `PARAGRAPH`；DOCX/PDF table（caption+grid+span）→ `TABLE`；list → `LIST`；figure+caption/OCR → 需要新 `FIGURE` kind 或明确拒绝。每一 heading、paragraph、list、table、figure 是一个结构单元和一个 Fragment；table cell、OCR word、PDF line都不是独立 Fragment。 +- `ContextRevision` 绑定原始 artifact SHA-256、compiler/config/model-bundle identities 和完整 canonical parsed document digest；`ContextFragment` 绑定 structural path、locator、source text/canonical text、parent headings 与同一 Revision。发布仍走 `prepared -> indexed -> active`，parser success 不等于发布。 + +#### 复刻配方 + +1. **先定 representation ADR。** 固定 format profiles（例如 `docx-config-v1`、`pdf-layout-config-v1`）、locator union、structural kinds、canonical serialization、hard bounds（文件/页/像素/blocks/cells/text/model runtime）与 refusal vocabulary。未知 profile 在打开文件或加载模型前拒绝。 +2. **拆 runner 输入。** 父进程只传 `artifact_bytes`、exact profile、fixed deadline、以及只读 `ModelBundleRef{name, revision, files[{sha256,size}], license_manifest_digest}`；生产 runner 不能读环境变量选择模型、不能访问 Hugging Face、不能接路径或 DB credential。 +3. **先落 DOCX。** 复制 `docx_parser.py` 到新注册 subtree；patch 移除 `rag_tokenizer`、`LazyImage`、`MAXIMUM_PAGE_NUMBER`、Pandas DataFrame。直接遍历 OOXML block order（paragraph/table interleave，而不是上游分别遍历 `document.paragraphs` 和 `document.tables`），输出 owned raw blocks。记录 part URI、block ordinal、style、XML digest、paragraph/table canonical text；图片先输出 typed attachment descriptor 或拒绝，绝不返回 mutable PIL/LazyImage。 +4. **加入 outline helper。** 从 `utils.py` 只保留 `extract_pdf_outlines` 的行为；异常按 profile 返回 `OUTLINE_UNAVAILABLE` warning 或使整个 compilation refusal，不能无条件 `except: return []`。outline title/depth/page 必须验证页范围并按源顺序稳定。 +5. **做 PDF source-only spike。** 复制候选九文件到 `/tmp` 或尚未分发的受控 spike；先 patch 掉 `snapshot_download`、所有 Ascend/LLM/application imports、`random.choices`、环境变量控制、HTML canonical output和吞异常。语言采样改为固定 page/character stride；所有几何排序加完整 tie-break `(page, top, x0, bottom, x1, source_ordinal)`。 +6. **关闭资产门。** 对每个 ONNX/XGBoost/dictionary 文件固定独立上游 repo revision、SHA-256、size、license、NOTICE、模型 card、训练/基础模型 obligations；任何一项未知则 PDF/OCR profile `NOT_ACTIVE`。将许可证文本和模型 components 写入 RAGFlow subtree SBOM及 wheel/sdist/container。 +7. **分离推理与结构内核。** ONNX 只输出 bounded raw detections；ContextEngine-owned constructor独立验证 bbox 有限/页内、class vocabulary、置信度范围、reading order、caption/table association、cell spans和总量。parser 提供的 dict/HTML/IDs 都是不可信中间态。 +8. **生成 coherent units。** 同表 caption、header、body、row/col spans合成一个 typed table Fragment；同 figure caption/OCR 合成一个 figure Fragment；heading ancestry在编译时复制到同 Fragment并计入 budget，不留 Runtime parent lookup。 +9. **确定性和 all-or-nothing。** 两个 fresh subprocess、不同 hash seed/CPU thread setting对同 fixture产生逐字节相同 canonical document与 digest；模型输出若平台不可逐位稳定，则在 ADR 中定义固定量化/排序语义并证明跨目标平台一致，否则该 profile只支持一个明确 runtime target。 +10. **接 publication。** 只有 typed constructor通过后，Supply worker在当前 exact WorkerLease generation下 staging；完整 Revision、Fragments、embedding/candidates、events准备完毕后才 CAS active pointer。runner永远不直接写这些表。 + +DOCX+outline 候选 `UPSTREAM.toml`（实施时拆成新的 registration block或新文件，不覆盖已注册 Markdown）： + +```toml +repository = "https://github.com/infiniflow/ragflow.git" +commit = "4391e03886b996201f3b8818f671b19eb24d0f7b" +source_paths = [ + "deepdoc/parser/docx_parser.py", + "deepdoc/parser/utils.py", +] +excluded_paths = [ + "deepdoc/parser/__init__.py", + "deepdoc/parser/pdf_parser.py", + "deepdoc/vision", + "rag/nlp", + "rag/utils/lazy_image.py", + "common/constants.py", +] +reuse_mode = "copy-patch" +approval = "PENDING: create a dedicated ContextEngine issue/ADR after format contract closure" +license = "Apache-2.0" +nested_dependency_audit = "PENDING: attach primary license texts before approval" +nested_dependencies = [ + { name = "python-docx", version = "1.2.0", license = "PENDING_PRIMARY_VERIFICATION" }, + { name = "pypdf", version = "6.13.1", license = "PENDING_PRIMARY_VERIFICATION" }, +] + +[[files]] +upstream_path = "deepdoc/parser/docx_parser.py" +vendored_path = "third_party/ragflow/deepdoc/parser/docx_parser.py" +sha256 = "891ffc11d2a3ac32e5c0d8b25b35aa62ab8cda1033c9e0a93782e9d45e759586" + +[[files]] +upstream_path = "deepdoc/parser/utils.py" +vendored_path = "third_party/ragflow/deepdoc/parser/utils.py" +sha256 = "7d1674fb7c92b2db24964575cb2290139a823a923da89a321cbdaea795452849" +``` + +PDF/OCR/layout/table 候选模板（这是**blocked registration**；`approval` 和所有 `PENDING` 关闭前不得复制进工作树）： + +```toml +repository = "https://github.com/infiniflow/ragflow.git" +commit = "4391e03886b996201f3b8818f671b19eb24d0f7b" +source_paths = [ + "deepdoc/parser/pdf_parser.py", + "deepdoc/parser/utils.py", + "deepdoc/vision/__init__.py", + "deepdoc/vision/ocr.py", + "deepdoc/vision/operators.py", + "deepdoc/vision/postprocess.py", + "deepdoc/vision/recognizer.py", + "deepdoc/vision/layout_recognizer.py", + "deepdoc/vision/table_structure_recognizer.py", +] +excluded_paths = [ + "deepdoc/server", + "deepdoc/parser/__init__.py", + "rag/res/deepdoc", + "rag/nlp", + "rag/prompts", + "rag/app", + "api", + "common/settings.py", + "common/file_utils.py", + "common/misc_utils.py", + "rag/utils/lazy_image.py", + "**/*.om", +] +reuse_mode = "copy-patch" +approval = "BLOCKED: model/dependency/source-lineage audit and representation ADR required" +license = "Apache-2.0-source-region" +model_assets = [ + { name = "layout*.onnx", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, + { name = "det.onnx", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, + { name = "rec.onnx", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, + { name = "tsr.onnx", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, + { name = "ocr.res", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, + { name = "updown_concat_xgb.model", sha256 = "PENDING", license = "UNVERIFIED_AT_PINNED_GIT_COMMIT" }, +] + +[[files]] +upstream_path = "deepdoc/parser/pdf_parser.py" +vendored_path = "third_party/ragflow/deepdoc/parser/pdf_parser.py" +sha256 = "edf235cff17e11eb7a541c11711055f1f57fcb3fdd0d130e4a65b9097d5282eb" +[[files]] +upstream_path = "deepdoc/parser/utils.py" +vendored_path = "third_party/ragflow/deepdoc/parser/utils.py" +sha256 = "7d1674fb7c92b2db24964575cb2290139a823a923da89a321cbdaea795452849" +[[files]] +upstream_path = "deepdoc/vision/__init__.py" +vendored_path = "third_party/ragflow/deepdoc/vision/__init__.py" +sha256 = "b48fcf785f6373a891a89b4a22cfe01771653fce3ff87b11abea8c32bf812b16" +[[files]] +upstream_path = "deepdoc/vision/ocr.py" +vendored_path = "third_party/ragflow/deepdoc/vision/ocr.py" +sha256 = "a4b1c380046584124edacb1d14c2c11fe293e843c0ae32bbe3e3a8a26b8474e3" +[[files]] +upstream_path = "deepdoc/vision/operators.py" +vendored_path = "third_party/ragflow/deepdoc/vision/operators.py" +sha256 = "0464aa347d171ca18cbf11b3ab7e000c8c69886d9da28c816baf75c6b80a2500" +[[files]] +upstream_path = "deepdoc/vision/postprocess.py" +vendored_path = "third_party/ragflow/deepdoc/vision/postprocess.py" +sha256 = "a63d34ef62721e9d7965c33827f408d576f829d61e47f90c493081c3f419e741" +[[files]] +upstream_path = "deepdoc/vision/recognizer.py" +vendored_path = "third_party/ragflow/deepdoc/vision/recognizer.py" +sha256 = "f7284fe38e2b88a720475a7d6e6e0214e7aa6654ebcab838244a3eb1c29809fa" +[[files]] +upstream_path = "deepdoc/vision/layout_recognizer.py" +vendored_path = "third_party/ragflow/deepdoc/vision/layout_recognizer.py" +sha256 = "52033843da3564f41f7f9c13269862f512cfcf41bfae821a4e19b5cd6e36d33a" +[[files]] +upstream_path = "deepdoc/vision/table_structure_recognizer.py" +vendored_path = "third_party/ragflow/deepdoc/vision/table_structure_recognizer.py" +sha256 = "387827e3c1299db7d2867a40eebf64b6765fc941d555c9e26e1f811c72cd3324" +``` + +Patch-diff sketch： + +```diff +- from huggingface_hub import snapshot_download +- from common import settings +- from rag.nlp import rag_tokenizer +- model_dir = snapshot_download(...) ++ from context_engine_runner_contract import ModelBundle, RawStructuralBlock ++ model_dir = verified_read_only_bundle.require_files(expected_manifest) ++ # no environment-derived mode, network, database, application service or LLM + +- random.choices(page_chars, k=min(100, len(page_chars))) ++ stable_stride_sample(page_chars, maximum=100) + +- except Exception: +- logging.exception(...) +- return [] ++ except ClosedParserError as error: ++ raise CompilationFailure(category=map_closed_category(error)) from None + +- return secs, tbls ++ return tuple(RawStructuralBlock(..., source_locator=...)) + +- return "..." ++ return TypedTable(caption=..., cells=..., row_spans=..., column_spans=...) +``` + +#### 验证计划 + +- `make test`：DOCX/PDF domain constructor、malformed zip/PDF、zip bomb、page/pixel/block/cell bounds、out-of-range bbox、duplicate/overlap tie、outline cycle/invalid page、table colspan/rowspan、no HTML execution、all-or-nothing、model manifest mismatch、runner timeout/kill、two-process digest determinism。 +- `make catalog`：只有 owned runner adapter 可 import vendored deepdoc；production runner禁止 `socket`/Hugging Face/DB/settings/host path；第三方 registration/hash/SBOM/制品 notice完整；Markdown 已注册文件不得被新 initializer 间接扩大依赖。 +- `make integration`（需要先由协调者按仓约启动 DB）：真实 PostgreSQL 下 staged Revision 不可见、完整结构单元原子激活、旧/新 Revision全有或全无、每 Fragment同 Article/Revision lineage、失败不产生部分 Fragment、恢复不重新调用已完成的确定性编译步骤。 +- `make security-gate`：PDF/DOCX候选混入 denied/cross-Organization 时正文进入 content-bearing consumer 数为 0；parser metadata、outline、bbox、index字段均不能授权。 +- 额外离线 corpus gate：按格式/语言/扫描质量/table/outline/异常切片报告 structural recall、reading-order、table exactness 与 refusal；未运行前均为 **[未取证]**，不得称 PDF/OCR 已可用。 + +#### 工作量估计与依赖 + +| 子项 | 工程日 | 依赖 | +|---|---:|---| +| representation ADR + format-neutral typed contracts | 4–6 | 维护者决定 PDF/DOCX locator 与 figure kind | +| DOCX+outline copy/patch、注册、单测 | 6–9 | 依赖许可证文本、OOXML fixtures | +| PDF source-only deterministic spike | 8–12 | representation ADR、离线模型 bundle contract | +| 模型/依赖/source-lineage 法务与 SBOM closure | 4–8 | 模型实际来源 owner；可能并行，但结果可否决 | +| PDF/OCR/layout/table runner 产品化与 integration | 15–22 | 前述两门全绿、固定 supported runtime target | + +总体：DOCX+outline 10–15 engineer-days；PDF/OCR 再增加 27–42 engineer-days。若模型资产门失败，PDF/OCR 立即停止，不把 sunk cost 当准入理由。 + +### 3.2 `rag/nlp/search.py` retrieval pipeline + +#### 层级与上游路径及逐函数 cut line + +| 函数 | 固定源码事实 | ContextEngine cut/结论 | +|---|---|---| +| [`get_vector`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L55-L62) | **[一手静态]** query embedding → dense match expression。 | 可借行为，继续用 owned query embedding/provider profile；只产生 candidate plan，不读 Fragment正文。 | +| [`get_filters`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L121-L132) | **[一手静态]** 从 request dict接受 KB/doc/id 等 filter。 | `do-not-take`；caller filter只可 RequestNarrowing，不能构造可信 scope。 | +| [`search`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L134-L245) | **[一手静态]** 默认 `src` 包含正文、title、doc name、position、tags等，并使用后端 weighted-sum。 | 整函数不能位于 pre-Kernel。保留本仓 FTS/pgvector ports，仅返回 named ranked `CandidateRef` lists和可选原始分数。 | +| [`_prune_deleted_chunks`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L64-L119) | **[一手静态]** 根据 DB document存在性过滤 stale index chunk，自称 temporary safety net。 | `do-not-take` 为授权；current Resource/Revision/tombstone由 Kernel/authoritative projection检查，index cleanup另行异步。 | +| [`_knn_scores`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L363-L394) | **[一手静态]** 对既有 candidate IDs取 KNN分数且不取 source fields。 | pre-Kernel 可等价实现为 content-free `RankerEvidence.score`，但分数不能进入 Kernel决策。 | +| [`rerank_with_knn`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L434-L459)、[`rerank`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L461-L492)、[`rerank_by_model`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L494-L519) | **[一手静态]** 全部读取 `content_ltks`/title/question/keywords；model版本还把 docs送外部 reranker。 | **精确 cut：全部 post-Kernel**；输入必须是 `AuthorizedRerankItem(AuthorizedProjection, rank evidence)`。模型调用还需 governed inference/egress/budget。 | +| [`_rerank_window`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L524-L547) | **[一手静态]** 使 candidate window成为 page size整倍数。 | 可 clean-room保留分页 invariant；server-owned bound，不能由 caller使候选提交超过 ceiling。 | +| [`retrieval`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L549-L745) | **[一手静态]** 一个函数内混合 recall、stale prune、正文 rerank、稳定排序、分页和正文 hydration。 | 必须拆开：`discover → fuse refs → Kernel → compact/reweight admitted ranks → authorized rerank → budget`。不能复制整函数。 | +| [`chunk_list`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L751-L799) | **[一手静态]** 按 doc_id直接读 chunk fields。 | 仅可由 Kernel构造的 `FragmentWindowSession` 间接触发；不是公共/可插拔 datastore port。 | +| [`retrieval_by_toc`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L839-L900) | **[一手静态]** 读取 TOC正文、调用 chat model、直接 get 新 chunk正文。 | post-Kernel clean-room；TOC model只接 AuthorizedProjection；它返回 expansion refs，不直接 hydrate。 | +| [`retrieval_by_children`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L902-L940) | **[一手静态]** child有 `mom_id` 时直接读取 parent chunk并用 parent正文替换。 | post-Kernel clean-room；同 Article/current Revision经 lineage验证继承，任何 parent指向另一 Article都返回 CandidateRef给 Kernel重授权。 | +| [`insert_citations`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/nlp/search.py#L251-L328) | **[一手静态]** 处理答案文本并插入 chunk IDs。 | `do-not-take` 到 ContextRuntime；引擎不生成答案，BotDelivery只从当前 ContextPackage/Evidence closure构造引用。 | + +#### 映射的本仓 seam/模块/ADR + +- Candidate discovery：`PostgreSQLHybridCandidateIndex` 返回 FTS/vector 两个 `RankedCandidateList`；`fuse_candidate_evidence` pre-Kernel exact-ref dedupe并仅携带 rank evidence。 +- Authorization：sealed `AuthorizationKernel`是唯一 `CandidateRef -> AuthorizedProjection`构造者；它不可见/不依赖 rank权重，避免排名成为授权侧信道或裁决输入。 +- 授权后排序：`join_authorized_ranking`只 join admitted projections，并对 admitted candidates压缩每 ranker position后应用 server-owned weights。RAGFlow的固定 `0.05/0.95` 不能变成 caller参数。 +- 内容相关性：本地/外部 reranker只接受 `AuthorizedRerankItem`；模型输入、输出、profile、成本、egress在 governed inference port记录。 +- Expansion：`FragmentWindowRequest(anchor=AuthorizedProjection, ...)`；同 Article/current active Revision继承必须由 authoritative reader与 Kernel双检，cross-Article refs回到 Kernel。 + +**Kernel cut line（不可含糊）：** `CandidateQuery/RankedCandidateList → bounded exact-ref dedupe/RRF → tuple[CandidateRef] → AuthorizationKernel.authorizeAndProject → tuple[AuthorizedProjection] → admitted-rank compaction/weights → content rerank → expansion refs → Kernel reauthorization → PackageBudget`。`search.py` 中凡读取 `field`、正文、title、tag、TOC、parent或调用 relevance model的代码一律在 cut 之后，且不能直接接受 RAGFlow dict。 + +#### 复刻配方 + +1. 冻结现有 `CandidateQuery`/`RankerEvidence` canonical fixtures，加入一个与 RAGFlow fixed behavior对应的 weighted-sum oracle，但将它标为“quality comparison”，不是 RRF provenance。 +2. 给 FTS/vector lists定义总排序：ranker position优先，随后 `_candidate_sort_key`；同一 ranker重复 exact ref只计首次，不同 Article即使 fragment_ref相同也不 dedupe。 +3. pre-Kernel只保存每 ranker position与optional finite score；禁止 title/snippet/path/tag/vector/body进入 `CandidateRef`或 pre-Kernel trace。 +4. 继续使用 uniform provisional RRF排 bounded authorization order；真正 server-owned weights只在 admitted candidates上重新压缩 position并计算，denied candidate不得改变最终 admitted order。 +5. 若引入 RAGFlow式 lexical/vector score blending，新建 `AuthorizedScoreBlendProfile`，输入是授权后的投影+分离 rank evidence；先离线 ablation，再由唯一 ReleaseManifest promote owner激活，request body不能提交权重。 +6. reranker adapter nominal type只接受 `tuple[AuthorizedRerankItem,...]`；测试用反射/静态 gate禁止 `CandidateRef`、candidate session、DB connection或raw index fields传入。 +7. TOC/parent flow先输出 bounded `ExpansionPlan`：`same_article_current_revision_fragment_refs` 与 `reauthorization_refs` 两组；Kernel核对 active Revision、source ACL projection lineage与Article identity后才返回 content。 +8. PackageBudget在 expansion/rerank后统一选择；heading/parent context必须已在projection中可见计费，不能出现隐藏 parent text。 +9. 所有 rejected/denied rank evidence只进入受限安全 lineage的类别/摘要，不进入 tenant-visible ContextRun、debug或Learning corpus。 +10. 用同一 fixture跑现有实现和行为 oracle；结果差异以 retrieval metrics/ordering报告，不以“上游一致”自动批准。 + +没有建议复制的 `search.py` 代码，因此不新增 `UPSTREAM.toml`；把整文件列入现有 RAGFlow registration的 `excluded_paths` 是正确状态。 + +#### 验证计划 + +- `make test`：weighted RRF worked examples、重复ref、同fragment不同Article、tie total order、NaN/inf/overbound、denied candidate不改变admitted权重、hostile rank evidence、reranker nominal input、TOC/parent same/cross Article矩阵、budget可见性。 +- `make catalog`：pre-Kernel模块import allowlist只含 content-free types；Kernel不导入/访问 rank evidence；content consumers的type hints只允许 `AuthorizedProjection`/`AuthorizedRerankItem`；index filter无授权语义。 +- `make integration`：真实PG混入authorized/denied/cross-org/stale/tombstoned refs；断言denied正文、title、tag、TOC、parent进入reranker/assembler为0；same Article/current Revision window成功，stale Revision与cross Article未重授权均失败。 +- `make security-gate`：Unauthorized Evidence=0，并验证rank顺序/阈值/aggregate不泄露 denied existence。质量变化另进 `make eval-v1-execute` 的预注册slice，不可抵消安全失败。 + +#### 工作量估计与依赖 + +- 固定行为 oracle与现有 RRF审计：3–5 engineer-days。 +- authorized score blend + release profile：5–8 engineer-days，依赖 golden corpus/ablation和 release manifest schema。 +- governed reranker adapter：5–8 engineer-days，依赖 inference/egress port及预算计量。 +- TOC/parent expansion：6–10 engineer-days，依赖现有 FragmentWindow integration与 format compiler structural metadata。 + +最低下一步（只补证据）3–5天；完整内容rerank+expansion 16–26天。 + +### 3.3 `task_executor_refactor/chunk_builder.py` 与 v3 compiler + +#### 层级与上游路径 + +- **[一手静态]** [`get_parser`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/chunk_builder.py#L38-L67) 是 RAGFlow application-level factory，import十余 `rag.app`模块并用字符串/`ParserType`选 parser。 +- **[一手静态]** [`run_chunking`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/chunk_builder.py#L70-L109) 把 tenant/kb/task config、progress callback与limiter传给 `chunker.chunk`，返回 loose chunk dict。 +- **[一手静态]** [`extract_outline`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/chunk_builder.py#L112-L133) 从第一 chunk弹出 `__outline__`并直接写 `DocMetadataService`，写失败仅warning。 + +#### 映射与结论 + +结论是 **skip代码、adopt三个行为、adapt到owned seams**： + +1. adopt“parser由显式profile选择”，但选择权属于 immutable SourceVersion/CompilationProfile，不是task字符串； +2. adopt“解析受bounded limiter/deadline控制”，但runner由父WorkerLease和server config控制，不把tenant/kb/callback传进parser； +3. adopt“outline是结构输出”，但它必须在同一个typed compilation digest/Revision内，不能作为best-effort metadata side write。 + +它与ADR-0079 v3 compiler不能合并：v3已固定 exact Markdown grammar、raw byte spans、self-validating constructor、2048-token ceiling和owned subprocess。`chunk_builder.py`既不提供这些contract，也会重新引入parser registry、DB write与ambient settings。对PDF/DOCX应复用runner envelope/失败语义，而不是复用Markdown grammar或这个application orchestrator。 + +#### 复刻配方 + +1. 新建closed `CompilationProfileRef -> DocumentCompilerFactory` composition map，只有composition root能注册；未知profile在artifact read/model load前失败。 +2. `CompilerRunnerRequest`只含artifact bytes/digest、profile/version、bounds、可选verified model bundle ref与父job binding摘要；不得含tenant-authored parser id、DB service或callback。 +3. runner返回`ParsedDocument | CompilationFailure`的canonical bytes；progress只由父进程根据bounded lifecycle阶段记录，不允许parser任意消息落入durable tenant-visible记录。 +4. outline成为`ParsedSection`/document structural index的一部分；空outline是profile允许的明确值，解析异常则按closed category处理。 +5. 删除任何“从first chunk pop magic key”的contract；domain constructor拒绝未知字段、重复structure path与out-of-order units。 +6. Markdown v1/v2/v3 dispatch保持冻结；PDF/DOCX使用自己的format version，绝不让新factory默默重新解释已有Revision。 + +无复制项，故无新TOML或patch。 + +#### 验证计划 + +- `make test`：profile registry闭集、unknown version before I/O、runner request字段shape、no callback/DB/network、outline在digest内、failure无部分document、v1/v2/v3 frozen fixture不变。 +- `make catalog`：parser factory只能从composition root引用；raw compiler/runner private capability不被生产其他模块import；`DocMetadataService`等持久化符号不在parser dependency closure。 +- `make integration`：profile change创建new Revision；失败/outline差异不修改active；recovery在同一compiler/config/model digest下复用prepared boundary。 + +#### 工作量估计与依赖 + +2–4 engineer-days，依赖3.1的representation ADR。它是小型composition cleanup，不应成为独立parser framework项目。 + +### 3.4 comparator differential verification → Adapter parity gate + +#### 层级与上游路径 + +- **[一手静态]** [`RecordingContext`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/recording_context.py#L110-L258) 用context-local dict记录任意中间Python values与timing；`NullRecordingContext`给production关闭记录。 +- **[一手静态]** [`WriteOperationInterceptor`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/write_operation_interceptor.py#L27-L133) 用allowlist和FIFO回放旧路径写操作返回值。 +- **[一手静态]** [`ContextComparator.compare/compare_value`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/comparator.py#L92-L257) 去掉若干时间字段后递归比较；chunk比较按count、ID set、selected content/vector fields分层，见 [`_compare_chunks`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/comparator.py#L298-L454)。 +- **[一手静态]** [`TaskManager.dry_run_task`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/task_manager.py#L109-L180) 用旧执行记录构造interceptor、跑新handler、再比较；当前差异只logging，不是release veto。 + +#### 映射的本仓 seam/模块/ADR + +- 放在 `eval/parity/`，不进入production Runtime/Supply composition，也不拥有ReleaseManifest promotion。 +- 比较对象不是arbitrary locals，而是每个public/internal seam定义的canonical artifact：`ParsedDocument`、serialized `SupplyChangePage`、`CandidateQuery/FusedCandidates`、`AuthorizedProjection`的安全测试投影、publication plan等。 +- 安全seam不允许“等价容差”：Organization/Article/Revision/Fragment refs、source locators、正文、field projection、policy lineage、拒绝类别必须exact；float容差只适用于明确声明的quality measurement，且不能把NaN/缺字段当相等。 +- parity PASS只是替换必要条件，不是安全/质量/发布充分条件；仍需`make catalog/integration/security-gate`与唯一promote authority。 + +#### 复刻配方 + +1. 定义`ParityFixtureManifest`：fixture bytes SHA-256、format/profile/compiler/model identities、old/new implementation refs、expected outcome kind、允许差异清单及审批issue。fixture不可从生产tenant抓取。 +2. 为每个seam实现独立canonical serializer（RFC8785 JSON或既有canonical bytes）；serializer先用domain constructor自验证，再计算domain-separated SHA-256。 +3. `ParityExecutor`在两个隔离runner中对相同immutable input分别执行old/new；禁止共享mutable object、clock、random、network、DB connection或上一个实现的return values。 +4. 捕获closed stage records：input digest、output kind、canonical output digest、structure count/order、warning/refusal category、implementation/config/model digest、duration bucket。默认不保留正文；需要diff时写owner-only短期artifact并记录删除策略。 +5. comparator先比outcome kind，再比schema/ordered structural identities，再比canonical digest；digest不同才生成bounded field-path diff。不得像上游那样浅层strip任意`seconds/_created_time`后宣称等价。 +6. 预期representation变化必须在fixture manifest逐条登记`expected_divergence`和迁移理由；没有审批的差异全部FAIL。missing key、extra key、duplicate ID、order change、unconsumed intercepted write均FAIL。 +7. 写mutation tests：删除一个Fragment、交换顺序、改bbox/byte span、改Article、改模型digest、改warning/refusal、同ID不同content、向量维度变化、NaN、只改non-authoritative timing。只有最后一种按contract忽略。 +8. 对带副作用Adapter使用ephemeral owned sink记录**intent canonical digest**，不回放生产写返回值；两个实现均不得获得production DB credentials。随后在integration中分别执行同一intent到fresh fixture DB验证durable state digest。 +9. 报告closed状态：`EXACT_MATCH`、`APPROVED_DIVERGENCE`、`MISMATCH`、`REFUSED`；只有前两者且安全/许可证门另行通过，候选才可提交激活评审。 +10. 把fixture manifest、report schema与implementation digests纳入版本控制；大/敏感fixture按现有golden corpus durable-root规则保存并验证lineage。 + +无复制项：RAGFlow comparator的核心价值是迁移方法，不是其Python-object比较代码；不新增TOML。 + +#### 验证计划 + +- `make test`：canonical serializer、上述mutation matrix、expected divergence审批、refusal、report determinism、owner-only/redacted diff、两个fresh process。 +- `make catalog`：Eval executor无production secrets/ports、无promotion import、禁止caller-authoredsecurity observations；parity report不能写active pointer。 +- `make integration`：fresh PG各执行old/new intent，比较durable rows/events/active pointer digest；失败路径均无部分publication或wrong-org row。 +- `make security-gate`：parity工具本身不能绕开CandidateRef→Kernel→AuthorizedProjection或伪造clean security observation。 + +#### 工作量估计与依赖 + +6–9 engineer-days建立通用gate；每新增Adapter fixture 1–2天。依赖canonical serializers和golden artifact存储；不依赖PDF模型准入，可先覆盖Markdown、Candidate fusion和Supply page。 + +### 3.5 Task executor job/queue shape → signed WorkerLease runner lifecycle + +#### 层级与上游路径 + +- **[一手静态]** [`collect`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor.py#L203-L272) 先尝试Redis unacked iterator，再消费新消息；空、未知或已取消task会ack。部分task fields（包括某些tenant信息）来自消息。 +- **[一手静态]** [`handle_task`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor.py#L1719-L1785) 在finally后ack消息；异常更新progress，operation log另行记录。ack与业务DB状态不是一个事务。 +- **[一手静态]** [`insert_chunks`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor.py#L1270-L1368) 分batch插入并更新chunk IDs，取消时尝试删除部分RAPTOR写入；这是值得测试的crash/cancel窗口，但不是可继承的发布原子性。 +- **[一手静态]** [`report_status`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor.py#L1798-L1876) 向Redis写worker heartbeat、pending/lag/done/failed/current tasks，并清理过期executor；这是liveness观测，不是job authority。 +- **[一手静态]** 上游`TaskContext`只强制`id`与`tenant_id`存在，其他字段有大量默认值，见 [`TaskContext`](https://github.com/infiniflow/ragflow/blob/4391e03886b996201f3b8818f671b19eb24d0f7b/rag/svr/task_executor_refactor/task_context.py#L70-L280)。ContextEngine不能接受“字段存在”作为可信tenant/job绑定。 + +#### 映射的本仓 seam/模块/ADR + +- ADR-0075/现有 Supply bridge：PostgreSQL durable job是唯一truth；server-minted WorkerLease精确绑定Organization、SourceVersion、job、ServiceActor、allowed operation、audience、policy epoch、generation、nonce、idempotency和expiry。 +- runner是一次性subprocess，只消费exact job envelope、执行纯adapter/compiler、把bounded page/artifact发回父进程；无DB/queue/index/checkpoint持久化。 +- checkpoint只有在引擎durably接受完整change page后推进；publication watermark另算。queue delivery、heartbeat、parser success都不推进二者。 +- Recovery使用更高lease generation；被替换generation/nonce再无任何写效果。已有ADR-0060的bounded retry上限优先于上游无限unacked reclaim形状。 + +#### 复刻配方：要写进规范的生命周期行为 + +1. **Durable-before-dispatch：** job row先提交`pending`，固定Organization/Source/SourceVersion/operation/input digest/config/model bundle/attempt ceiling；dispatch消息只含opaque job ref，不能含可信tenant或可覆盖fields。 +2. **Atomic claim：** scheduler在function-only DB authority中选择当前可执行job，验证source active、ServiceActor、policy epoch、retry budget，原子写`claimed(generation, lease_digest, expires_at)`后签发lease。签名成功但未送达可在expiry后以更高generation恢复。 +3. **Pre-spawn verification：** 父worker在打开artifact、模型、DB transaction前验证signature、job/org/source/audience/operation/generation/nonce/expiry；缺字段、默认tenant、消息与DB不一致统一`work not available`且effect=0。 +4. **One-shot runner：** subprocess只收canonical request，通过固定IPC；限制wall time、CPU/memory、stdout/stderr、page/artifact bytes；超时/kill/crash转closed refusal，子进程没有凭证也不能ack job。 +5. **Stage recording：** append-only content-free events `claimed`, `runner_started`, `artifact_emitted`, `prepared`, `indexed`, `active`, `completed`, `refused`, `interrupted`, `reclaimed`；只记digests、counts、closed categories和generation，不记正文、denied IDs或parser stack trace。 +6. **Write interception：** 在candidate implementation/parity模式，runner输出被ephemeral sink捕获，绝不触发publication。生产模式中所有写由父进程已有definer functions/transaction执行；allowlist按typed command而不是字符串method name定义，未知command拒绝。 +7. **Idempotent boundaries：** 每个committed step比较job/generation/input/output digest；相同重放返回原receipt，不同digest拒绝。`prepared`完整包含Revision/Fragments/candidates；`active`只有CAS pointer transaction可见。 +8. **Recovery：** reclaim从最后一个verified durable checkpoint继续；prepared digest匹配时不重跑parser/embedding，缺失或不匹配则拒绝而不是猜测。旧generation在每个write function再次验证，不能在长runner结束后偷写。 +9. **Cancel/preempt：** cancel/preempt是trusted Control操作，提交cancel state并使当前generation失效；runner可被kill，但可见性由active pointer/tombstone决定，不靠best-effort index delete。需要替换still-live lease时必须显式operator reason digest。 +10. **Terminal acknowledgement：** 只有`completed/refused/terminal_failed` durable transition后才认为dispatch完成；若外部queue存在，ack在该receipt之后，ack失败只会导致安全的idempotent replay。V1不因上游有Redis就新增第二queue。 +11. **Liveness不授权：** heartbeat可报告opaque worker、stage、lease-expiry bucket、bounded counters；不能延长lease、选择Organization或证明job仍可写。lease renewal若未来需要，另立ADR，不能由heartbeat隐式实现。 +12. **Fault matrix：** 在claim前/后、runner spawn前/后、artifact receipt前/后、prepared/indexed/active transaction前/后、terminal transition与ack之间逐点kill；每点断言old-or-new visibility、effect≤1、old generation effect=0、checkpoint/watermark不倒退。 + +无复制项：Redis consumer、TaskContext、task manager和write interceptor均列入`excluded_paths`；吸收的是observable lifecycle oracle。 + +#### 验证计划 + +- `make test`：lease codec/envelope、missing context、wrong job/org/source/audience、expiry/replay/generation、runner timeout/oversize/malformed output、event redaction、typed write allowlist、idempotent step reducer。 +- `make catalog`：runner import/network/DB/credential禁令；worker role只能调用允许的functions；所有durable tenant tables FORCE RLS；无第二queue/active pointer authority。 +- `make integration`：真实PG concurrent claim、token-not-delivered、旧generation迟到、preempt、每个fault point、prepared recovery、terminal receipt后重放、checkpoint与watermark分离。 +- `make smoke`：API/worker/runner process readiness和一次one-shot job；green smoke不升级为publication/security证明。 +- `make security-gate`：cross-job/cross-org/replay/missing context的业务effect全为0，且worker refused details不进入tenant ContextRun/Learning。 + +#### 工作量估计与依赖 + +8–12 engineer-days补齐规范、fault fixtures与runner stage receipts；如果现有Supply bridge已覆盖部分项目，实际可压到5–8天。依赖真实PostgreSQL harness、现有WorkerLease/Supply bridge与compiler-runner envelope，不依赖RAGFlow部署。 + +## 4. 不可借鉴清单与必须杀死的隐含前提 + +| 上游形状/诱人捷径 | 处置 | 必须杀死的隐含前提 | ContextEngine硬证据 | +|---|---|---|---| +| 根Apache许可证覆盖一切 | 不接受 | 根license自动覆盖外部模型、训练资产、复制片段和所有依赖notice | 每个source/model/dependency精确revision+hash+license+SBOM；缺一即不准入 | +| `server/README`写Apache即可下载ONNX | 排除资产 | 文档声明等于模型本体/训练依赖法律审查 | 固定模型repo revision、本体hash、license/notice/model card；当前[未取证] | +| 复制整个`deepdoc/parser`或initializer | do-not-take | 同目录就是一个许可证/依赖区域 | exact path allowlist；initializer、server、rag/app/nlp/common耦合全排除 | +| parser成功返回loose dict/tuple | do-not-take | 有文字/框就能发布 | self-validating typed constructor、all-or-nothing、canonical digest、bounds、immutable publication | +| PDF bbox可冒充UTF-8 byte span | 杀死 | 所有格式共享Markdown provenance坐标 | format-specific locator ADR，绑定artifact/page/member/render digest | +| OCR word/table cell各自成chunk | do-not-take | 越细越利于检索且不损语义 | 一structural unit一Fragment；cell/word仅typed metadata；同Fragment ancestry/budget | +| HTML table是canonical内容 | do-not-take | 上游生成的HTML可安全存储/渲染 | typed table canonical form；任何UI rendering另做escaping/security review | +| runner可按环境变量/缺文件联网下载模型 | 禁止 | 运行环境是可信且网络永远可复现 | 父进程传digest-bound read-only bundle；runner网络=0 | +| random sampling和并行浮点推理仍天然deterministic | 杀死 | 同输入通常同输出即可支撑Revision identity | fixed sampling/ties/runtime target/quantization + fresh-process digest proof | +| `search.py`已有RRF | 纠正 | weighted_sum就是RRF | pinned源码只证明weighted_sum；本仓owned RRF worked fixture | +| candidate search默认取正文/title/tag | do-not-take | index-side过滤后取正文安全 | pre-Kernel content-free type/static gate；denied content consumer count=0 | +| stale document prune/index available flag授权 | 禁止 | index存在性或cleanup状态可裁决访问 | current Article policy/Revision/epoch由Kernel+PG authority核验 | +| child授权可继承parent/TOC/neighbor | 禁止 | 文档内部引用天然同授权atom/current revision | same Article/current Revision lineage验证；cross Article逐项重授权 | +| rerank/fusion阈值可在request里配 | 禁止 | 质量参数无安全影响 | server-owned versioned profile；授权后admitted ranks压缩；ReleaseManifest激活 | +| `chunk_builder`可作为通用compiler framework | do-not-take | parser registry+TaskContext等于typed deterministic seam | owned profile map + pure runner + typed result；DB write不在parser | +| outline best-effort side write | 禁止 | outline失败不影响Revision语义 | outline进入同compilation digest或closed warning/refusal；一次原子publication | +| arbitrary RecordingContext差异少即等价 | do-not-take | 被记录字段集合代表完整behavior | canonical seam artifacts + mutation-complete comparator + missing/extra/order/digest veto | +| 回放生产write返回值可证明新实现正确 | 禁止 | 相同返回值意味着相同持久化/安全效果 | ephemeral intent sink；fresh real-PG durable-state comparison | +| parity PASS可批准发布 | 禁止 | 行为相同就自动安全/合规/质量合格 | parity、license、Security、Reliability、Quality、Budget独立门；promote owner唯一 | +| Redis message携带tenant_id可构造worker context | 禁止 | queue producer与消息完整性就是tenant authority | DB-owned exact job + signed WorkerLease + ServiceActor；message只含opaque ref | +| unacked reclaim/heartbeat等于lease | 禁止 | liveness或consumer group ownership授予业务写权 | DB generation+nonce+expiry每次write重验证；heartbeat绝不续权 | +| 最后ack即可消除DB/queue crash window | 禁止 | ack与业务commit天然exactly once | durable idempotent terminal receipt先于ack；重放effect≤1 | +| cancel后best-effort删index即可撤权 | 禁止 | cleanup决定可见性/撤权 | active pointer/tombstone+Policy Epoch；cleanup异步且非authority | +| `make smoke`或上游单测数量证明能力 | 禁止 | process可启动等于安全/质量闭合 | `make test/catalog/integration/security-gate`分层；实际PDF/OCR corpus仍[未取证] | + +## 5. 推荐实现顺序 + 给 coordinator 的开放问题 + +### 推荐顺序 + +1. **先做检索与parity的低风险证据增量（3–5天）。** 给现有RRF/authorized weight compaction补RAGFlow weighted-sum对照fixture和cut-line catalog；明确报告“RRF不是来自固定`search.py`”。这不引入第三方代码或模型。 +2. **落通用Adapter parity gate（6–9天）。** 先覆盖Markdown v3、SupplyChangePage、Candidate fusion；以后DOCX/PDF候选都复用同一canonical digest/mutation gate。 +3. **决策format-neutral ParsedDocument/provenance ADR（4–6天）。** 未回答PDF bbox/DOCX OOXML locator、figure kind与model identity之前，不写parser产品代码。 +4. **DOCX+PDF outline copy+patch（6–9天）。** 完成专属third-party registration、依赖license文本、patch、SBOM、determinism与typed fixtures；仍不激活PDF OCR。 +5. **加固WorkerLease runner lifecycle（5–12天）。** 在DOCX runner进入production publication前补fault matrix、stage receipts、preempt/reclaim与terminal idempotency。 +6. **PDF/OCR source-only spike（8–12天，可与模型法务审计并行）。** 只在`/tmp`验证去网络、确定性、typed mapping和质量；不得复制到工作树。模型/source-lineage任一门失败即停。 +7. **满足明确质量需求后才产品化PDF/OCR（15–22天）。** 先选一个固定runtime target和最小profile（PDF text+outline，或scanned PDF OCR+layout+table），不要同时承诺Ascend、vision LLM、所有layout domains。 +8. **最后做authorized rerank/TOC/parent expansion（11–18天）。** 以golden slice ablation决定是否激活；所有内容从AuthorizedProjection开始，cross-Article reauth由integration/security gate证明。 + +### 给 coordinator 的开放问题 + +1. **ParsedDocument family：** 是否授权写新ADR，将当前Markdown-specific `ParsedDocument/SectionKind/SourceSpan`提升为format-neutral contract？建议答案是“共享publication interface，保留各format locator/profile的nominal subtype”，避免PDF bbox伪装byte span。 +2. **首个非Markdown格式：** 先做DOCX（低资产风险、6–9天）还是PDF text+outline（用户价值可能更高但会触发PDF大文件和provenance设计）？建议先DOCX，再PDF text-only，OCR另立准入。 +3. **Figure语义：** figure+caption/OCR是否是V1 structural Fragment kind，还是首版明确refuse/omit整份文档？不能静默丢图；建议新增`FIGURE`并使image bytes走separate bounded artifact/projection policy。 +4. **模型分发策略：** ContextEngine制品是否物理携带ONNX bundle，还是由独立受控安装步骤取得并校验？无论哪种都要固定revision/hash/license manifest；建议首版离线安装到只读bundle，runner不下载。 +5. **准入runtime target：** PDF/OCR是否只支持一个CPU/ONNX Runtime/architecture组合以取得determinism，还是必须跨macOS/Linux/arm64/x86_64？后者会显著增加量化与golden tolerance工作。 +6. **复刻批准记录：** DOCX/outline与PDF/OCR应各自使用哪个ContextEngine issue作为`UPSTREAM.toml.approval`？现有#124仅批准Markdown parser，不应被复用为新区域批准。 +7. **检索范围：** coordinator是否只需要补现有RRF的provenance/parity，还是要排期server-owned authorized score blend/reranker？建议前者立即做，后两者等待golden corpus ablation。 +8. **parity gate的替换对象：** 第一批应比较哪一对实现？建议先比较frozen Markdown v2/v3 canonical fixtures与现有candidate fusion，而不是尚未获准的RAGFlow PDF模型。 + +在这些问题得到决定前，可安全启动的只有顺序1–2；顺序3需要架构授权，顺序4需要新的第三方批准，顺序6–7还受模型许可证与动态质量证据双重否决。