feat(omni): S4 policy pipeline — three-modality degradation policies, Stage B tools, modelAccess gating - #8815
Merged
Merged
Conversation
…very/ingestion/storage)
- omni.upload.maxFileBytes -> omni.processing.transportGuard.maxUploadFileBytes (<=1GiB)
- omni.transport.maxEstimatedTokens -> omni.processing.transportGuard.maxEstimatedTokens
- omni.upload.cacheTtlHours -> omni.delivery.upload.urlTtlHours
- omni.download.maxFileBytes -> omni.ingestion.localization.url.maxFileBytes
- new: omni.processing.{limits,fixedPolicies,transportGuard.policies,policyTools},
omni.storage.quarantine.{retentionDays,maxBytes}
- core getters renamed to match; error messages cite new key paths
Experimental branch: one-shot rename, no migration shim.
ffprobe already reports these; the parse branches discarded them. The policy condition DSL (resource.bitRate / resource.sampleRateHz / resource.channels) and degradation disclosures both need the real numbers. Format-level bit_rate is preferred over the stream's.
…tor, modelAccess gate) Protocol core for the S4 policy pipeline: - ToolExecutionOrigin (model | client | fixed_policy) on ToolCallRequestInfo; only settable by in-process callers, never deserialized from protocol payloads; missing origin fails closed as model. - MediaPolicyToolDescriptor as a DeclarativeTool code-registration getter (default undefined) — config can never turn an ordinary tool into a policy tool. - PolicyArtifactBatch channel on ToolCallResponseInfo, capturing raw successful media-policy artifacts before PostToolUse hook merging; error/timeout results promote nothing. - Shared modelAccess resolver + call gate (omni/policy/model-access.ts): media-policy tools are fixed-policy-only unless omni.processing.policyTools.<name>.modelAccess.enabled; model calls get defaultArguments/lockedArguments merged and are rejected when they name a locked key; a forged fixed_policy origin on a non-media-policy tool is rejected. - Gate enforced on every model surface: registry declaration lists (incl. includeDeferred for subagents), ToolSearch keyword + select:, CoreToolScheduler pre-build, and ACP Session.runTool. - fixed_policy scheduler calls skip the interactive permission flow but keep PermissionManager tool-enablement and hook execution. - omni.processing.policyTools threaded through ConfigParameters (core + cli).
Storage design §4.3/§4.4/§6.1 for the policy pipeline: - OmniObjectStore gains staging/ and quarantine/ areas (0o700, symlink refusal in ensureLayout) plus the invocation lifecycle: exclusive createStagingDir (16-hex id validation, no silent reuse), removeStagingDir, and quarantineInvocation — which writes reason.json (policyId/toolName/reason/failedAt) into the staging directory before a single atomic rename into quarantine/<invocationId>/. - Startup recovery deletes everything under staging/ (entries belong to invocations that never committed) and trims quarantine/ to a retention window (default 7 days) and size budget (default 5 GiB, oldest-first), following the existing isRealDirectory containment convention: symlinked roots and entries are never traversed, sized, or deleted through.
Implement the restricted when-condition DSL from the policy design (§8.3): recursive all/any combinators over gt|gte|lt|lte|eq comparisons, with operands drawn from three read-only namespaces (resource.*, request.*, session.*) or literals. No arbitrary code, no JSONPath. Evaluation is three-valued. A comparison over an unresolvable field yields `unavailable` with the missing fields recorded — never a silent false — so the caller can apply the policy's onConditionUnavailable behavior and surface the fields in the run record. Combinators use strong Kleene logic: `all` with a false branch is a determinate no_match regardless of unavailable siblings; `any` with a true branch is a determinate match. Vacuous semantics: all [] → match, any [] → no_match. The evaluator is a total function that never throws — structurally malformed nodes degrade to `unavailable`, the fail-safe outcome. The separate structural validator (for the startup config-normalization pass, policy design §13 #5) rejects malformed conditions with path-annotated errors: exactly-one-of comparison/all/any, non-empty combinator arrays, exactly-one-of field/value per operand, known-field membership, and finite numeric literals for ordering operators. Condition types are re-exported from omni/policy/types.ts alongside the rest of the policy protocol surface.
Commit 6 of the S4 policy pipeline: the three built-in degradation tools (mapping doc §6), each a real BaseDeclarativeTool carrying a media_policy descriptor so the orchestrator and the modelAccess gate can key off code-level facts. - omni/policy/tools/media-policy-tool.ts: shared base — validates against the NATIVE parameter schema (never the model-visible projection Stage B will narrow), async io assertions (lstat: symlink input refused, real output directory required), per-tool policyTools.<tool>.runtime.timeoutMs resolution (default 600s), uniform success/error ToolResults; success emits exactly one lossy artifact whose metadata.omniDisclosure carries the D8 disclosure and whose workspacePath is staging-relative. - omni_downsample_image (sharp, lazy-loaded per D9; load failure is an execution failure): fit 1568px / JPEG q75, EXIF orientation baked in, animated inputs refused outright (sharp would silently keep only the first frame). - omni_downscale_video (ffmpeg): scale to even target height computed in JS (no filtergraph expressions), fps 10, x264 crf 28 veryfast; audio stream-copy with AAC 64k fallback. - omni_downsample_audio (ffmpeg): AAC 64kbps/16kHz/mono, -vn strips cover art. - omni/ffmpeg.ts: runFfmpeg — never rejects, callers branch on the exit code and must check signal.aborted explicitly. - Registered lazily behind isOmniEnabled(); the commit-3 modelAccess gate keeps them hidden from model surfaces by default.
Commit 7 of the S4 policy pipeline (decision D2): reusing a previous
transcode instead of re-paying it — a 424MB video downscale is
minutes-long and must not run once per delivery round.
- omni/json-cache-file.ts: the file mechanics extracted verbatim from
upload-cache (per-file serialized load-modify-save, atomic tmp+rename
0600 writes, corrupt backup+rebuild with capped .corrupt-* backups,
unreadable-but-existing file = operation no-op) as a shared
OmniJsonCacheFile — the design doc mandates the degradation cache
mirror these exact semantics, so they now exist once.
- omni/upload-cache.ts: refactored onto the shared file; entry
semantics (model/scope keys, TTL clamp, invalidation) unchanged — all
22 existing tests pass untouched.
- omni/policy/degradation-cache.ts: (originalSha256, policyFingerprint)
→ { degradedSha256, extension, disclosure, mimeType } at
.qwen/omni/policy-cache.json. policyFingerprint = sha256(toolName +
key-sorted tunables + tool version); per-invocation io params
(inputPath/outputDir) are excluded — they are plumbing, not policy
identity. Entries carry no TTL (content-addressed identities never go
stale); removeByOriginalSha256 / removeByDegradedSha256 serve the
GC/corruption cascades. Object existence checks stay with the
orchestrator.
…peline Introduce runFixedPolicies: for each recognized media resource it matches normalized fixed policies (modality, origins, when-conditions), executes the policy's media tool through the real executeToolCall protocol (fixed_policy execution origin, recordToolResult:false, isolated staging outputDir), validates the returned policy artifacts against the tool's declared descriptor (workspace containment, recognized kind/mime match, mandatory disclosure for lossy outputs), promotes derivatives into objects/ and records them in the degradation cache keyed by original sha + policy fingerprint. Reorder processMediaForOmniDelivery so transport guards judge the FINAL delivery set instead of the source: recognize -> recovery -> fixed policies -> guards -> hash -> upload. Degraded deliveries carry a disclosure that is emitted as a text Part immediately before the media Part everywhere media surfaces (read tool results, tool-result inline media conversion), and the OpenAI converter moves the disclosure together with its media part when splitting tool media into a follow-up user message.
…cies Startup normalization of the fixed-policy pipeline configuration (policy design §13 applicable subset): - omni/policy/config.ts: normalizeOmniProcessingConfig merges user settings over system defaults (id-merge, whole-entry replacement, null tombstones for fixedPolicies only) and validates structure, enums, when-conditions, tool references (registered + media_policy descriptor + required/lossy-disclosure outputs), fixed arguments against the io-stripped settingsSchema, reserved io keys, guard rules (no when, source=omit, mandatory three-modality coverage), limits (§12.2 defaults), the 1 GiB upload cap and the 48h URL TTL. Any violation throws OmniPolicyConfigError and aborts startup — a mis-configured guard must never degrade into sending over-limit media. - System defaults (D7 dual registration): the three degradation tools registered as preprocessing fixedPolicies WITH when-thresholds and as transportGuard.policies WITHOUT when. - core Config: thread fixedPolicies / transportGuard.policies / limits / quarantine settings; normalize in initialize() after tool warmup; expose getOmniProcessingConfig and quarantine getters. - cli: thread the new omni.processing / omni.storage.quarantine keys.
…tine (Stage B) - runFixedPolicies enforces maxPolicyRunsPerRoot (checked before each execution), maxArtifactsPerRoot and maxDerivedBytesPerRoot (checked as derivatives land); exhaustion records outcome 'budget_exhausted' while committed deliveries stand; maxLineageDepth clamps reprocessing. - Failed invocations quarantine their staging dir with a reason.json (policyId, toolName, reason, failedAt); user aborts and quarantine failures fall back to plain staging removal. Startup recovery receives quarantine retention/size settings from config. - Transport-guard violations now run modality-matched guard policies for up to maxTransportPasses passes on the final delivery; a still-violating resource is explicitly omitted (【媒体省略】notice replaces the part in both the read path and tool-result funnel) instead of delivered oversized; guard-pass failures stay fail-closed. Configs without a normalized processing config keep the Stage A throw. - Media-policy tools project a model-visible declaration (D6): locked arguments removed from properties/required, optional narrowing-only parameterSchema merge and description override; validation keeps the native schema.
…criptors All three degradation tools emit a disclosure at runtime, but their descriptors never declared the text output, so the system default policies failed the lossy-requires-disclosure validation (config §13 #8) at real startup. Stub-based unit tests missed the drift; config.test.ts now normalizes the defaults against the real tool instances.
…ssions, settings, sweep grace, concurrency) - validate degradation-cache entries and object-store path components (sha256/extension shape) so a poisoned policy-cache.json cannot traverse paths or serve malformed derivatives; malformed entries self-heal - re-hash cache-hit objects before reuse: mismatched bytes trigger a fresh transcode and heal the object store (D2 integrity) - default media policy tools to 'ask' permission via a shared BaseMediaPolicyToolInvocation (model-origin calls confirm outside yolo; fixed_policy runs are unaffected) - consume policyTools settings from config: tool-level settings defaults merge under policy arguments and feed the cache fingerprint so settings edits invalidate cached derivatives - deliver @url omission notices and degradation disclosures: omission replaces the fileData part with the notice text; disclosure text lands immediately before its fileData part (D8) - give staging sweep a 1h multi-process grace window (only entries older than the window are deleted; symlink entries removed regardless of age) - gate concurrent fixed-policy runs per omni root with a FIFO counting semaphore honoring maxConcurrentResources
Orchestrator: exclude animated images from policy matching (D9), self-heal unverifiable degradation-cache entries, key cache fingerprints with the descriptor version, and warn when reprocess origins exclude 'policy'. Scheduler: fixed_policy invocations no longer bounce PreToolUse 'ask' to an unanswerable awaiting_approval (deny stays fail-closed), and processToolResultImages short-circuits for fixed_policy results at the method level so all three call sites skip the omni re-delivery / vision bridge re-entrancy. Delivery/CLI: chain preprocessing and guard disclosures instead of replacing (D8), exclude hidden media-policy tools from the /context per-tool breakdown to keep it aligned with getFunctionDeclarations(), annotate fixed-only media-policy tools in /tools, and drop the stale maxConcurrency mention from the policyTools settings description. Tests: add negative coverage for artifact validation (no artifacts, undeclared media type, kind mismatch, missing required output) and for the fixed_policy hook ask/deny and image-funnel paths.
The three degradation tools were registered twice: as transportGuard policies (design-mandated, always-on, transport-limit triggered) and as fixedPolicies with invented when-thresholds (1568px/480p/96kbps). The upstream design gives fixedPolicies pure user-experiment semantics — a zero-config setup must not trigger any preprocessing below transport limits. Remove the fixedPolicies-side registration and its threshold constants; keep the guard defaults. Threshold acceptance for #8186 moves to explicitly-configured fixedPolicies in E2E.
…dio/clip/convert) - omni_extract_keyframes: ffmpeg scene detection (select+showinfo) with uniform-fps fallback, multi-artifact batch, per-frame disclosures - omni_extract_audio: audio track extraction to WAV/MP3/M4A, 16kHz mono WAV default (ASR-recommended shape) - omni_clip_video: frame-accurate time-axis cut (input-side -ss/-t, libx264 re-encode, faststart), rejects no-op invocations - omni_convert_image: sharp re-encode to JPEG/PNG/WEBP with EXIF orientation baked in, animated inputs refused - uniform lossy declaration per mapping doc §6.1: every output carries a disclosure since the representation change itself is lossy - extract shared sharp loader (sharp-module.ts) and describeChannels helper; register the four tools lazily behind isOmniEnabled()
… origin
The when-condition evaluator has supported the request.* and session.*
namespaces since the initial S4 landing, but no caller populated them, so
every policy referencing those fields resolved to 'unavailable'. This
fills both per policy design section 8.3:
- request.totalEstimatedMediaTokens is computed inside the orchestrator
from the pending-delivery work-item set at each pass start, so it is
recomputed as derivatives enter later passes. If any pending resource
is unestimable the whole namespace reads unavailable - a partial sum
must never pass thresholds as a smaller total. Callers can override
via options.conditionContext.request (future multi-root aggregation).
- session.* is snapshotted once per media delivery by the new
policy/session-context.ts (contextWindowTokens from
contentGeneratorConfig.contextWindowSize, promptTokenCount from the
CURRENT chat's getLastPromptTokenCount, availableContextTokens as a
clamped subtraction only when both inputs are known) and reused across
preprocessing and every transport-guard pass of that delivery.
Also wires the client execution origin: slash commands scheduling a tool
(schedule_tool in useGeminiStream) now stamp executionOrigin
{kind:'client'}, putting the in-process client channel under the same
media-policy modelAccess gate as model calls.
…cies A fixed policy producing multiple media deliverables (omni_extract_keyframes promoting N frames) previously failed delivery with "exactly one is supported" after all frames were already atomically promoted to objects/ — zero frames reached the model, violating the multi-output transaction visibility acceptance. Fixed-side contract now: deliveries[0] is the primary (unchanged path: guard re-check, upload, disclosure); the rest ride along as OmniMediaDelivery.additionalMedia, each independently judged against the transport limits and uploaded through the same hash → upload-cache → objects promotion → DashScope pipeline. Extras are processed lazily after the primary settles, so the primary uploads first and a fail-closed primary never wastes extra uploads. An over-limit extra becomes an explicit omission entry (visible reason, no guard-policy re-derivation — it is already a policy product). The guard side keeps its strict 1:1 semantics. All three delivery consumers (readMediaViaOmniDelivery, tool-result media conversion, the @-command URL funnel) materialize extras through a shared buildAdditionalMediaParts helper: per-extra [disclosure?, fileData | omission text] pairs after the primary slot, before transcripts, preserving disclosure adjacency per pair.
…ard db mime/lite's default database carries no mapping for .mkv/.avi/.flac/.aac, so detectFileType fell through to the binary content sampler and a real Matroska movie was rejected by the 10MB inline cap instead of entering media delivery. Generalize the existing .m4v override map to cover these containers (authoritative types from the full mime db), scoped to extensions the omni recognizer can sniff-confirm so extension lies still fall back to the legacy path.
- Validate operator-configured defaultArguments/lockedArguments against a tool's NATIVE parameterSchema instead of the model-visible projection, which strips exactly those keys and rejected every legitimate locked or operator-only argument at startup; regression test uses real tools. - Write cache files with noFollow so the atomic rename replaces a symlink planted at the cache path instead of writing (and chmod-ing) through it. - Route every .part staging write through prepareOmniDownloadsDir, a fail-closed symlink guard (mkdir recursive succeeds silently on a link). - Detect ADTS AAC in the MPEG-audio sniffer (layer bits 00) so .aac streams stop being labeled audio/mpeg. - Preserve metadata.omniRole in degradation-cache entries so cached reruns keep the same artifact matching as fresh derivations. - Fall back to the default quarantine retention when the configured value is zero/negative/NaN instead of expiring the whole quarantine. - Propagate error causes through OmniTransportGuardError and document the operatorOnlyParams contract on MediaPolicyToolDescriptor.
- Single-source tool names from ToolNames across all eight policy tools and the config validator. - Share the config view on the base class (protected configView) instead of eight shadowed per-subclass fields; move the view type and the isPlainRecord helper to policy/types. - Memoize the model-visible schema projection on the settings object's identity (one normalized settings object per initialize()). - Extract sharpTimeoutSeconds for the duplicated libvips timeout conversion and route clip-video's io validation through the base validateToolParamValues.
…sumers Extract buildTranscriptParts (§6.2/D8 ordering: each transcript follows its media part or omission notice, preceded by its own disclosure) and use it from readMediaViaOmniDelivery, the tool-result funnel, and the CLI @-command URL funnel instead of three hand-rolled loops. Also extract the textOnlyDelivery helper, upload additional-media extras concurrently (content-addressed store and per-file serialized upload cache make them independent; map preserves deliverable order), and drop the now-consumerless formatTranscriptText barrel export.
Validate a batch's artifacts and promote them into the object store concurrently — they are independent files in a shared staging dir and the store is content-addressed with race-safe puts; map keeps the batch order and a sequential partition preserves derived/derivedFiles order. File artifacts hash the bytes already in memory for the UTF-8 check instead of streaming the file a second time.
LaZzyMan
requested review from
tanzhenxin,
wenshao and
yiliang114
as code owners
August 9, 2026 12:17
LaZzyMan
added
the
omni-experiment
Collaborator
Author
S4 E2E 结果表（真实 DashScope API，模型 qwen3.5-omni-plus）执行方式： 验收组（A–G，issue #8186/#8187 验收项）
小结：21 PASS / 1 N/A / 0 FAIL。 8 工具 × {固定调用, 模型调用} 全矩阵
小结：20/20 PASS（M4a 初测 FAIL → 修复后复验 PASS）。 扩展组：复杂条件编排 / metadata 字段 / 真实电影场景
小结：扩展组 13/13 PASS；期间修复 1 个真实缺陷（ 自查 review（合并前质量轮）多轮自查共修复 12 项正确性发现（含启动校验误用模型可见投影 schema、缓存写入符号链接写穿、ADTS AAC 误判 mp3、quarantine 保留期非法值等），全部附回归测试；另有并发化（extras 上传 / orchestrator 验证与晋升）与结构整理（transcript 物化器共享、工具管件合并）。终态验证：core tsc 0 error、omni 单测 656/656、cli 相关单测 75/75、E2E 冒烟复验通过。 |
Collaborator
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
Animated WebP/APNG containers report no nb_frames, so the D9 animation exclusion silently passed them and the image tools kept only the first frame with no disclosure. Two independent gates now close this: - the ffprobe path re-probes animation-capable containers (gif/webp/png/apng) with -count_frames when nb_frames is absent - both image tools additionally check sharp's metadata().pages before encoding and refuse multi-page inputs
- maxWidth/maxHeight are same-value aliases of width/height, not independent axis caps - request.totalEstimatedMediaTokens is the per-resource estimate (resource plus its derivatives) within the current scheduling pass, not a cross-resource total
A JSON settings file can carry "__proto__" as an ordinary key; spreading it into a plain object routes it through the prototype setter and the entry silently vanishes. Merge policy maps onto a null-prototype object so the id reaches normalizePolicy and is rejected by the id pattern.
Settings load performs no runtime type checks and guard.ts compares the threshold with <=/>: a string value makes both comparisons false and silently disables the token guard (fail-open). Reject anything but a finite number >= 0 at normalization, and thread the value through Config so the check actually sees it.
The runtime accessors treat non-positive values as absent and fall back to the defaults, so a configured 0 never means "delete immediately" / "no budget" — advertising minimum: 0 in the IDE schema promised a semantic the code does not honor. Document the fallback in the descriptions as well.
There are no built-in default fixed policies, so "null tombstones a default policy" described a merge target that does not exist. Describe what the merge actually does: entries merge by id across settings scopes and a null entry tombstones a policy from a lower-priority scope.
Media policy tools override getDefaultPermission but left
toAutoClassifierInput at the empty-string sentinel, so the AUTO-mode
classifier saw "Arguments: {}" and its path-based block rules could
never fire on a model-origin call. Project exactly inputPath and
outputDir — the fields those rules key on, carrying no secrets.
A typo like "settigns" or "modelaccess" read as an absent optional section downstream, so the intended configuration silently never took effect. Fail startup on unknown keys at the entry, runtime, and modelAccess levels, and drop the stale maxConcurrency mention from the runtime doc comment (timeoutMs is the only supported limit).
…aints Design §11.2: a projection may only narrow the tool's native schema — 类型、枚举、范围 included, not just the property set. The declaration merges each projected property's keys over the native ones, so an override raising maximum, lowering minimum, widening an enum, changing the type, or relaxing min/max length/items promised the model a range the native per-call validation then rejects. Validate the merged property schema against the native effective bounds at startup (minimum/exclusiveMinimum interplay included).
A source already below the target bit rate, sample rate, or channel count re-encoded into a LARGER derivative that the transport guard counted as progress, under a disclosure claiming 高频细节丢失/声道合并 losses that never happened. Clamp each target to the probed source (the withoutEnlargement analogue of the image/video tools) and build the loss clause from the drops that actually occurred (D8).
clip_video is a time-axis cut, not a degradation tool: a span covering the entire video only burns a lossy re-encode while the disclosure falsely claims content outside the span was discarded. Reject startSec 0 with no durationSec at the parameter layer, and a probed-duration-covering span before the transcode starts.
When some buckets yield no frame (scene attempt and midpoint fallback both empty), the blanket 全片分桶采样 note overstated coverage — the un-sampled buckets' time ranges were silently invisible to the model. Disclose 仅覆盖 N/M 个分桶，其余时段未采样 whenever fewer frames than buckets survive (D8).
…urces A JPEG→JPEG re-encode cannot drop a transparency channel that never existed, yet the disclosure asserted 透明通道丢弃 unconditionally. Derive the loss clause from the probed input codec: alpha-capable formats (png/webp/gif/tiff) keep the claim, others get 元数据丢弃, and an unknown codec hedges with 如有 (D8).
A 360p@8fps input downscaled against the 480p/10fps defaults lowers neither dimension, yet the disclosure always claimed 分辨率与帧率下降 — contradicting the before/after figures shown right next to it. Emit 分辨率下降/帧率下降 only for the parameters that actually dropped, falling back to 重新编码压缩 for a pure re-encode (D8).
…zoom hint On the degradation path delivery.recognized re-recognizes the DERIVATIVE, so the hint's width/height are the downsampled dimensions. Calling them "full resolution" contradicted the degradation disclosure pushed immediately below and steered the model away from zoom_image — the exact remedy for degradation-stripped detail, since it reads the original from disk.
The '[fixed-only: runs via media policies, not the model]' suffix was appended as a hard-coded English string, so zh/zh-TW tool listings showed a mixed-language description. Route it through the i18n table like the display names it accompanies.
降清视频 was a coined term inconsistent with the 降采样图像/降采样音频 siblings; use 降采样视频 (zh-TW: 降採樣影片) across the CLI locales and the web-shell tool-name table.
…ions
The write-temp files are named <name>.<random>.tmp; filtering on
'.tmp-' could never match anything, so the leak assertions were
vacuously green. Filter on the real endsWith('.tmp') suffix.
…f the span floor The 好的， unit (3 chars × 7 reps = 21 chars) sat below the 24-char span floor, so the test passed for the wrong reason and REPETITION_MIN_REPS itself was unpinned. Use a 6-char unit so 7 reps already clear the span floor, and assert the collapse output at 8 reps.
Startup recovery's sample verification deleted a corrupt object and its upload-cache entry but left policy-cache.json untouched, accumulating entries that can never be served again. Cascade the deletion both ways — the object may be a policy SOURCE (its derivatives can no longer be re-verified) or a policy DERIVATIVE (entries pointing at it are dead) — mirroring the storage design's upload-cache cascade. The orchestrator's lazy self-heal only fires on a cache hit; the sweep side is proactive.
The role existence check consulted a Set while the transcript shape check re-scanned descriptor.outputs with a find, duplicating the first-declaration-wins semantics across two structures. A single Map<role, spec> now serves both.
when is preprocessing's ONLY trigger mechanism: a normalization regression that drops or rewrites it would silently widen every user condition to ALL matching resources, with no failing test. Pin that a valid condition tree survives normalization verbatim.
…ery §5) The timeoutMs < STAGING_GRACE_MS cap is a cross-file invariant: a tool allowed to run for the full grace window could have its live staging directory swept as crash leftovers mid-run by another process's startup recovery. Pin BOTH sides of the boundary against the exported constant so removing, inverting, or relocating the cap fails a test.
wangyuenlp
approved these changes
Aug 11, 2026
This was referenced Aug 13, 2026
Closed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does
Implements the S4 policy slice of the omni experiment — S4a (policy pipeline: fixed degradation policies for all three modalities) and S4b (capability completion: Stage B tools, model access gating, quarantine/budget guardrails) — as one coherent pipeline: recognize → match fixed policies → derive in staging → promote content-addressed → deliver with mandatory disclosures.
Policy protocol (
tools/tools.ts,omni/policy/): media-policy tools are realDeclarativeTools with amediaPolicyDescriptor(input modalities, output declarations with mandatory lossy disclosures, settings schema, operator-only params). They execute through the ordinary scheduler under three origins —fixed_policy(config-driven, no confirmation),client, andmodel(both gated).Conditions DSL (
policy/conditions.ts):whenis a Mapbox-style expression array — comparisons[">", ["field", "resource.width"], 3000]with>/>=/</<=/==/!=, combinators["all", …]/["any", …]/["!", expr]— overresource.*(width/height/durationMs/frameRate/sampleRateHz/channels/bitRate/sizeBytes/estimatedTokenCount/maxWidth/maxHeight…),request.*andsession.*namespaces, with startup schema validation and fail-fast on contradictory policy sets (e.g.reprocessMediawith no downstreamorigins: ["policy"]consumer). Evaluation is three-valued: an unresolvable field yieldsunavailable(never silently false), handled per-policy viaonConditionUnavailable.Orchestrator (
policy/orchestrator.ts): per-root pass loop with budgets (maxPolicyRunsPerRoot,maxDerivedBytesPerRoot, committed deliveries always stand), staging-dir isolation per invocation, artifact validation + content-addressed promotion (concurrent per batch), policy→policy chaining viaoutput.reprocessMedia+origins, and a degradation cache keyed by source hash + policy identity so unchanged inputs never re-derive.Nine Stage B tools: downsample image/audio, downscale video (thresholded degradation), extract keyframes (multi-frame delivery), extract audio, clip video, convert image, transcribe audio (qwen3.5-omni ASR, §6.2 transcript-protocol text delivery), all ffmpeg/sharp with per-tool
runtime.timeoutMs.modelAccess gate: tools hidden from the model by default;
enabledexposes a projected schema withlockedArgumentsand operator-only params (baseUrl,apiKeyEnv, …) stripped; locked arguments are hard-rejected when supplied, injected at execution. Startup validation ofdefaultArguments/lockedArgumentsruns against the tool's native schema.Multi-deliverable delivery:
additionalMedia[]+ a shared transcript materializer so multi-output policies (keyframes) and text deliverables (transcripts) reach all three funnels (user@-mention, tool results, URL ingestion) with D8-adjacent disclosures; transport-guard re-filtering and explicit omission notices for over-limit extras.Quarantine + recovery (
omni/recovery.ts): failed derivations move staging residue toquarantine/<invocationId>/withreason.json; lazy sweeps enforce retention days and a size budget (oldest-first); uncommitted staging from crashes is removed after a grace window.Config namespace rename: settings move to the design namespace (
omni.processing/delivery/ingestion/storage). This is a rename without a compatibility shim — the omni namespace is experimental and carries no migration path (design §2.2/§12); legacy keys are simply no longer read.policyTools.<name>carries settings/runtime/modelAccess per tool.Media detection fix (
fileUtils.ts):.mkv/.avi/.flac/.aacare missing from mime/lite's db and previously fell through to binary sampling — a 986MB movie never reached omni delivery. Generalized the existing.m4voverride table toMIME_LITE_MISSING_MEDIA_TYPES(only sniffer-confirmable containers).Progressive-understanding guidance (
omni/media-guidance.ts): a system-prompt section explaining the three disclosure markers (【媒体降质】/【媒体省略】/【媒体转写】), stating that degraded deliveries are an overview rather than the complete content, forbidding extrapolation beyond delivered evidence, and listing exactly themodelAccess-enabled media tools so the model can fetch higher-fidelity evidence itself.Full-duration keyframes (
extract-keyframes.ts): bucketed sampling replaces first-N scene detection — the timeline splits intomaxFramesequal buckets, each contributing one frame (scene change in the bucket's opening window, midpoint fallback), with per-frame absolute timestamps in the disclosures. Previously all frames clustered in the opening minutes of a long video.Chunked long-audio transcription (
transcribe-audio.ts): audio longer thanchunkSeconds(default 180s) is cut and transcribed per segment (concurrency 3, one shared wall-clock budget), assembled with[MM:SS-MM:SS]/[H:MM:SS-…]range labels; per-segment failures degrade to inline markers instead of failing the run; a repetition-degeneration detector collapses runaway repeated tails (e.g.Hej! Hej! Hej! …from single-shot ASR on an 81-minute track).Design mapping:
.qwen/design/omni-s4-policy-pipeline.md(decisions D1–D11, §10 in-flight revisions ①–⑥).Why it's needed
Closes #8186
Closes #8187
After S3, every media file is delivered as-is: a 4096×3072 screenshot burns ~5.9k tokens, a 1080p30 video costs megabytes of upload for a question answerable at 480p10, and nothing lets an operator say "downsample anything over N px before it reaches the model". S4 adds the policy layer that degrades media deterministically under operator-declared conditions — with mandatory disclosures so the model knows what it is looking at — and completes the toolset so the model itself can clip/extract/transcribe under the same gates when the operator opts in.
Reviewer Test Plan
How to verify
Unit suites (all green):
E2E acceptance ran against the real DashScope API with
qwen3.5-omni-plus(npm run build && npm run bundle,node dist/cli.js): a 22-case acceptance suite (groups A–G), an 8-tool × {fixed, model-invoked} full matrix (20 cases), and an extended suite for complex condition orchestration (C1–C5), metadata-field coverage (MD1–MD7), and a real 986MB movie scenario combining fixed policies with agent tool calls (R1). Full results table is posted as a PR comment; per-case evidence (hashes, ffprobe output, request payloads) lives in the working notes.Spot-checks worth doing in review:
"when": [">", ["field", "resource.width"], 3000]→ oversized image delivers a derivative (different sha256) with a【媒体降质】…text part immediately before the media part; small image delivers byte-identical.modelAccess.enabled + lockedArguments: {quality: 75}→ request schema omitsquality; supplying it anyway is hard-rejected with a retryable error.output.reprocessMedia: true) → downsample-audio (origins: ["policy", …]) delivers exactly one final audio part; dropping theoriginsentry fails startup with a contradictory-config error.kill -9mid-transcode → next media run sweeps the uncommitted staging dir; a failed derivation lands inquarantine/withreason.json.Evidence (Before & After)
See the E2E results comment (pre/post columns per case).
Tested on
Environment (optional)
Local vitest + tsc; E2E against DashScope with
qwen3.5-omni-plus(real uploads, real ASR).Risk & Scope
runtime.timeoutMsand amortized by the degradation cache (repeat deliveries are cache hits). Derivation failures never fail the turn —onFailuresemantics deliver the original or omit explicitly.PolicyExecutionRecordis currently discarded by callers (S5 Memory will consume it); GC/capacity budgets forobjects/(S6); animated-image frame policies (excluded by D9).omni.processing.fixedPolicies,omni.processing.policyTools,omni.processing.transportGuard,omni.storage.*.