feat: add Video Bridge frame sampling

[diegosouzapw]

Summary

• Add an opt-in Video Bridge that converts video inputs into timestamped visual descriptions before requests reach models without known native video support.
• Support OpenAI- and Anthropic-shaped video parts across Chat Completions and the Responses API, including HTTPS URLs and bounded data:video/*;base64,... inputs.
• Preserve native video payloads for models with confirmed video support. For proven text-only targets, replace video parts with captions or safe failure markers so unsupported raw media is never forwarded.
• Sample 1–16 uniformly spaced midpoint frames per video (default 8), caption them through the configured Vision Bridge model, and preserve the actual successful fallback producer in cache, metadata, statistics, and x-omniroute-modality-bridge.
• Replace the Video “coming soon” state with functional settings, runtime availability, model selection, bounded controls, honest telemetry, and localized copy across all 43 locale catalogs.
• Document the runtime, management APIs, security boundary, quotas, FFmpeg dependency, settings, telemetry, and failure behavior.

Architecture

• VideoBridgeGuardrail runs after Vision and Audio in the existing pre-call guardrail pipeline.
• Shared media detection recognizes OpenAI input_video / video_url and Anthropic type: "video" source shapes without inferring video from ordinary filename text.
• Capability resolution adds a trusted tri-state supportsVideo: true | false | null contract from registry/spec/catalog modality data rather than model-name heuristics.
• /api/models exposes real supportsVision values through a request-local bulk capability snapshot. Optional custom-model DB overrides fail open and use one bulk read per request instead of N+1 SQLite queries.
• Per-frame captions reuse the existing Modality Bridge cache and record attempts, success/failure, cache hits, frames requested/extracted/used, duration, actual producer model, total sampled latency, and latencySamples.

Security and bounded runtime

• The public /v1 request path never imports or launches FFmpeg. It downloads or decodes bounded bytes and self-hops to an internal extraction broker.
• Remote videos require HTTPS on the initial URL and every redirect and reuse the public-only SSRF guard with DNS pinning.
• Limits are explicit: remote/broker input 50 MiB, inline decoded video 36 MiB, serialized broker response 32 MiB, queue 4 pending / 100 MiB, source duration 600 seconds, 1–16 frames, JPEG frame 4 MiB, aggregate raw frames 23 MiB, maximum edge 1,024 px, and a fixed 120-second broker deadline.
• Inline base64 length is estimated and rejected before allocating a decoded buffer.
• /api/modality-bridge/video/ is both LOCAL_ONLY and SPAWN_CAPABLE, requires trusted stamped loopback plus a timing-safe per-process token, accepts only application/octet-stream, and exposes no URL/path/executable/argv input.
• Queue exhaustion maps to 503 + Retry-After, client disconnect to 499, and broker deadline to 504.
• FFmpeg/ffprobe use fixed execFile argument arrays with no shell, -nostdin, one thread, file-only protocols, and a format allowlist that excludes playlists/manifests.
• Attached-picture cover streams are excluded from playable candidates. All playable streams are validated; the default stream is preferred with deterministic lowest-index fallback, then passed to an exact -map 0:<index>.
• Abort propagates through download, broker, extraction, and captions. Private temporary directories are recursively removed in finally. Runtime errors are sanitized and do not expose commands, paths, stderr, stacks, or upstream secrets.

Related Issues

• Closes feat(backend): Video bridge (frame sampling + captioning) - backlog da série Modality Bridge #9760
• Related to feat(guardrails): modality bridge core — vision mode/task-aware/cache/input_image + modalityBridge settings #9759
• Related to feat(dashboard): Modality Bridge settings page (vision tabs, model selector, stats, test button) #9782
• Related to feat: bridge audio inputs for text-only models #9807

Validation

Choose the change type and focused loop from the
Contribution Golden Path. The full unit suite,
Vitest, the 60% coverage gate, and the production build all run in CI on this PR (#8329):

• Change type: routing / UI / i18n / DB / guardrail / internal runtime / security / docs
• Focused tests and category gates from the golden path
• npm run lint
• Implementation freeze reconciled with the active release base; focused checks rerun afterward
  • Implementation freeze base: ee221d870c199bc1aa1f3b90303ff2bc7c74509b
  • Final remote head: fab604ad287527eaba18f3318a19e2abef1c51e4
  • Final tree: 66a58aa057a520643958933efcad4ccf4b805abf
  • Merge-gate release tip: 282c087c271aa6fa43012bba349ad5644e22214b; GitHub reports the PR MERGEABLE/CLEAN with no review threads
  • Mutation coverage catalog repair: npm run check:mutation-test-coverage GREEN (4,031 unit files / 31 mutated modules / zero drift)
  • Post-rebase Node focused: 114/114
  • Post-rebase UI focused: 15/15
  • Post-rebase quick hermetic/full-ci: releaseGreen=true, HARD=0, DRIFT=0
• Production-code changes include new and updated automated tests in this PR
• SonarQube is temporarily opt-in while the private project has no quota; it is not a PR gate.

Full VPS .15 matrix

• Feature tracers captured RED before their corresponding fixes and GREEN afterward, including broker/authz, stream mapping, DB fail-open, fallback identity/cache, HTTP status semantics, Anthropic URL handling, inline allocation cap, telemetry, docs, and attached-picture handling.
• Source review along Standards and Spec axes: PASS after four rounds, with no remaining feature finding.
• Focused Node aggregates: 159/159 and final post-rebase 114/114.
• Focused UI aggregates: 24/24 and final post-rebase 15/15.
• Dashboard unit (concurrency 4): 111/111.
• Unit serial: 22/22.
• Vitest: 40 files, 362/362.
• Coverage, using the official 4,065-file selection sharded at concurrency 4 because monolithic V8 JSON exhausted the homologation disk:
  • statements 82.08% (526109/640948)
  • lines 82.08%
  • functions 87.01% (19032/21873)
  • branches 81.24% (98725/121513)
  • manifest 4,065 / processed 4,065 / unique 4,065 / duplicates 0 / gaps 0 / report+merge failures 0
• Quick hermetic/full-ci static matrix: 52/52 checks, 42 HARD with 0 bad, 10 DRIFT with 0 bad.
• Lint, core typecheck, dashboard ratchet, Open-SSE ratchet, cycles, discovery, tracked artifacts, dead code, file size, complexity, type coverage, DB rules, public credentials, docs, i18n, OpenAPI, route guards, security tiers, changelog fragment, workflows, secrets, licenses, vulnerabilities, test masking, and diff-check.
• Full PR production build: Fast Production Build passed on final head fab604ad287527eaba18f3318a19e2abef1c51e4 (Build App run 31897271838).
• PR CI: Fast Quality, four unit shards, Vitest, ESLint, docs, merge integrity, DAST, CodeQL, Semgrep, and production build all passed on the final remote head.
• Exact merged-release artifact / tarball boot smoke: post-merge deployment gate. The 5.8 GiB VPS cannot physically compile the Next.js application locally:
  • default Turbopack: kernel OOM;
  • Webpack with 3 GiB and 4 GiB heaps: V8 heap OOM;
  • backend-only fallbacks: also exceeded the VPS memory ceiling.
    No TypeScript or functional compiler error was emitted; no incomplete artifact was treated as valid.
• Live FFmpeg proof, deploy, and authenticated browser QA: intentionally post-merge gates.

⚠️ base-red inherited: #9985

The candidate was compared with a pure snapshot of its exact release base. The branch does not absorb unrelated release repairs.

• Main unit (concurrency 4): 31,244 tests; 31,219 pass; 3 fail; 22 skip. The same three failures reproduce on pure base:
  1. Continue/cn runtime detection;
  2. consoleInterceptor notice emission;
  3. adopted PID state in ServiceSupervisor.
• Integration: 954 tests; 859 pass; 20 fail; 75 skip across 11 files. All 20 failures and the batch/resilience cleanup stalls reproduce on pure base.
• typecheck:noimplicit:core: the candidate and pure base produce the same inherited diagnostic set in open-sse/utils/usageTracking.ts and src/shared/services/cliRuntime.ts; normalized diff is empty.
• Base issue: 🔴 Release branch not green: release/v3.8.50 #9985 remains open and records release-level unit/integration/package reds.

Result of the discriminator: zero unit or integration regression attributable to Video Bridge.

Tests Added Or Updated

Added

• tests/unit/guardrails/videoBridge.test.ts
• tests/unit/guardrails/videoBridgeHelpers.test.ts
• tests/unit/guardrails/videoBridgeRuntime.test.ts
• tests/unit/modality-bridge-video-i18n.test.ts
• tests/unit/modality-bridge-video-runtime-route.test.ts
• tests/unit/remote-media-fetch.test.ts
• tests/unit/ui/modality-bridge-video-tab.test.tsx
• tests/unit/video-bridge-broker.test.ts
• tests/unit/video-bridge-header-stats.test.ts
• tests/unit/video-bridge-media-capabilities.test.ts
• tests/unit/video-bridge-route-security.test.ts
• tests/unit/video-bridge-settings.test.ts

Updated

• tests/unit/api-models-hide-paid-6328.test.ts
• tests/unit/authz/spawn-capable-prefixes-client-safe.test.ts
• tests/unit/body-size-guard.test.ts
• tests/unit/guardrails-registry.test.ts
• tests/unit/guardrails/visionBridgeHelpers.callVisionModel.test.ts
• tests/unit/ui/modality-bridge-audio-tab.test.tsx
• tests/unit/ui/modality-bridge-vision-tab.test.tsx

Coverage Notes

• Production changes are covered at the media parser, capability resolver, DB snapshot, remote-fetch, runtime, broker, guardrail, request pipeline, API route, telemetry/header, settings, UI, i18n, authz, and documentation-contract layers.
• Safe-failure coverage includes native/unknown/known-text capability, missing runtime, client abort, queue saturation, deadline, oversized input/output, unsafe metadata, malformed streams, attached-picture covers, DB failure, model fallback, cache identity, and partial caption failure.
• Vision and Audio regressions remain covered for shared fetch/abort, stats, cache, and header behavior.
• Final consolidated coverage is 82.08 / 82.08 / 87.01 / 81.24 (statements / lines / functions / branches), above the project 60% thresholds.

Reviewer Notes

• Video Bridge is disabled by default. Enabling it adds preprocessing latency and vision-model usage.
• FFmpeg and ffprobe are optional operational dependencies and are not bundled. No configurable executable path is exposed.
• There is no database migration; settings use the existing validated dynamic key-value storage.
• The internal extraction endpoint is not a public upload API. It is an exact trusted-loopback, per-process-authenticated byte broker classified as both LOCAL_ONLY and SPAWN_CAPABLE.
• A target with supportsVideo === true is untouched. A known text-only target receives descriptions or safe stubs. An unknown-capability target preserves failed/unprocessed original video parts.
• maxVideos limits caption work, not detection, so excess raw video cannot silently reach a known text-only target.
• An empty Video model inherits the Vision model; empty Video and Vision settings use the credential-aware Vision Auto router.
• Cache/header/meta identity reports the actual successful fallback producer and retains it on cache hits.
• Vision and Audio operations without measured latency do not create fabricated zero-millisecond samples.
• Scene detection, deduplication, transcription/audio fusion, yt-dlp, persistent video-result caching, and interactive drill-down are intentionally outside feat(backend): Video bridge (frame sampling + captioning) - backlog da série Modality Bridge #9760.