feat: bridge audio inputs for text-only models

changelog.d/features/9807-audio-modality-bridge.md

@@ -0,0 +1 @@
+- **feat(modality bridge):** Transcribe chat audio for text-only models through the existing speech-to-text providers, with configurable limits, caching, runtime stats, and a dashboard self-test ([#9807](https://github.com/diegosouzapw/OmniRoute/pull/9807))

docs/security/GUARDRAILS.md

@@ -7,7 +7,7 @@ lastUpdated: 2026-08-08
 # Guardrails
 
 > **Source of truth:** `src/lib/guardrails/`
-> **Last updated:** 2026-08-08 — v3.8.50 (Modality Bridge PR-2: dashboard settings page, live stats, test action, and media-provider shortcuts)
+> **Last updated:** 2026-08-08 — v3.8.50 (Modality Bridge PR-3: Audio Bridge runtime and functional Audio settings tab)
 
 Guardrails enforce safety, policy, and content transformations at the boundary
 between OmniRoute and upstream providers. Each guardrail can inspect (and
@@ -20,12 +20,13 @@ request. Blocking is an explicit decision (`block: true`), never an accident.
 
 ## Built-in Guardrails
 
-The registry auto-loads four guardrails in priority order on import
+The registry auto-loads five guardrails in priority order on import
 (see `registry.ts` → `registerDefaultGuardrails()`):
 
 | Priority | Name                | Stage(s)       | File                  |
 | -------- | ------------------- | -------------- | --------------------- |
 | `5`      | `vision-bridge`     | `preCall`      | `visionBridge.ts`     |
+| `6`      | `audio-bridge`      | `preCall`      | `audioBridge.ts`      |
 | `10`     | `pii-masker`        | `pre` + `post` | `piiMasker.ts`        |
 | `20`     | `prompt-injection`  | `preCall`      | `promptInjection.ts`  |
 | `95`     | `credential-masker` | `pre` + `post` | `credentialMasker.ts` |
@@ -113,8 +114,8 @@ The new `modalityBridge*` keys are Zod-validated in `updateSettingsSchema`
 `modalityBridgeVisionMode`, `modalityBridgeVisionModel`,
 `modalityBridgeVisionTaskAware`, `modalityBridgeVisionPrompt`,
 `modalityBridgeVisionTimeout`, `modalityBridgeVisionMaxImages`, the
-`modalityBridgeCache*` trio, and the PR-3-reserved `modalityBridgeAudio*`
-group. Migration `141_modality_bridge_settings.sql` copies existing legacy
+`modalityBridgeCache*` trio, and the `modalityBridgeAudio*` group used by the
+Audio Bridge. Migration `141_modality_bridge_settings.sql` copies existing legacy
 `visionBridge*` values to the matching new keys (idempotent, never overwrites
 an operator-set `modalityBridge*` value); the legacy keys stay accepted as a
 read fallback for one release cycle.
@@ -130,20 +131,21 @@ swap is already visible in the response body's `model` field.
 
 `GET /api/modality-bridge/stats` (management auth, same tier as
 `GET /api/settings`) returns the in-memory per-modality counters
-`{ bridged, cacheHits, failures, lastUsedAt }` for `vision` (and the
-PR-3-reserved `audio`). Counters reset on process restart by design
+`{ bridged, cacheHits, failures, lastUsedAt }` for `vision` and `audio`.
+Counters reset on process restart by design
 (telemetry, not accounting).
 
 #### Dashboard configuration
 
 The dedicated dashboard page is
 `/dashboard/settings/modality-bridge`. Its URL-addressable `Vision`, `Audio`,
 and `Video` tabs preserve query parameters while switching the `tab` value.
-The Vision tab is live: it exposes enablement, mode, model selection (including
-the automatic default), task-aware prompting, advanced timeout/image/cache
-limits, runtime counters, and a guarded sample request. Audio and Video are
-explicit placeholders: Audio is reserved for PR-3, while Video remains tracked
-in issue `#9760`.
+The Vision tab exposes enablement, mode, model selection (including the automatic
+default), task-aware prompting, advanced timeout/image/cache limits, runtime
+counters, and a guarded sample request. The Audio tab is also live: it exposes
+enablement, an STT-only model picker with Auto, timeout/max-clip limits, audio
+counters, and an `input_audio` sample test. Video remains the explicit placeholder
+tracked in issue `#9760`.
 
 The former Vision Bridge card under AI settings is a compatibility link to the
 new page; it no longer owns a second copy of the form. Media Providers also
@@ -165,6 +167,64 @@ new mode/task-aware/cache defaults and the settings resolver live in
 `deps` constructor option so tests can inject fake `getSettings` and
 `callVisionModel` implementations.
 
+### Audio Bridge (`audioBridge.ts`) — Modality Bridge PR-3
+
+Intercepts audio-bearing chat requests before they reach a target that is not
+known to accept audio input. It never reroutes the chat request: audio parts are
+transcribed through the existing OpenAI-compatible multipart endpoint and the
+chosen chat model continues with text transcripts.
+
+Flow:
+
+1. Resolve `supportsAudio` through `getResolvedModelCapabilities()`. Explicit
+   provider-registry metadata wins, then static model metadata, then synced
+   `modalities_input`. A declared input list without `audio` is `false`; no
+   capability evidence remains `null`. Both `false` and `null` activate the
+   conservative bridge, while `true` bypasses it.
+2. Resolve `modalityBridgeAudio*` settings and extract spliceable top-level
+   audio parts from every message through the shared `detectMediaParts()`
+   detector. Supported wire shapes are OpenAI `input_audio`, `audio_url`, and
+   `source.media_type: "audio/*"`. Nested audio is detected for routing but not
+   removed by the splice path. Work is capped by `modalityBridgeAudioMaxClips`;
+   later parts stay untouched.
+3. Honor a configured `provider/model`, or let `selectAudioBridgeModel()` walk
+   `AUDIO_TRANSCRIPTION_PROVIDERS` in stable catalog order and select the first
+   model with a usable active provider credential.
+4. `callAudioTranscription()` converts base64/data-URI audio to a multipart
+   `file`, or downloads a remote `audio_url` through the public-only outbound
+   guard with DNS pinning and a 25 MB bound. It then POSTs the file and selected
+   model to the local `/v1/audio/transcriptions` self-loop, authenticated with
+   `resolveSelfLoopBearer()`. The existing transcription route performs normal
+   credential lookup, cooldown/rate-limit handling, and provider dispatch.
+5. Successful calls replace their parts with `[Audio N]: <transcript>`. Calls
+   run with `Promise.allSettled`: an individual failure preserves that original
+   audio part (#4012 contract). If every call fails and the target is proven
+   `supportsAudio === false`, the parts become
+   `[Audio N]: (unavailable — no STT provider connected)` (#8430 contract). For
+   an unknown target (`null`), an all-failure result stays untouched. A proven
+   text-only target with no usable STT credential receives the same explicit
+   stub without issuing a network call.
+
+Successful transcripts use the process-wide Modality Bridge LRU/TTL cache. The
+key combines the audio reference, the stable `audio-transcription` operation
+label, and selected STT model; failures are never cached. Audio attempts update
+the shared `bridged`, `cacheHits`, `failures`, and `lastUsedAt` counters.
+Transformed responses carry
+`x-omniroute-modality-bridge: audio->text;model=<sttModel>;parts=<n>`; untouched
+requests do not receive an Audio Bridge segment.
+
+Runtime settings are DB-backed and Zod-validated:
+
+| Key                           | Default | Range          |
+| ----------------------------- | ------- | -------------- |
+| `modalityBridgeAudioEnabled`  | `true`  | —              |
+| `modalityBridgeAudioModel`    | `""`    | Auto or STT ID |
+| `modalityBridgeAudioTimeout`  | `60000` | 1000–300000    |
+| `modalityBridgeAudioMaxClips` | `3`     | 1–10           |
+
+The shared cache remains controlled by `modalityBridgeCacheEnabled`,
+`modalityBridgeCacheTtlMinutes`, and `modalityBridgeCacheMaxEntries`.
+
 ### PII Masker (`piiMasker.ts`)
 
 Runs on **both** stages.
@@ -370,8 +430,8 @@ Environment variables read by the built-in guardrails:
 | `PII_REDACTION_ENABLED`               | `pii-masker`              | When `true`, request PII is redacted (independent of injection mode).                               |
 | `PII_RESPONSE_SANITIZATION` / `_MODE` | `pii-masker` (downstream) | Controls response-side masker behavior.                                                             |
 
-The Vision Bridge reads runtime config from the DB-backed settings store
-(`getSettings()`), not env vars. The primary keys are
+The Modality Bridge guardrails read runtime config from the DB-backed settings
+store (`getSettings()`), not env vars. Vision's primary keys are
 `modalityBridgeVisionEnabled`, `modalityBridgeVisionMode`,
 `modalityBridgeVisionModel`, `modalityBridgeVisionTaskAware`,
 `modalityBridgeVisionPrompt`, `modalityBridgeVisionTimeout`,
@@ -382,6 +442,11 @@ fallback; dashboard writes use the primary keys. Defaults and the fallback
 resolver live in `src/shared/constants/modalityBridgeDefaults.ts`, with legacy
 constants retained in `src/shared/constants/visionBridgeDefaults.ts`.
 
+Audio uses `modalityBridgeAudioEnabled`, `modalityBridgeAudioModel`,
+`modalityBridgeAudioTimeout`, and `modalityBridgeAudioMaxClips`, plus the shared
+`modalityBridgeCache*` settings. Audio has no legacy-key fallback because these
+keys were introduced with the Modality Bridge schema.
+
 ## Custom Guardrails
 
 ```typescript
@@ -419,9 +484,11 @@ Steps:
 
 Use `resetGuardrailsForTests()` between tests to start from a known state.
 Pass `{ registerDefaults: false }` to start with an empty registry and
-register only the guardrails under test. The Vision Bridge guardrail accepts
-dependency injection (`deps.getSettings`, `deps.callVisionModel`) so tests can
-exercise the full flow without DB or network access.
+register only the guardrails under test. Vision Bridge accepts dependency
+injection (`deps.getSettings`, `deps.callVisionModel`); Audio Bridge exposes the
+equivalent seams for settings, capabilities, STT model selection, credential
+checks, and transcription. Tests can therefore exercise both flows without DB
+or network access.
 
 ## See Also
 
@@ -430,6 +497,7 @@ exercise the full flow without DB or network access.
   prompt-injection and PII masking
 - `src/shared/constants/visionBridgeDefaults.ts` — Vision Bridge defaults and
   forced-bridge model list
+- `src/shared/constants/modalityBridgeDefaults.ts` — shared Vision/Audio runtime defaults
 - `docs/architecture/RESILIENCE_GUIDE.md` — orthogonal layer (circuit breaker, cooldowns)
 - `docs/reference/ENVIRONMENT.md` — full env var reference
 

open-sse/config/providers/shared.ts

@@ -50,6 +50,7 @@ export interface RegistryModel {
   supportsReasoning?: boolean;
   supportedThinkingEfforts?: readonly string[];
   supportsVision?: boolean;
+  supportsAudio?: boolean;
   supportsXHighEffort?: boolean;
   maxOutputTokens?: number;
   targetFormat?: string;

src/app/(dashboard)/dashboard/settings/components/modalityBridge/ModalityBridgeAudioTab.tsx

@@ -0,0 +1,197 @@
+"use client";
+
+import { useCallback, useEffect, useState } from "react";
+import { useTranslations } from "next-intl";
+
+import { Card, ModelSelectField, Toggle } from "@/shared/components";
+import type { ApiModel } from "@/shared/components/ModelSelectField";
+import {
+  MODALITY_BRIDGE_DEFAULTS,
+  resolveAudioBridgeRuntimeSettings,
+} from "@/shared/constants/modalityBridgeDefaults";
+
+import ModalityBridgeAudioTestButton from "./ModalityBridgeAudioTestButton";
+import ModalityBridgeStatsRow from "./ModalityBridgeStatsRow";
+
+interface AudioState {
+  modalityBridgeAudioEnabled: boolean;
+  modalityBridgeAudioModel: string;
+  modalityBridgeAudioTimeout: number;
+  modalityBridgeAudioMaxClips: number;
+}
+
+function fromApi(data: Record<string, unknown>): AudioState {
+  const runtime = resolveAudioBridgeRuntimeSettings(data);
+  return {
+    modalityBridgeAudioEnabled: runtime.enabled,
+    modalityBridgeAudioModel: runtime.model,
+    modalityBridgeAudioTimeout: runtime.timeoutMs,
+    modalityBridgeAudioMaxClips: runtime.maxClips,
+  };
+}
+
+function asSettingsRecord(value: unknown): Record<string, unknown> {
+  return value && typeof value === "object" ? (value as Record<string, unknown>) : {};
+}
+
+function clampNumber(raw: string, min: number, max: number, fallback: number): number {
+  const parsed = Number.parseInt(raw, 10);
+  return Math.min(max, Math.max(min, Number.isFinite(parsed) ? parsed : fallback));
+}
+
+export default function ModalityBridgeAudioTab() {
+  const t = useTranslations("settings");
+  const [settings, setSettings] = useState<AudioState | null>(null);
+  const isSttModel = useCallback(
+    (model: ApiModel) => model.type === "audio" && model.subtype === "transcription",
+    []
+  );
+
+  useEffect(() => {
+    let cancelled = false;
+    fetch("/api/settings")
+      .then((response) => (response.ok ? response.json() : null))
+      .then((data: unknown) => {
+        if (!cancelled) setSettings(fromApi(asSettingsRecord(data)));
+      })
+      .catch(() => {
+        if (!cancelled) setSettings(fromApi({}));
+      });
+    return () => {
+      cancelled = true;
+    };
+  }, []);
+
+  const update = async (patch: Partial<AudioState>) => {
+    try {
+      const response = await fetch("/api/settings", {
+        method: "PATCH",
+        headers: { "Content-Type": "application/json" },
+        body: JSON.stringify(patch),
+      });
+      if (response.ok) {
+        setSettings((previous) => (previous ? { ...previous, ...patch } : previous));
+      }
+    } catch (error) {
+      console.error("Failed to update Audio Bridge settings:", error);
+    }
+  };
+
+  if (!settings) return null;
+
+  const setLocal = (patch: Partial<AudioState>) => {
+    setSettings((previous) => (previous ? { ...previous, ...patch } : previous));
+  };
+  const commitNumber = (
+    key: "modalityBridgeAudioTimeout" | "modalityBridgeAudioMaxClips",
+    raw: string,
+    min: number,
+    max: number,
+    fallback: number
+  ) => {
+    const value = clampNumber(raw, min, max, fallback);
+    setLocal({ [key]: value });
+    void update({ [key]: value });
+  };
+
+  return (
+    <Card
+      title={t("modalityBridgeAudioTitle")}
+      subtitle={t("modalityBridgeAudioDesc")}
+      icon="graphic_eq"
+    >
+      <div className="space-y-4">
+        <Toggle
+          checked={settings.modalityBridgeAudioEnabled}
+          onChange={(checked) => void update({ modalityBridgeAudioEnabled: checked })}
+          label={t("modalityBridgeAudioEnabled")}
+          description={t("modalityBridgeAudioEnabledDesc")}
+        />
+
+        <ModelSelectField
+          label={t("modalityBridgeAudioModel")}
+          value={settings.modalityBridgeAudioModel}
+          placeholder={t("modalityBridgeAudioModelAuto")}
+          allowEmpty
+          modelFilter={isSttModel}
+          modelSource="catalog"
+          onChange={(value) => void update({ modalityBridgeAudioModel: value })}
+          className="text-sm"
+        />
+
+        <details className="rounded-control border border-border p-3">
+          <summary className="cursor-pointer text-sm font-medium">
+            {t("modalityBridgeAdvanced")}
+          </summary>
+          <div className="mt-4 grid grid-cols-1 gap-4 md:grid-cols-2">
+            <NumberField
+              testId="modality-bridge-audio-timeout"
+              label={t("modalityBridgeTimeoutMs")}
+              min={1000}
+              max={300000}
+              value={settings.modalityBridgeAudioTimeout}
+              onChange={(value) => setLocal({ modalityBridgeAudioTimeout: value })}
+              onBlur={(raw) =>
+                commitNumber(
+                  "modalityBridgeAudioTimeout",
+                  raw,
+                  1000,
+                  300000,
+                  MODALITY_BRIDGE_DEFAULTS.audioTimeoutMs
+                )
+              }
+            />
+            <NumberField
+              testId="modality-bridge-audio-max-clips"
+              label={t("modalityBridgeAudioMaxClips")}
+              min={1}
+              max={10}
+              value={settings.modalityBridgeAudioMaxClips}
+              onChange={(value) => setLocal({ modalityBridgeAudioMaxClips: value })}
+              onBlur={(raw) =>
+                commitNumber(
+                  "modalityBridgeAudioMaxClips",
+                  raw,
+                  1,
+                  10,
+                  MODALITY_BRIDGE_DEFAULTS.audioMaxClips
+                )
+              }
+            />
+          </div>
+        </details>
+
+        <ModalityBridgeStatsRow kind="audio" />
+        <ModalityBridgeAudioTestButton />
+      </div>
+    </Card>
+  );
+}
+
+interface NumberFieldProps {
+  testId: string;
+  label: string;
+  min: number;
+  max: number;
+  value: number;
+  onChange: (value: number) => void;
+  onBlur: (raw: string) => void;
+}
+
+function NumberField({ testId, label, min, max, value, onChange, onBlur }: NumberFieldProps) {
+  return (
+    <label className="block text-sm font-medium">
+      {label}
+      <input
+        type="number"
+        data-testid={testId}
+        min={min}
+        max={max}
+        value={value}
+        onChange={(event) => onChange(Number.parseInt(event.currentTarget.value, 10) || 0)}
+        onBlur={(event) => onBlur(event.currentTarget.value)}
+        className="mt-1 w-full rounded-control border border-border bg-surface px-3 py-2 text-sm"
+      />
+    </label>
+  );
+}