Skip to content

Add OpenAI-compatible TTS provider - #14

Draft
matthewhand wants to merge 582 commits into
mainfrom
feat/openai-compatible-tts
Draft

Add OpenAI-compatible TTS provider#14
matthewhand wants to merge 582 commits into
mainfrom
feat/openai-compatible-tts

Conversation

@matthewhand

@matthewhand matthewhand commented Aug 18, 2026

Copy link
Copy Markdown
Owner

OpenAI-compatible TTS provider

Draft against matthewhand/OpenMausBot main.

ElevenLabs or an OpenAI-compatible /v1/audio/speech server (Kokoro, LiteLLM, OpenAI).

Finished

  • Per-provider keys, voice ids, and model (openaiKey / openaiVoice / openaiModel).
  • Per-bot openaiVoice is stored separately from bot.voice. Speak / Call / auto-speak / Settings use botVoiceId(provider, bot) so a leftover ElevenLabs id is never sent to Kokoro.
  • Verify and speak send the saved model + response_format: mp3.
  • Voice list: /audio/voices then /voices. OpenAI host fallback is alloy/echo/…, not Kokoro af_heart.
  • Custom voice id field. openaiKeyConfigured + Clear.
  • README is not ElevenLabs-only.

Tests

pnpm exec vitest run server/tts/tts.test.ts server/tts/speech-text.test.ts server/config.test.ts src/lib/tts-provider.test.ts

@amazon-q-developer amazon-q-developer Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

This PR successfully adds OpenAI-compatible TTS provider support alongside ElevenLabs. The routing logic is well-structured with proper backward compatibility.

Critical Issues Found

Security (5 findings)

  • Unvalidated baseUrl parameter creates SSRF vulnerability in three functions (verifyKey, listVoices, synthesize). URLs must be validated before being used in fetch calls to prevent potential exploitation.

Logic Errors (2 findings)

  • Voice ID handling in listVoices can produce invalid empty voice IDs
  • Voice filtering doesn't properly handle whitespace-only IDs

All identified issues have blocking severity and must be fixed before merge.


You can now have the agent implement changes and create commits directly on your pull request's source branch. Simply comment with /q followed by your request in natural language to ask the agent to make changes.

Comment thread server/tts/openai-compatible.ts Outdated
if (!Array.isArray(list) || list.length === 0) return fallbackVoices();
return list
.map((v: any): Voice => ({
id: String(v.voice_id ?? v.id ?? ""),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 Logic Error: Empty voice IDs can pass through the mapping. When all properties (v.voice_id, v.id) are missing or empty, line 76 creates id: String(""), which is an empty string. Line 80's filter checks v.id for truthiness, but empty strings are falsy, so they would be filtered out. However, if any property exists but is an empty string, it could pass through, creating invalid voices.

Suggested change
id: String(v.voice_id ?? v.id ?? ""),
id: String(v.voice_id ?? v.id ?? "").trim(),

Comment thread server/tts/openai-compatible.ts Outdated
label: String(v.name ?? v.label ?? v.id ?? "Voice"),
description: v.description || v.labels?.description || undefined,
}))
.filter((v: Voice) => v.id);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 Logic Error: The filter checks v.id but should explicitly check for non-empty strings. Currently, whitespace-only IDs would pass through since .trim() is not applied before filtering, potentially causing issues when voices are used.

Suggested change
.filter((v: Voice) => v.id);
.filter((v: Voice) => v.id.trim().length > 0);

Comment thread server/tts/openai-compatible.ts Outdated

export async function verifyKey(baseUrl: string, key?: string): Promise<VerifyResult> {
try {
const url = `${baseUrl.replace(/\/$/, "")}/audio/speech`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 Security Vulnerability: Unvalidated user input (baseUrl) is used directly in URL construction without sanitization. Malicious URLs could potentially exploit the fetch API or trigger SSRF attacks. Validate the URL format before use.1

Suggested change
const url = `${baseUrl.replace(/\/$/, "")}/audio/speech`;
const parsedUrl = new URL(baseUrl);
if (!['http:', 'https:'].includes(parsedUrl.protocol)) {
throw new Error('Invalid URL protocol');
}
const url = `${parsedUrl.origin}${parsedUrl.pathname.replace(/\/$/, "")}/audio/speech`;

Footnotes

  1. CWE-918: Server-Side Request Forgery (SSRF) - https://cwe.mitre.org/data/definitions/918.html

Comment thread server/tts/openai-compatible.ts Outdated

export async function listVoices(baseUrl: string, key?: string): Promise<Voice[]> {
try {
const url = `${baseUrl.replace(/\/$/, "")}/voices`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 Security Vulnerability: Unvalidated baseUrl used directly in URL construction. This pattern is repeated across multiple functions and creates SSRF risk. Apply the same URL validation as suggested for the verifyKey function.1

Suggested change
const url = `${baseUrl.replace(/\/$/, "")}/voices`;
const parsedUrl = new URL(baseUrl);
if (!['http:', 'https:'].includes(parsedUrl.protocol)) {
throw new Error('Invalid URL protocol');
}
const url = `${parsedUrl.origin}${parsedUrl.pathname.replace(/\/$/, "")}/voices`;

Footnotes

  1. CWE-918: Server-Side Request Forgery (SSRF) - https://cwe.mitre.org/data/definitions/918.html

Comment thread server/tts/openai-compatible.ts Outdated
baseUrl: string,
key?: string,
): Promise<Audio> {
const url = `${baseUrl.replace(/\/$/, "")}/audio/speech`;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛑 Security Vulnerability: Unvalidated baseUrl creates SSRF risk in the synthesize endpoint. Apply URL validation as suggested for other functions.1

Suggested change
const url = `${baseUrl.replace(/\/$/, "")}/audio/speech`;
const parsedUrl = new URL(baseUrl);
if (!['http:', 'https:'].includes(parsedUrl.protocol)) {
throw new Error('Invalid URL protocol');
}
const url = `${parsedUrl.origin}${parsedUrl.pathname.replace(/\/$/, "")}/audio/speech`;

Footnotes

  1. CWE-918: Server-Side Request Forgery (SSRF) - https://cwe.mitre.org/data/definitions/918.html

@matthewhand
matthewhand force-pushed the feat/openai-compatible-tts branch 2 times, most recently from 0147bd6 to e2754d9 Compare August 23, 2026 22:20
milind-soni and others added 27 commits August 25, 2026 13:12
…ew-live-desktop

feat(computer): open live desktop from preview
…y-foundation

fix(ios): pair through every trusted route
…nion-endpoints

feat(companion): carry secure endpoints from desktop to iOS
…ne-auth

feat(auth): add Better Auth D1 control plane
polar: supamaus in FUNDING.yml puts the Sponsor button on the repo,
with the direct checkout link (one-time pay-what-you-want or monthly)
as the custom entry. The README section says the quiet part out loud:
nothing ever sits behind a paywall.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add Polar funding: Sponsor button + README support section
rahul-vanyar and others added 3 commits August 28, 2026 16:17
…sion eats composition keywords (milind-soni#547)

Codified from the milind-soni#544 field failure: schedule was a oneOf of const-branches,
several engines' MCP-to-provider converters flattened it, and models guessed
shapes forever. The rule, the coercion posture, and the errors-must-teach
posture now live next to the driver SPI guidance so the next tool surface
doesn't relearn it in production.

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
milind-soni and others added 25 commits August 29, 2026 03:28
* fix(stream): make chat delivery resilient

* fix(stream): supervise live event delivery

* fix(stream): pin task and buffer inspector refresh

* fix(stream): keep raced sends on their task

* fix(stream): acknowledge replacement snapshots

* fix(stream): reject stale steer acknowledgements

* fix(stream): decouple peripheral snapshot failures

* test: cover send idempotency routes

* fix(stream): recover retried chat sends

* fix(stream): close final client races
…i#553)

Restore the two-option list. The chip only changes its name (Ask for
approval / Auto mode), not its color. Picking Auto mode still sets
autoApprove, so the profile switch turns on with it.

Co-authored-by: Max <ajsdaksfjhs@gmail.com>
…i#551)

* fix(chat): rest the last bubble one gap-3 above the composer

pb-24 was a guessed overlay slot, so scrolling to the end left more
black above the pill than sits between two bubbles. Pad the transcript
by the measured composer height plus gap-3 (0.75rem) so the rest
position matches the stack.

* fix(chat): stop snapping the transcript when near the end

Re-pinning follow inside a 48px zone called scrollTo and yanked the
viewport. Only arm follow at the rest position, and do not scroll just
because follow flipped true. New rows still stick; Jump to latest is
the explicit trip to the bottom.

* docs(chat): capture rest-at-bottom without the magnet snap

Reference for the gap-3 composer rest and the dropped 48px end yank.

* docs: drop PR-only scroll demos

---------

Co-authored-by: Max <ajsdaksfjhs@gmail.com>
Co-authored-by: milind-soni <milindsoni201@gmail.com>
* feat(tasks): search the header task switcher

The picker was a 320px scroll of every context on the bot, so a long
task list had no way to jump to a name. Add a search field that
filters titles as you type, ranks prefix hits first, and lets Enter
take the top match.

* fix(tasks): use valid picker accessibility semantics

---------

Co-authored-by: Max <ajsdaksfjhs@gmail.com>
Co-authored-by: milind-soni <milindsoni201@gmail.com>
…el (milind-soni#548)

* feat(openai-compat): pin OpenRouter upstream provider and default model

The openai-compat driver sent only `{model, messages, stream}`, so there was
no way to pin an OpenRouter upstream provider or seed a default model. Add two
optional `openaiCompat` config fields:

- `model`   — seeds the picker's default selection (survives /models refresh)
- `provider`— OpenRouter routing; sent as
  `provider: { order: [provider], allow_fallbacks: false }`

Both thread through the existing workspace-default carry in instanceConfigs
(parallel to `url`) and fall back to `OPENAI_COMPAT_MODEL` /
`OPENAI_COMPAT_PROVIDER`. Endpoints that don't speak OpenRouter routing ignore
the extra field. Adds unit tests for decode, catalog seeding, and body shape.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix: gate OpenRouter provider routing by host; sync model/provider env on save

- openai-compat driver: only serialize the OpenRouter-specific `provider`
  routing object when the configured URL's hostname is openrouter.ai or a
  subdomain (parsed via URL, not substring-matched) — strict OpenAI-compatible
  endpoints like Groq reject unknown top-level fields.
- config: syncCredentialEnv now keeps OPENAI_COMPAT_MODEL and
  OPENAI_COMPAT_PROVIDER in step with a mid-session save, matching the
  existing key/url behavior (set when truthy, delete when cleared, untouched
  when absent) so boot-injected env no longer shadows a save until relaunch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: milind-soni <milindsoni201@gmail.com>
…turns while things boot

Stolen from vercel-labs/fx's terminal monitors. A bot that starts a dev
server or kicks off a long job currently polls with repeated computer_exec
or screenshot calls — one model inference per peek. wait_for runs the whole
wait in ONE round trip: a 2s poll loop on the box (http_ready / tcp_ready /
output_matches / file_exists, bounded 1-240s), then the settled screen rides
back in the same result, matching the file's act-and-observe contract.

The schema is flat (enum + per-condition fields described in words) and bad
input answers with a copyable example instead of a wall — both per the
CONTRIBUTING "MCP tool schemas" rules from milind-soni#544. A timeout returns advice
(inspect with computer_exec) rather than an invitation to wait again, and
the preview poker learns the tool name so the panel refreshes.

Verified: 22/22 proxy contract tests (schema flatness guard, free-of-charge
guidance on bad input, single-round-trip with frame, timeout advice, bash -n
on every generated shell); mutation check (breaking the loop's sentinel
fails 3 tests); tsc clean; lint parity with main (14 = 14 findings).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…t-for

feat(computer): wait_for — one box-side poll loop instead of a model inference per peek
…ceipts, and read-back tools

Bots could hand work to a peer but never learn what happened: delegate_bot
was fire-and-forget by contract, a busy target CANCELED the handoff outright,
and ask_bot's synchronous wait was the only way to get an answer back.
Modeled on fx's durable subagent message ledger, adapted to the existing
queue in delegations.ts:

- queueDelegation returns a task id; the delegating bot receives it in the
  tool reply as its claim ticket.
- A busy target no longer cancels the handoff. The item stays queued with a
  bounded retry count (3); any of the target's turns settling re-drains the
  waiting source threads. Fixed en route: a drain request arriving while a
  drain was already running was silently dropped by the drainingThreads
  lock — precisely where the waiting-on retry lands — so requests are now
  remembered and honored when the running drain finishes.
- Every terminal outcome writes a durable receipt (done/failed/denied/
  busy_gave_up/dropped/error + the peer's bounded reply), persisted to
  delegation-receipts.json, pruned by count (100) and age (48h), loaded at
  boot, and written before any mirror short-circuits.
- New MCP tools: check_delegation (status now) and wait_delegation (bounded
  long-poll up to 240s — ONE parked HTTP request instead of a model
  inference per status peek), over GET /api/internal/delegations/:id with
  sender/thread ownership enforced. Flat schemas, guiding errors.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…er panel, driven over CDP

Squash of feat/browser-surface with origin/main merged (routine-type +
browser imports unioned in server/index.ts; browser-surface startup block
kept in electron/main.mjs) and the three exact-match features assertions
taught the new browser flag. Linear history because the original branch
stopped receiving pull_request workflow runs entirely.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…edger

feat(comms): delegation ledger — task ids, busy retries, durable receipts, read-back tools
The branch's hide-until-skin-handshake (waitsForSkinSync + show:!...) lost
its definition when main's windowChromeOptions refactor auto-merged over
the BrowserWindow options; the kept fallback block then referenced an
undefined identifier and createWindow threw at startup — caught by the
Linux packaged-app smoke.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ace-v3

feat(browser): a built-in browser per bot — Browser tab in the computer panel, driven over CDP
…soni#573)

* fix(browser): harden built-in browser isolation and takeover

* test(browser): make hardening checks cross-platform

* test(browser): isolate Windows LPAC runner setup

* test(browser): keep macOS fixture sandboxed

* docs(browser): clarify headless Electron fixture

* test(browser): preserve renderer sandbox on Windows

* docs(browser): describe Windows fixture scope precisely

* fix(browser): close review hardening gaps

* test(browser): preserve Windows Electron sandbox

* test(browser): mirror Chromium Windows sandbox setup

* test(browser): avoid Windows runner GPU regression

* test(browser): avoid GPU process in headless Windows fixture

* test(browser): isolate Windows runner regression

* test(browser): use Windows desktop sandbox runner

* fix(browser): close remaining isolation gaps

* test(browser): strengthen nested secret regression

* fix(browser): close final isolation gaps

* test(browser): handle Windows child termination

* test(browser): allocate isolated server ports safely
@matthewhand
matthewhand force-pushed the feat/openai-compatible-tts branch from 890824d to d952355 Compare August 30, 2026 06:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.