[arun-elr]

Fixes #592

What does this PR do?

Worker, engine and decoder exceptions reached API clients verbatim. That is a security risk (#592): any client, with no credentials (#506), sending a bad upload or hitting a server fault could read server paths, the server's user name and arena segment names with pids, and one engine error could show a client text about another request in its batch. Now deliberate refusals keep their message with paths cut to base names; anything else becomes an error id, and the server log keeps the full error under it.

• Helper. client_error_text(exc, what) -> (text, error_id) and scrub_paths in mstar/utils/client_errors.py: one place for client error text; /v1/realtime: language and prompt skip validation, and a failed step echoes them whole into the error event and the log #499 and Harden remote media fetch when MSTAR_ALLOW_REMOTE=1 (follow-up to #411) #424 can reuse it.
• Data worker (data_worker.py): decoder errors named the upload; status stays 400/500.
• Worker and engine (worker.py, engine.py): worker ingest and main loop, and engine per-request stages; Error in worker: carried arena and tensor-file paths. A failed tensor read ([Mooncake][Worker] Fail only the request whose async tensor read failed, at TP>1 too #419) gets an error id on both the worker and the data worker side.
• Conductor (conductor.py): ingest failures, the same leak one hop earlier (ValueError stays a 400), and a request failed after a conductor step raised.
• Frontends. The /generate and /generate/ws catch-alls in entrypoint.py and the six OpenAI route catch-alls in openai/router.py echoed str(e); a transcription decoded in the frontend (a header past the model's window) could send the decoder's message with the upload path. Deliberate messages are unchanged. The Rust frontend bridge in rust_frontend.py sent repr(e); the Rust code itself only relays that text, plus its own refusals and I/O errors, which name no path.

How was it tested?

On 016d546 (head, rebased on main 6430016, umask 002): ruff check . is clean. CPU Core selection: 6,941 passed, 879 skipped, 4 failed; the same 4 fail on main 6430016 here (6,903 passed), because this venv lacks prometheus_client and LingBot's ftfy. TP lockstep sim: 11 passed. The rebase kept #419's one-line log for a failed tensor read (client_error_text takes traceback=False for it) and sends an error id for that read on the worker side too; the GPU runs below are from before the rebase. python -m pytest test/modular/test_client_error_text.py (new): 37 passed (31 fail on main; 14 fail on the branch before the OpenAI route change). The changed test files (test_conductor_tick_exception.py, test_openai_router.py, test_request_failure_propagation.py, test_preplan_error_paths.py, test_teardown_barrier.py, test_ingest_rollback.py, test_kv_layout_refusals.py): 131 passed, 3 skipped. python -m pytest test/rust/ against a release mstar-server built from main (this PR changes no Rust): 463 passed, 5 skipped, 1 failed (test_runtime_lockstep.py::test_only_the_tp_leader_counts_new_tokens[1], which fails on main here too).

Each OpenAI route gets a handler error carrying an upload path and a fake API key: the client sees only an error id, the log has both. Hostile transcriptions (random bytes, a truncated header, headers claiming 4 h and 600 s over a 1 s body, a 40 s FLAC with noise in the middle) get the engine's answer or, for the FLAC that fails in the frontend decode, a 500 with an error id; no path, no upload left, RSS within 6 MiB.

e2e matrix on one H200, main against this PR (e2e-matrix run --base main --head 08d6077dd --models whisper_large --rp --parallel 1): no regressions, 28/28 request-path cases the same.

e2e matrix table

Model	Modality	Starts	Correct	c=1 latency, base to head	c=8 throughput, base to head	Verdict
whisper_large	asr (multipart upload)	yes	yes	128 ms to 127 ms (+0.5%)	14.6 req/s to 14.8 req/s (+1.4%)	same
whisper_large	asr (SSE stream)	yes	yes	69 ms to 68 ms (+1.3%)	15.9 req/s to 16.5 req/s (+4.0%)	same
whisper_large	request path (26 cases)	yes	yes (28/28 same)			same

e2e matrix details and chart

No regressions: 3 rows across 1 models, every one starts, keeps its outputs, and stays within 5% (or within noise) at c=1 and c=8.

Served through mstar serve on one H200 per model, base and head back to back on the same VM (order alternated per model), 3 measured passes per point, 1 server session(s) per arm. Deltas are signed so that positive means head is better (lower latency, higher throughput).

Measured on 08d6077dd against main 294cbab; the commits since add #807, which both arms then have, and a test-only change. The request-path cases include garbage audio, an empty file, a 513 MiB upload and malformed multipart.

Per-row numbers (mean +- std over passes), correctness detail, extra metrics

Model / case	Correctness	c=1 latency	c=8 throughput	Extra (c=1)
whisper_large / upload	WER 1.53% to 1.53%, 100/100 identical transcripts	128 ms +- 1.1% to 127 ms +- 0.4% (+0.5%)	14.6 req/s +- 10.0% to 14.8 req/s +- 5.6% (+1.4%)	rtfx 49.8 to 49.7 (-0.1%)
whisper_large / sse	WER 1.53% to 1.53%, 100/100 identical transcripts	69 ms +- 1.1% to 68 ms +- 0.8% (+1.3%)	15.9 req/s +- 0.8% to 16.5 req/s +- 1.9% (+4.0%)	e2e_s_p50 128 ms to 127 ms (+0.7%)

Startup, peak VRAM, errors

Model	Startup s, base / head	Peak VRAM GB, base / head	Request errors, base / head
whisper_large	292 / 57	27 / 27	0 / 0

Startup uses warm compile caches shared by both arms, so the first arm can pay compiles the second reuses; it is reported, not judged.

Setup

• base: main (82375ad11), head: 08d6077ddad9dc95d2fa74c9e18300301620f299 (08d6077dd)
• GPU: H200, NVIDIA H200; one model per VM at a time, 1 VM (e2e-h200-4)
• Dependencies: one venv per lock (pyproject plus the serving Dockerfile's torch pins), shared by both arms; torch 2.9.1+cu128
• Checks: greedy text identity at c=1, Whisper-large-v3 WER for speech, PSNR against base for seeded images and video, max action difference for VLA, status codes for the request path
• Regression: worse than 5%, more than 2 combined standard deviations of the passes, and more than 1.5 times the largest drift the same row showed between two processes of identical code (A/A calibration runs); better by the same margin is reported as faster. Where base and head returned different output lengths, c=1 latency uses only the requests whose lengths matched (else latency per token or audio second) and throughput counts output tokens or audio seconds per second
• Correctness limits: the manifest's, widened to 1.5 times the largest swing the row showed between two processes of identical code (A/A runs); never tighter than the manifest; 1 session(s) per arm, a perf row flagged with one session is rerun with two (ABBA) before it is reported

Not tested

The Rust frontend served on a GPU: the matrix's only Rust frontend model is Qwen3-Omni (qwen3_omni_rustfe), which is large and has no request-path cases. Its error text comes from the Python bridge, which test_the_rust_frontend_bridge_answers_with_an_error_id covers.

Not here: remote-fetch errors (#424) and the /v1/realtime server_error echo (#499) will use client_error_text.

Depends on and merge order

Depends on: none. git merge-tree against #618, #617, #645 and #807 is clean for each pair, and the four still open merged onto main 82375ad (#807 has since merged) pass the CPU Core selection (6,854 passed, 845 skipped, the same 4 environment failures), test/rust/ (464 passed, 5 skipped) and cargo test --release in rust/server (75 passed).

With #617: its upload-dir refusal (PermissionError, which names the directory) reaches clients through these catch-alls as an error id on the Python side; the Rust frontend still sends that text in its 500.

Design

• Deliberate means ValueError or TypeError, the rule the data worker already used to pick 400 over 500, so [Model] Cap the decoded size of request images and videos #510's media refusals and model validation messages read as before; only absolute paths change, to base names (a directory path becomes <path>).
• The OpenAI routes keep an HTTPException's status and detail as before. Anything else goes through the helper: ValueError/TypeError is a 400, an exception class that names its own status_code keeps it (AudioTooLongError stays a 413), and the rest is a 500 with an error id. Speech used to answer a stray ValueError with a 500; it is now a 400 like the other routes.
• /generate, /generate/ws, the OpenAI routes and the bridge pass type_name=False, so their kept messages stay byte-identical to main. The data worker, worker and conductor keep their <stage> failed: <Type>: <message> shape.
• The helper logs <what> [error id <id>]: <Type>: <message> with the traceback, so the engine's three per-stage logger.exception calls moved into register_failure rather than logging twice.
• /generate keeps status 500 for a ValueError from submit_request, as on main. Scheduler admit refusals carry the resource's own message; the one that held an exception's text, a failed KV retrieve (its message named the snapshot path), now carries the error id instead (kv/manager.py).

Tests in detail

• test_client_error_text.py: a real PIL decoder failure on an upload under a per-user dir, a deliberate decoder ValueError naming that upload, an arena I/O error, a worker main-loop exception, a per-request stage failure, /generate, /generate/ws and the Rust bridge each answer with no /tmp, /dev/shm, mstar_arena or user name, and the server log holds the full error (under the client's error id for the generic ones). Plus a guard that /generate keeps a deliberate message byte-identical, and 8 scrub_paths cases (URLs, ratios and relative paths untouched; a quoted path is cut whole, so directories with spaces or brackets do not leak). A failed KV retrieve reaches the admit refusal as an error id, with the path only in the log.
• The OpenAI routes: for each of the six, a handler raising a RuntimeError that carries an upload path and a fake API key answers 500 with only an error id (the log has both), and a ValueError naming the upload answers 400 with the base name. A transcription with a 40 s corrupt FLAC reaches the frontend decode (the test counts the call), whose fallback raises with the saved upload's path: the client gets an error id and the upload is deleted. Five hostile uploads get a clean answer with bounded RSS, and an AudioTooLongError keeps its 413.
• test_conductor_tick_exception.py: the two cases that pinned Conductor error ...: RuntimeError: <text> now expect the error id, and one of them finds the text in the log under that id.
• Changed files: 6 assertions that pinned the raw exception text in the client message now expect the error id, one new conductor case checks a deliberate ingest ValueError keeps its 400 without the path, and the speech test's OSError case expects an error id instead of the OSError text.
• Fail-on-main count: commit, git checkout upstream/main -- <the source files>, remove the new helper, run, restore. Of the 37 cases in test_client_error_text.py, the 14 route and frontend-decode cases fail on the previous head (with the other leaks fixed), and the 4 other hostile-upload cases pass there too: they guard the paths that already answered cleanly.

Checklist

• ruff check . passes
• Added or updated tests / docs where relevant

AI assistance: written with AI coding agent

[github-actions]

Automated review

The PR adds mstar/utils/client_errors.py with client_error_text, which logs the full error and traceback under a random error id. For ValueError/TypeError it returns the message with absolute paths cut to their base names; for any other exception it returns internal error (error id <id>). The data worker, conductor ingest, worker ingest and main loop, the engine's per-request register_failure, the /generate and /generate/ws catch-alls and the Rust frontend bridge now build their client error text with it. The code matches the description.

Findings: none.

I checked these paths:

• Engine: ExecutingBatch.graph_walk exists as a property. The text stored in failed_requests reaches only error messages; control flow reads only the keys. So different error ids on different TP ranks don't change any control flow.
• Engine logging: the three per-stage logger.exception calls that were removed are replaced by the log line inside the helper, so these errors are still logged once each.
• /generate: request_id is bound when the catch-all runs.
• Test count: the "15 passed" for the new test file matches 9 tests plus 6 parametrized cases.

The description gives test commands and results, and says what was not tested and why. Nothing is missing for a change of this size.

---

_{Advisory, not a merge gate. Rules: AGENTS.md. Reply to discuss.}

[merceod]

Checked 8c60efa served on klone and read 0e11f83 (the rebase over #362 gives register_failure a walk argument used in the log line, adds the KV retrieve error text and quoted paths to the scrub, and adjusts the cosmos3 test stub; the served paths are unchanged, not re-served). Injected worker and decoder errors come back as internal error (error id ...) on all five surfaces (SSE, NDJSON, websocket, non-streamed, bridge), ValueError paths are cut to base names, and all 21 ids a client received are in the server log. The ordinary 400s (bad JSON, missing model, bad sample rate) keep their text.

One thing I checked because it looked wrong on the first head: /generate/ws and the bridge turned the server's own HTTPExceptions ("Request timed out", a fatal 503) into the generic text. The current heads keep #519's HTTPException branch first and the timeout frame is {"error":"Request timed out","status":500}, the same as main. Fine as is.

Outside this PR: chat finish_reason is hard-coded to "stop", so a max_tokens cut reports "stop", and a streamed /generate cut by --timeout ends without an error line, on both arms.

[NSagan271]

🤖 Automated live test, AI-generated (Claude Code). Not checked line by line by a human. Numbers come from one allocation, so treat small deltas as provisional.

Live test: [API server][Security] Keep worker and decoder error text out of client responses

Works: base sends the client the upload path and server user name for audio decode errors; the PR sends an error id and logs the full text under it.

1x H200 per server, Qwen3.5 0.8B and Whisper large-v3, Python and Rust frontends, SHM transport · PR head 88d3410e0 vs main 56018ce1a · head now 0e11f83ec: adds "keep KV retrieve error text and quoted paths out" and reworks the /generate/ws and bridge error replies; retested (all served rows on both frontends, 3/3: same results). PR tests pass (62)

Claims

Claim	Result
Decoder errors no longer show server paths or the user name	✅ Whisper, garbage WAV and truncated M4A, /generate and /v1/audio/transcriptions. Base: 500 "Could not open input file: <upload path under the user's home>/…". PR: 500 "internal error (error id …)" (3/3)
The full error is logged under the id	✅ the server log has [error id a3e011b4e7a8]: RuntimeError: Could not open input file: …
Deliberate ValueError/TypeError messages are unchanged	✅ bad max_output_tokens, temperature, seed and top_k give the same 400 text on both arms and both frontends (3/3)
/generate/ws, the Rust bridge and the catch-alls	✅ ws: error id. Rust /generate: error id; max_output_tokens 1e30 gives 400 on both Rust arms; on Python it is 500 "OverflowError…" on base and an error id on the PR

Other

• Trade-off for reviewers: errors that never contained a path are now generic too. A corrupt PNG or GIF on Qwen3.5 was a descriptive 500 on base ("Out of bound read in decode_png…", "Unsupported image file…") and is now "internal error (error id …)". Clients lose that hint unless the loader turns it into a ValueError ([Model][Bug fix] Decode request images as RGB, so RGBA, CMYK and grayscale images stop failing #633 does this for images).
• Not tested: cross-request text in a batched engine error; worker or arena failure paths. The Qwen3.5 CPU-overhead row was skipped because the change is on failure paths only.

[arun-elr]

Thanks for the served check.

/generate/ws and the bridge turned the server's own HTTPExceptions ("Request timed out", a fatal 503) into the generic text. The current heads keep #519's HTTPException branch first

Yes, that branch stays first on purpose: an HTTPException is the server's own message with its own status, so it goes out unchanged, and only other exceptions get the error id.

Outside this PR: chat finish_reason is hard-coded to "stop", so a max_tokens cut reports "stop", and a streamed /generate cut by --timeout ends without an error line, on both arms.

Confirmed on main and filed as #709. In the code, finish_reason is hard-coded on both frontends. The missing timeout line is on the Python frontend only: _iter_result_chunk_batches raises after the NDJSON or binary body has started, and nothing catches it there. Chat SSE and the Rust /generate stream already send the timeout in-band.

No code change since the approval.

[arun-elr]

Thanks for the run. The current head is the retested one, so the rows above still hold.

Trade-off for reviewers: errors that never contained a path are now generic too. A corrupt PNG or GIF on Qwen3.5 was a descriptive 500 on base ... Clients lose that hint unless the loader turns it into a ValueError (#633 does this for images).

• Intended, and covered by [Model][Bug fix] Decode request images as RGB, so RGBA, CMYK and grayscale images stop failing #633 for the base loader. [Model][Bug fix] Decode request images as RGB, so RGBA, CMYK and grayscale images stop failing #633 is on main now and turns decode failures in Model.load_image into a ValueError, which this PR passes through as a 400 with its text. Checked on this branch rebased onto main, on CPU: a truncated PNG, a PNG with a broken IDAT and a truncated GIF give 400 "could not decode image: the file is corrupt or truncated"; a garbage GIF and a non-image give 400 "the image is not in a format this server can read". The PR head itself predates [Model][Bug fix] Decode request images as RGB, so RGBA, CMYK and grayscale images stop failing #633, so the rebase brings this in.
• Real, outside this PR: Flux2 Klein and Waypoint override load_image without that wrapper, so a corrupt image there is still a 500 with an error id. Filed as [Model][Bug fix] Flux2 Klein and Waypoint answer 500 for a corrupt image: their load_image skips the ValueError wrapper from #633 #720.

Not tested: cross-request text in a batched engine error; worker or arena failure paths.

• Cross-request text in a batched engine error: unchanged from main. A batch-wide failure goes through client_error_text like any other, so anything but a ValueError/TypeError reaches every row as an error id, and no other request's text gets through. A deliberate ValueError raised for the whole batch keeps its message for every row, as on main; the PR adds no new text to it.
• Worker and arena failure paths: covered on CPU. test_a_worker_exception_reaches_the_client_as_an_error_id, test_a_per_request_stage_failure_reaches_the_client_as_an_error_id and test_an_arena_io_error_reaches_the_client_as_an_error_id in test/modular/test_client_error_text.py inject those failures and check that the client gets an error id while the log has the full text. Not served.

No code change since the approval.

[arun-elr]

@Gaurav-Shah05 could you take a look when you get a chance, and merge if it looks good? @merceod approved it and CI's green. it keeps worker and decoder error text out of client responses.

[arun-elr]

@merceod heads up, since your approval the OpenAI route catch-alls and the conductor's step-failure text go through the same error-id scrubbing too, and a stray ValueError on speech is now a 400 like the other routes. mind another look?

[merceod]

Approve again on 141060f. Since my last look the error-id policy reached the OpenAI routes and the conductor. All six catch-alls go through _exception_error (router.py lines 66 to 87), an HTTPException keeps its status and detail, a ValueError or TypeError is a 400 with paths cut to base names, anything else a 500 with an error id and the full error in the log, a class with its own status_code keeps it, and _fail_after_error sends the error-id text too. The new cases in test_client_error_text.py, test_conductor_tick_exception.py and the speech-status case fail on the head I approved and pass here, 194 over the ten touched files.

Two behaviour changes worth a line in the description. A ValueError or TypeError on /v1/audio/speech was a 500 and is a 400 now, and other 500s on the OpenAI routes show only an error id. Held for a second approval as before, and it conflicts with main in conductor.py and worker.py, so a rebase is due.

[arun-elr]

@Gaurav-Shah05 rebased on main, CI's green and merceod's approval stands. would you mind merging when you get a chance?

@merceod FYI one commit on top since your approval: #419's worker input-read failure sent the raw error text to the client, so it now goes through the same error id, w/ #419's test asserting it.

[merceod]

Read 016d546. The worker's input-read failure now goes through client_error_text with the traceback off, since the tensor manager logged the read where it failed, which is this PR's own rule applied to #419's new text. Approval stands at this head.

[Gaurav-Shah05]

The log change can come in a follow-up if you'd rather merge this first.

[Gaurav-Shah05]

client_error_text logs every exception at ERROR with its traceback, so every 400 for a bad request, such as a chat request that lists tools, now writes a traceback to the log where main wrote nothing. The 400's text doesn't carry the error id, so I think _exception_error should log only the 500s.