perf(engine): avoid timed waits for Engine responses

[jthomson04]

Summary

Engine calls pass no HTTP request object, but each response wait still uses asyncio.wait_for. Its timeout path checks for an HTTP disconnect and otherwise starts another wait.

Await state.event.wait() directly when request is None. HTTP callers keep the timed wait and disconnect check. Both paths continue through the existing iterator, output handling, and request cleanup. This adds no public API and does not change notification settings or scheduler emission.

Validation

• Focused request-lifecycle tests: 30 passed, 6 subtests passed using pytest -q test/registered/unit/managers/test_tokenizer_manager_rid_cleanup.py.
• New coverage checks available and delayed output order, terminal output, cancellation releasing the event waiter, abort/error delivery, and HTTP timeout disconnect detection. Status 503 coverage exercises error delivery, not a full Engine shutdown integration test.
• All applicable checks passed with pre-commit run --files python/sglang/srt/managers/tokenizer_manager.py test/registered/unit/managers/test_tokenizer_manager_rid_cleanup.py.

The patch changes one conditional in the response iterator and adds coverage to the existing CPU CI test file. It does not change model or kernel code.

Speed Tests and Profiling

In one matched Dynamo SGLang TCP push comparison, this wait change improved output token throughput by 37.7% at batch_notify_size=16, SGLang's default. Both controls used fixed concurrency 1024, ISL/OSL 33/1024, stream_interval=1, and no worker runtime thread override. Only the response wait changed.

Response wait	Output tokens/s	Completed requests	Duration including drain (s)	p99 TTFT (s)
Existing timed wait	18,494.13	3072	170.093	0.936
Direct event wait	25,464.63	3072	123.533	1.375

Throughput uses completed output tokens divided by actual duration including drain. Each control used 2048 separate warmup requests, a 120-second measurement, and a 90-second grace period. Both controls completed without request errors and had exact token lengths. Separate diagnostic captures retained one token per token-bearing backend frame. TTFT tails increased, as shown above.

For context, native SGLang HTTP serving in the same campaign had 45.243 seconds p99 TTFT, versus 1.375 seconds for the patched Dynamo control. Both used notification size 16 and fixed concurrency 1024. Native used 4 tokenizer / 4 detokenizer processes; Dynamo used 1/1. The patched Dynamo TTFT therefore remained well below this native reference, despite increasing relative to the unchanged Dynamo control.

This is one control per configuration, so the result does not establish repeatability or a GIL mechanism. GPU measurements used SGLang main 5c2de3f35567ffceec6cea86ba18e075692e9101 plus this patch, before the PR was applied to the current main base.

---

CI States

Latest PR Test (Base): ✅ Run #35367509826
Latest PR Test (Extra): ❌ Run #35367509417
Latest PR Test (AMD ROCm 10): ❌ Run #35367509900

[ishandhanani]

/tag-and-rerun-ci

[jthomson04]

/rerun-failed-ci

[jthomson04]

/rerun-failed-ci

[jthomson04]

/rerun-failed-ci

[ishandhanani]