fix(frontend): render chat templates off the async runtime (restores #15092); make the CallHome listen backlog configurable

[tzulingk]

Overview:

Two changes that together stop an intermittent warmup failure on a 2,934-concurrency agentic workload (DeepSeek-V4-Pro, disaggregated 5P×6 / 1D×16, 46 GPUs):

1. fix(runtime): the CallHome listen backlog goes back to 4096, the behavior before the mio 1.2.2 bump silently cut it to 128.
2. fix(frontend): restores the closed PR fix(frontend): render the chat template off the async runtime #15092, which renders the chat template via block_in_place.

In our runs neither change was enough alone (3/8 and 2/6 passing); together, 11/12 passed.

On cost: this PR does not claim that chat-template rendering is expensive. Measured renders are cheap: a mean of about 1.5–2.6 ms, and under 1% take more than 50 ms even during the burst. The case for #15092 is the failure-rate result below. How it helps is a hypothesis, stated as one in "Why #15092 might matter".

Details:

What fails

At AIPerf's warmup gate, about 3,600 requests start within 0.5 s, and a single error on a root request aborts the whole run. The two arms without #15092 fail in two different ways, one for each fix:

	backlog 128 (arm A)	backlog 4096, no #15092 (arm B)
failed runs	3 / 3	5 / 8
error that aborts the run	a burst of ClientOSError(104) connection resets (3–23 at once)	exactly one ClientOSError(110) timeout
when, after the gate	1.3–2.6 s	0.34–0.44 s
did the request reach Dynamo?	no: 38 of 39 reset request ids are absent from the frontend log (job 2934)	yes: the frontend logged a 400 for that request

• Raising the backlog removes the resets entirely: there are 0 in all 8 arm-B runs. What remains is a request that Dynamo accepted but never completed.
• With fix(frontend): render the chat template off the async runtime #15092 as well (arms C and E), runs with any request error during the gate fell from 5/8 to 2/12. Only one of those two errors hit a root request.

The measured experiment

29 runs of the same workload on the same topology and hardware. The listen backlog was set with an LD_PRELOAD listen(2) shim, so it could be held fixed independently of the build. Each run is scored at the warmup gate.

arm	backlog	map_while	#15092	passes
A	128	✗	✗	0 / 3
B	4096	✗	✗	3 / 8
D	128	✓	✓	2 / 6
C	4096	✓	✓	5 / 6
E	4096	✗	✓	6 / 6

comparison	result	Fisher one-tailed p
#15092, at backlog 4096 (B → E)	3/8 → 6/6	0.028
backlog, with #15092 (D → C)	2/6 → 5/6	0.12
backlog, with #15092, pooled (D vs C+E, post hoc)	2/6 vs 11/12	≈ 0.02
backlog, without #15092 (A → B)	0/3 → 3/8	0.34
map_while (C → E)	5/6 → 6/6	0.50

map_while is an unrelated filter_map → map_while change in addressed_router.rs that was present in some test builds. It is not part of this PR and adds nothing here. Arm D is therefore effectively "#15092 with the backlog at 128", and it passed only 2/6.

Why #15092 might matter (hypothesis, not measured)

• The LIFO slot. Tokio's multi-thread scheduler gives each worker one LIFO slot, which holds the task most recently scheduled on that worker, including wakeups from the I/O driver. Other workers can steal from a busy worker's run queue, but not from that slot (tokio-1.52.4/src/runtime/scheduler/multi_thread/worker.rs:114-120, 451-453).
• Without fix(frontend): render the chat template off the async runtime #15092, the render runs inline, so the task in the slot waits until the render finishes. That task is often a socket task that was just woken.
• With fix(frontend): render the chat template off the async runtime #15092, block_in_place first moves that task to the stealable queue and hands the worker to a replacement thread (worker.rs:454-457).
• Tokio documents this starvation mode. It is the stated reason for the unstable Builder::disable_lifo_slot: "task 'scheduled' time is high but the runtime is underutilized" (rt: make the LIFO slot in the multi-threaded scheduler stealable tokio-rs/tokio#4941). That matches this workload, where 352 workers are mostly idle.

What is not shown:

• The delay's size. Each render is about 2 ms, so the delay per worker is small, and we haven't measured that it is what produces the 110 above.
• The 110 itself. It fires about 0.4 s after the request is sent, far below any TCP timeout: tcp_retries2 ≈ 15 min, SYN retries ≈ 2 min, AIPerf's TCP_USER_TIMEOUT 30 s. A packet capture on the HTTP port across the gate would settle it.

block_in_place is used rather than spawn_blocking because apply_template borrows the request generically (&R), so there is no 'static value to move into a spawned closure. It does not create unbounded threads: the replacement comes from the same capped blocking pool as spawn_blocking.

Why #15092 was closed, and why this revisits it

#15092 was closed after measuring the render as negligible, and after a throughput comparison showed no change. Our data agrees on both counts: the render is cheap, and throughput of runs that pass is unchanged. What that comparison could not see is how often runs fail at the warmup gate, which is what changes here.

The backlog change

tokio::net::TcpListener::bind listens with mio's backlog. On 2026-08-24, commit 138668ba94 ("feat(mm-routing): add Kimi K3 support", #13371) bumped the workspace pin from mio = "=1.1.1" to "=1.2.2". mio 1.1.1 passed -1 on Linux (sys/mod.rs:132, "Silently capped to /proc/sys/net/core/somaxconn"), while mio 1.2.2 hardcodes 128. Every dynamo-runtime listener silently lost 32× of accept-queue depth on hosts with somaxconn = 4096. #15080 restored this for the HTTP listener, but the CallHome listener was missed.

CallHome is the exposed one. Every response stream is its own connection (tcp/client.rs create_response_stream → TcpStream::connect, no pooling, once per request), so its accept rate scales with in-flight requests. Sampling sk_ack_backlog from /proc/net/tcp every 500 ms on live runs, the CallHome listener reached depths of 404 and 669, against the 128 limit.

Adds DYN_TCP_LISTEN_BACKLOG (default 4096, capped by the kernel at somaxconn) and binds via tokio::net::TcpSocket.

Listeners deliberately NOT changed

The same regression affects five other production binds. All were measured, and none is included:

site	why not
runtime/.../shared_tcp_endpoint.rs	request plane, worker-side; the client pools at 100 connections per host, so load is O(workers)
runtime/system_status_server.rs	measured at depth 0 across 21,524 samples; it also binds a host:port string, which would need new multi-address resolution code
llm/http/service/service_v2.rs:1082	RL worker discovery; opt-in (rl_router is None unless configured)
llm/block_manager/metrics_kvbm.rs	standalone Prometheus /metrics, driven by scrape rate
llm/kv_dc_relay/wan/grpc/server.rs	gRPC; HTTP/2 multiplexes streams per connection, so load is O(peers)

Caveats

• Build confound. Arms A and B used the published nightly 1.5.0.dev20260910; arms C, D and E used local builds at 7540ca80f. So B → E measures fix(frontend): render the chat template off the async runtime #15092 plus a newer build, and no run separates the two.
• Small samples. n ≤ 8 per arm; E's 6/6 is n = 6.
• The fix(frontend): render the chat template off the async runtime #15092 mechanism is a hypothesis (see above). The backlog mechanism is measured: queue depth, and the reset errors that disappear at 4096.

Where should the reviewer start?

• lib/llm/src/preprocessor.rs: off_runtime and its single call site in preprocess_request_with_options
• lib/runtime/src/pipeline/network/tcp.rs: DEFAULT_TCP_LISTEN_BACKLOG, parse_tcp_listen_backlog, bind_listener
• lib/runtime/src/pipeline/network/tcp/server.rs:763: the single call-site change
• environment_names.rs, snapshot/constants.py, checkpoint/compatibility.go: DYN_TCP_LISTEN_BACKLOG registration and snapshot-restore allowlists

A note on TCP semantics.

• Delay, not reset: with net.ipv4.tcp_abort_on_overflow=0, an overflowing accept queue usually shows up as delay. The kernel drops the client's final ACK and relies on SYN-ACK retransmission (1 s, 3 s, 7 s). A lab test of 3,000 dials at backlog 128 with no acceptor gave ListenOverflows +11484 and zero client-side resets.
• Where arm A's resets likely come from: these hosts also run tcp_syncookies=1. There, a connection can complete via a SYN cookie while the accept queue is full, and its later data segments can then be answered with a reset. That is the leading explanation for arm A's resets, but it has not been confirmed with a packet capture.

Validation

• cargo test -p dynamo-llm --lib --no-default-features -- preprocessor: 222 passed
• cargo test -p dynamo-runtime --lib -- backlog: test_tcp_listen_backlog_env_var covers default, zero, invalid, overflow, i32::MAX and whitespace input. The bind/accept path is covered by the existing TcpStreamServer round-trip tests. The standalone test_bind_listener_accepts_connections was removed in review as a duplicate.
• cargo build -p dynamo-runtime and cargo build -p dynamo-llm --no-default-features: clean
• cargo fmt --all -- --check: clean
• cargo clippy -p dynamo-runtime --no-deps --lib -- -D warnings: one pre-existing failure, unused variable: join_reuseport in quic_response.rs (a macOS-only cfg path), which reproduces identically on unmodified main
• Field: 29 runs across 5 arms, as tabulated above, with sk_ack_backlog sampled at 500 ms.

Related Issues

• Restores fix(frontend): render the chat template off the async runtime #15092 (closed), with failure-rate evidence rather than throughput evidence.
• Follow-up to fix(frontend): make the HTTP listen backlog configurable, default 4096 #15080, which fixed the same backlog regression for the HTTP listener.

🚫 This PR is NOT linked to an issue:

• Confirmed — no related issue

🤖 Generated with Claude Code

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository: ai-dynamo/dynamo/.coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c2ec258f-e65c-4457-9f10-d6e5104b0014

📥 Commits

Reviewing files that changed from the base of the PR and between 2164d01 and ada9497.

📒 Files selected for processing (3)

• lib/runtime/src/config/environment_names.rs
• lib/runtime/src/pipeline/network/tcp.rs
• lib/runtime/src/pipeline/network/tcp/server.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The TCP CallHome listener now uses a configurable listen backlog. The setting defaults to 4096 when it is missing or invalid. The listener creates and binds a socket with the parsed backlog.

Changes

TCP listener backlog

Layer / File(s)	Summary
Configure and bind the TCP listener lib/runtime/src/config/environment_names.rs, lib/runtime/src/pipeline/network/tcp.rs, lib/runtime/src/pipeline/network/tcp/server.rs	Adds the DYN_TCP_LISTEN_BACKLOG environment-variable constant and parsing for values from 1 through i32::MAX. The binding helper selects the address family, enables address reuse, binds, and listens with the parsed backlog. tcp_listener uses the helper. Tests cover defaults, bounds, whitespace-padded input, and accepting a connection.

Priority: ➖ Normal

Estimated code review effort: 2 (Simple) | ~12 minutes

Merge Risk: ⚪ Minimal · up to ada94

The CallHome response-stream listener now uses a configurable listen backlog, defaulting to 4096 and falling back to the default for invalid values. On the supported Unix targets, socket options match the previous behavior. No blocking risk remains.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 57.14% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 7 functions across 3 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title identifies the configurable CallHome listen backlog, which matches the pull request changes. The frontend clause is not supported by the provided file-change summary, but the title remains p…
Description check	✅ Passed	The description includes all required template sections and provides detailed implementation, testing, experiment, and review guidance. The Related Issues section is marked as not linked while still r…

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: The previously raised redundant narration remains at lib/llm/src/preprocessor.rs:1391-1392; it only compares this helper with tokenization and does not preserve an additional runtime constraint.
• Original discussion: The unresolved block_in_place concern remains: each concurrent template render occupies its Tokio worker while Tokio creates replacement workers, bypassing the configured DYN_RUNTIME_MAX_BLOCKING_THREADS limit. A burst of long renders can therefore grow process threads and memory without the configured backpressure. Move rendering to a bounded blocking executor after owning the required request data.
• Original discussion: Verified still present: off_runtime uses block_in_place for every concurrent template render, bypassing the configured bounded blocking executor. A burst of long renders can retain caller threads and cause Tokio to create replacement workers per blocked render, allowing thread and memory growth instead of applying the configured blocking-work backpressure.
• Original discussion: Verified still present: off_runtime uses block_in_place, which replaces blocked worker threads outside the configured DYN_RUNTIME_MAX_BLOCKING_THREADS limit. A burst of long template renders can therefore create unbounded replacement workers and exhaust process thread or memory capacity rather than applying the configured backpressure.

[KrishnanPrash]

/ok to test 4b58878

[jthomson04]

Approving the code change, with two caveats:

1. Remove unsupported claims before merge. This includes the code comment that template rendering leaves request bodies unread for “tens of seconds,” and categorical claims that the changes “stop” the failures. State the observed results and their limits. The B → E comparison uses different builds, so it does not isolate the template change or establish its mechanism.

2. Non-blocking follow-up: determine the cause. As discussed in the evidence request on Slack and the follow-up question, we still need to establish why this small amount of template work affects the failure rate, and whether the same mechanism applies elsewhere in the request path. Use comparisons on the same build and direct measurements during the failing burst to distinguish rendering cost, scheduling delay, and other causes. Keep the LIFO explanation marked as a hypothesis until that evidence exists.

Source review only; I did not run tests or inspect CI.

[rmccorm4]

/ok to test 4b58878

[rmccorm4]

/ok to test d35f4a0