fix(runtime): fix QUIC reassembly and reduce response stalls

[jthomson04]

Summary

QUIC response connections can fail with too many gaps in stream buffer when quinn-proto 0.11.17 retains many small, contiguous chunks. Its stream assembler can reach the chunk limit even with lossless, ordered data. Update Quinn from 0.11.9 to 0.11.12 and quinn-proto from 0.11.17 to 0.11.18 in all four lockfiles to include the upstream fix that coalesces contiguous chunks during defragmentation (quinn-rs/quinn#2826).

Include the full error cause chain when a QUIC response lane or worker connection bundle fails. Frontend lane warnings also include Quinn's connection-close reason as a best-effort observation; another task can initiate teardown before that reason is read. The once-per-bundle worker warning includes the bundle ID, failure path (writer or control_reader), and remote frontend address. Worker registration failures receive the richer cause text through the existing failure handler. The worker warning records the failure reason as a string field so console logs escape embedded newlines and terminal control characters. Added formatting and address lookup run on error paths.

Reduce receive-side scheduling stalls by driving each lane's reader and reverse-control writer as pinned futures in one task. Charge Tokio's cooperative budget at 16-frame or 4 KiB thresholds so buffered token frames can advance without charging each frame separately, while still yielding for control traffic. Both futures stay pinned across polls so partial reads are preserved.

Increase Linux frontend receive endpoints from 8 to 32, using the existing reuse-port group and advertised address. This adds 24 sockets per response server. The endpoint count remains one on other platforms. The wire format, worker connection/lane counts, failure counters, and public configuration are unchanged. TCP remains the default; the shipped QUIC batch interval remains 5 ms.

This is a dependency release update. In addition to reassembly, quinn-proto 0.11.18 changes send-window accounting, ACK bundling, blocked-frame emission, datagram/path handling, and transport-parameter validation. The evidence below does not isolate the performance effects of those changes.

The Quinn 0.11.9 → 0.11.12 update also changes connection-handle reference counting to use atomics, cleans up receive-stream stop/drop state, and uses the clock to detect timer expiry when Tokio cooperative-budget limits can delay Sleep::poll. Quinn's minimum supported Rust version increases from 1.74.1 to 1.85; the repository's pinned Rust 1.96.1 toolchain meets that requirement. The measured performance differences cannot be assigned to an individual Quinn or quinn-proto change.

Validation

Checks on the PR branch

• On 1ec03db608ae462282cdb8747f24cfa210ea54ab, cargo test --locked -p dynamo-runtime --lib pipeline::network::quic_response::tests -- --test-threads=8 — 22 passed, including 1,000 ordered responses, cancellation, connection failure/reconnection, failure before prologue, and endpoint shutdown.
• cargo clippy --locked -p dynamo-runtime --lib — passed with no warnings.
• cargo fmt --all -- --check and git diff --check — passed.
• A one-off local formatting check with the locked tracing 0.1.44 / tracing-subscriber 0.3.23 versions used the compact formatter with colors disabled. A reason containing a newline and ANSI escapes produced two physical lines with %reason, but one physical line with escaped payload characters using reason. The check and output remain local artifacts, outside the commit.
• Prior locked dependency check: cargo tree --locked --manifest-path <manifest> -i quinn-proto --prefix none --depth 1 — all four workspaces resolve Quinn 0.11.12 and quinn-proto 0.11.18 without lockfile changes.
• Prior diagnosis against the same published quinn-proto crates: upstream's ordered_lossless_stream_is_not_rejected test fails with TooManyChunks on 0.11.17; all 26 assembler tests pass on 0.11.18, including the excessive-gap limit tests. The new crate checksum matches the PR lockfiles. Application-frame count alone does not guarantee the assembler retention needed to reproduce this defect, so the deterministic reproduction remains in the upstream assembler tests.

Deferred validation

The requested baseline-versus-head benchmark with the shipped 5 ms batch interval is deferred for this PR. No new performance benchmark was run for this follow-up. The existing Tyche results use a zero batch interval and common campaign changes; they do not validate this exact PR under the shipped configuration. That performance-validation gap remains open.

Existing Tyche comparison: Quinn update

The two measured QUIC arms differ only in the Quinn manifest/lockfile update: old 829f64238cfe7cf06c52d4d52087a60ea5d13e62, updated fee3f7408a9a754245a18ed611020f8cb44d7997. Both include other campaign changes and use a zero batch interval, rather than the exact PR code with the shipped 5 ms interval.

Cross-node mock-worker runs used eight worker processes, a separate load-generator node, fresh services, consistent warmup, identical request fixtures, and two 90-second repetitions per case with reversed arm order. Coverage was 128-input/32-output requests at concurrency 512, 1,024, and 4,096, plus mixed requests at concurrency 1,024. These are existing campaign results; they precede the reader-budget and 32-endpoint changes below.

QUIC arm	Cases	Attempts	Completed requests	Client errors
Previous Quinn	8	18,689,295	17,421,429	1,267,866
Updated Quinn	8	20,827,303	20,827,303	0

The updated arm's exported request lengths also matched, with no bundle-failure counter increments in the measured resource windows. The previous arm included one response with 24 output tokens instead of the requested 32.

For the short workload at concurrency 512:

Repetition	Old p99, ms	Updated p99, ms	Old client errors	Updated client errors
0	131.56	183.19	6,677	0
1	133.41	188.13	7,865	0

Latency percentiles exclude failed requests, so these are not equivalent successful cohorts. The connection-failure fix is supported in this tested scope, but p99 increased and frontend UDP receive drops remain unresolved. This evidence does not establish latency parity, isolate ACK or send-window effects, or qualify QUIC as the default. TCP remains the default.

Tyche receive-path evidence

The receive-task change and the 16-frame / 4 KiB budget thresholds match the measured campaign implementation; the Linux endpoint count is 32. These are focused ports of campaign commits 2c41ea8476, 901cb517bc, and f969ff855f.

In the earlier single-run short-response screens (128 input / 32 output tokens, concurrency 512), previous QUIC had 192.571 ms total p99 at 38,765.47 requests/s. Reader batching plus 32 endpoints had 24.398 ms p99 at 40,637.41 requests/s. Its paired TCP run had 10.635 ms p99 at 36,176.16 requests/s. The approximately 87% QUIC p99 reduction is a single-run observation; it does not establish repeatability or TCP latency parity.

The full 8-versus-32 endpoint comparison has 36 runs, three alternating pairs per case, concurrency 1/16/64, one/eight mock workers, and 6,651,658 successful requests with no client errors. Worst paired regressions were 3.10% throughput, 1.60% total p99, and 0.17% serving CPU per completed request.

There is a memory cost: with eight workers at concurrency 64, median frontend peak RSS increased from 169.4 to 219.8 MiB. The extra endpoints add 69 MiB of Quinn receive-buffer allocation capacity under the checked Tyche Linux GRO settings. UDP-drop variation and isolated TTFT/token-time tail regressions are retained in the linked results.

Both campaigns used other common frontend/QUIC changes and explicitly set the batch interval to zero. The campaign also used a 32-frame sender batch cap and reorder-limit backpressure; this PR retains the existing 512-frame cap and reorder behavior. These measurements are not a benchmark of this exact PR commit or its shipped 5 ms interval. The tests above validate the port. The remaining high-concurrency and workload coverage gaps still block a default transport change.

Related Issues

• Confirmed — no related Dynamo issue.

Summary by CodeRabbit

• Bug Fixes

  • Improved QUIC connection failure messages with more readable error details.
  • Added connection close reasons to server-side warning messages, making network issues easier to diagnose.

• Chores

  • Updated the QUIC networking dependency to a newer patch release while preserving existing runtime and security configuration.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: e6df3dfa-87b9-407b-8799-394b9472cfb7

📥 Commits

Reviewing files that changed from the base of the PR and between 95cadce and 2bcd9ae.

⛔ Files ignored due to path filters (4)

• Cargo.lock is excluded by !**/*.lock
• lib/bindings/kvbm/Cargo.lock is excluded by !**/*.lock
• lib/bindings/python/Cargo.lock is excluded by !**/*.lock
• lib/runtime/examples/Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (2)

• Cargo.toml
• lib/runtime/src/pipeline/network/quic_response.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 10 remain after this review.

---

Walkthrough

The workspace updates Quinn from 0.11.9 to 0.11.12. QUIC server and client failure logs now use alternate error formatting. Server lane warnings also include the connection close reason.

Changes

QUIC updates

Layer / File(s)	Summary
QUIC dependency version Cargo.toml	The workspace updates quinn to 0.11.12 and retains its existing features.
QUIC failure reporting lib/runtime/src/pipeline/network/quic_response.rs	Server lane warnings include the connection close reason. Client writer and control-reader failures use pretty-formatted error chains.

Priority: ⬇️ Low

Estimated code review effort: 1 (Trivial) | ~5 minutes

Merge Risk: ⚪ Minimal · up to 2bcd9

The dependency versions are consistent, and the diagnostic changes preserve QUIC cleanup and registration-error behavior without demonstrated production impact.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (1 skipped: 1…	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title accurately identifies the main changes: the QUIC reassembly fix and reduced response stalls. It is concise and specific.
Description check	✅ Passed	The description provides detailed change context, validation results, limitations, and a completed Related Issues section. It does not use the template headings exactly and omits a dedicated reviewer-…

Full details: Docstring Coverage

Explanation

Docstring coverage is 50.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 2 functions across 1 files. (1 skipped: 1 unsupported.)

• Fix all pre-merge checks with AI

---

_{Comment @coderabbitai help to get the list of available commands.}

[jthomson04]

Automated review (Claude Opus 5, xhigh effort). Every claim below was checked against the quinn 0.11.9 / quinn-proto 0.11.17 sources in the local cargo registry and against the published quinn-proto 0.11.18 crate; line references are to head 2bcd9ae.

Six inline comments. The two I'd want answered before merge:

1. The new close_reason field is overwritten by our own teardown before most lanes read it, so it usually reports our close rather than the cause it was added to expose.
2. 0.11.18 carries behavior changes beyond the reassembly fix — send_window accounting and ACK bundling — that the Validation section doesn't cover.

One item with no diff line to anchor to: no regression test pins either half of this PR. The fix lives entirely in a transitive crate version, and the existing quic_response::tests already ran real local QUIC connections without catching the original failure — they don't push enough small contiguous frames through one lane to reach COMPACT_THRESHOLD = 2 * MAX_CHUNKS = 2048 retained chunks on the receiver (quinn-proto src/connection/assembler.rs:358-364). A test that streams >2048 small frames through a single lane with a slow reader and asserts the stream survives would fail on 0.11.17, pass on 0.11.18, and would also catch a future Cargo.toml relaxation from 0.11.12 back to 0.11.

[jthomson04]

Review follow-up in 24001ac44b: the logging cleanup and worker failure identifiers are implemented. All 22 existing QUIC runtime tests pass, as do formatting, diff checks, and locked dependency resolution for all four workspaces. The description now includes the prior Tyche evidence and its latency and configuration limits. I replied to all six inline threads and left them open for agreement on the partial acceptances and deferrals.

For the review-summary request for a >2,048-frame regression test: application frames do not map one-to-one to retained Quinn assembler chunks. Packetization, buffering, and consumption affect that state, so the proposed count alone does not make the test deterministic. The existing upstream reproduction inserts small chunks directly: it fails with TooManyChunks on 0.11.17, while all 26 assembler tests pass on 0.11.18. That evidence remains in the description. Also, relaxing a manifest requirement would not downgrade an already locked dependency, so the proposed test would not itself pin that requirement. No additional transport stress test was added.

The CodeRabbit docstring-coverage warning is advisory for existing private functions. No documentation expansion is needed for this change.

[dmitry-tokarev-nv]

Reviewed at 24001ac. Intent, as stated in the description and readable from the diff: move quinn 0.11.9 -> 0.11.12 / quinn-proto 0.11.17 -> 0.11.18 to pick up the stream-assembler coalescing fix, and widen QUIC failure reporting from a single-line Display to the full cause chain plus bundle / lane / peer identity.

What I checked by execution, on macOS ARM64 with the pinned toolchain (rust-toolchain.toml = 1.96.1):

Lockfiles resolved, not hand-merged. find . -name Cargo.lock returns exactly the four files this PR touches, and all four move both crates together. The only manifest naming quinn is the root Cargo.toml; lib/runtime/Cargo.toml takes it from the workspace, so there is no second pin to drift. I downloaded the four .crate files from crates.io and the SHA-256s match the lockfile lines byte for byte (quinn 0.11.12 4051e23e…d9d11, quinn-proto 0.11.18 a9746dbd…c9fc). Diffing the published manifests: quinn-proto 0.11.18 changes no dependencies at all, and quinn 0.11.12 changes only dev-dependencies and a test-only feature, its one runtime change being proto 0.11.12 -> 0.11.18. That is exactly the shape the lock diff shows — no dependency-list churn, which is what a hand-merge would have gotten wrong.

The fix is actually in the bump. quinn-proto-0.11.18/src/connection/assembler.rs introduces MIN_RETAINED_CHUNK_SIZE = 128 and coalesces any chunk below max(buffered.div_ceil(MAX_CHUNKS), MIN_RETAINED_CHUNK_SIZE) during defragment(), and ships ordered_lossless_stream_is_not_rejected as its regression test. That is the change the description points at.

MSRV moves. quinn's rust-version goes 1.74.1 -> 1.85 (quinn-proto was already 1.85). No impact at 1.96.1, but it is a real constraint change.

Tests. cargo test --locked -p dynamo-runtime --lib pipeline::network::quic_response -- --test-threads=8 -> 21 passed here against the description's 22; the gap is the #[cfg(target_os = "linux")]-gated test at line 3106, not the diff.

One P2 inline below. I also left replies with independent verification on the five open threads, including one measurement that retracts part of an earlier claim.

[dmitry-tokarev-nv]

Approving at 24001ac. Nothing I found is blocking: zero P0/P1, two non-blocking items, both named below.

Verified by execution, not by reading (macOS ARM64, toolchain pinned at 1.96.1):

• Lockfiles are re-resolved, not hand-merged. All four Cargo.lock files in the repo are the four touched here, and they move both crates together. The root Cargo.toml is the only manifest naming quinn (lib/runtime takes the workspace dep), so there is no second pin to drift. I fetched the four .crate files from crates.io and their SHA-256s match the lockfile lines exactly. Diffing the published manifests: quinn-proto 0.11.18 changes no dependencies at all, and quinn 0.11.12 changes only dev-dependencies plus a test-only feature, its one runtime change being proto 0.11.12 -> 0.11.18 — which is precisely why the lock diff has no dependency-list churn.
• The fix is in the bump. assembler.rs in 0.11.18 adds MIN_RETAINED_CHUNK_SIZE = 128, coalesces sub-threshold chunks in defragment(), and ships ordered_lossless_stream_is_not_rejected.
• Transport-parameter strictness is receive-side only (transport_parameters.rs: a remaining_before - r.remaining() != len check rejecting Error::Malformed). Both 0.11.17 and 0.11.18 write well-formed parameters, so a mixed-version fleet during a rolling upgrade does not break on it.
• MSRV moves 1.74.1 -> 1.85 on quinn; no impact at 1.96.1.
• cargo test --locked -p dynamo-runtime --lib pipeline::network::quic_response -- --test-threads=8 -> 21 passed. The description's 22 is the #[cfg(target_os = "linux")] test at line 3106, not a discrepancy.

Outstanding, non-blocking:

• P2 — line 1972: {error:#} now carries a peer-supplied CONNECTION_CLOSE reason phrase into reason, and %reason is written unescaped by the console formatter, so a peer can inject newlines and ANSI into the worker's log. Measured over a real connection on the pinned crates: base string 15 bytes, head string 1201 bytes with an embedded newline, from a 65 KiB reason the peer offered. Size-bounded, so this is a log-integrity issue rather than a resource one. One-token fix (%reason -> reason), verified to compile, test and lint clean.
• P3 — the description covers quinn-proto 0.11.18 thoroughly but not quinn's own 0.11.9 -> 0.11.12 span, which changes connection-handle refcounting, RecvStream stop/drop, and — relevant to this file's cooperative-budget comment — drive_timer no longer trusting Sleep::poll for expiry. Details on the Cargo.lock thread.

Deliberately deferred, as decisions rather than oversights — I have replied on each with independent verification:

• Benign-vs-invariant classification of lane exits and the unreachable close_reason().is_none() guard in the accept loop. Confirmed unreachable on 0.11.12 too, and confirmed pre-existing.
• Coordinated first-cause capture for close_reason; the description now labels it best-effort. Keeping the explicit close on 938-939 is correct — fail_server_connection returns early when no bundle is registered, which is exactly the pre-registration failure case.
• send_prologue's to_string(): I followed enqueue_on and every arm is a flat error with no source(), so {error:#} there would be inert. Correct to leave alone.
• send_window accounting: the unacked() -> buffered() = unacked_len + front_trimmed change and the 8 x 1.25 MB priority-connection arithmetic both check out, but nothing I have says it bites in practice. Accepting the deferral.

I authored no commits on this PR.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Still present: the new %reason log field renders peer-controlled QUIC close-reason text without escaping. A peer can inject newlines or terminal escapes into worker logs; record reason as a normal string field instead.
• Original discussion: Verified still present: the new full cause chain can contain a peer-controlled QUIC CONNECTION_CLOSE reason, and %reason writes it unescaped to the worker log. A peer can inject physical log lines or ANSI control sequences. Record reason as a normal string field instead of a Display field so tracing escapes it.
• Original discussion: Verified still present: the newly expanded Quinn error chain can contain a peer-supplied CONNECTION_CLOSE reason, and %reason logs it without escaping. Newlines or ANSI escapes can forge physical log records. Record reason as a normal string field instead of a Display field.

[jthomson04]

Summary

Tyche follow-up: 8 versus 32 QUIC receive endpoints

The focused low-concurrency comparison found no repeatable throughput, total-p99 latency, or serving-CPU regression above the agreed limits from increasing the frontend receive endpoint count from 8 to 32. There is a material memory cost.

Scope: these measurements use separate campaign commits, both with the reader scheduling change and other common frontend campaign changes. They isolate the endpoint count. PR commit a3079ba6fa712a778efdca071ef04029a6cc6d25 now includes the receive-task change, reader budget batching, and 32 Linux receive endpoints. The campaign also used a 32-frame sender batch cap and reorder-limit backpressure, while this PR retains its existing 512-frame cap and reorder behavior. These measurements do not measure the effect of the Quinn update alone or benchmark this exact PR commit. All 22 QUIC runtime tests and formatting checks pass on the ported PR code.

Validation

• 36 selected runs / 18 paired comparisons / 6,651,658 successful requests. Full exported request counts and final input/output token lengths match. Zero client errors and zero measured QUIC bundle failures or mailbox resets in the selected runs.
• Concurrency 1, 16, 64, with one and eight mock-worker processes, one frontend limited to four CPU cores, and a separate load-generator node. Cross-node Tyche Neoverse-V2 hosts with 144 CPUs each and a 200 Gbit/s network interface.
• Exact 128-input / 32-output tokens, fixed 512-record prompt corpus, seeded output records, seed 20260915, Qwen3-0.6B tokenizer, round-robin routing, block size 16, mock prefill/decode speedup 1000.
• Fresh services, 512 warmup requests, then 60 seconds of measurement per run. Three adjacent pairs per case; endpoint order is reversed in the middle repetition. Actual run timestamps verify this order.
• Both arms explicitly use DYN_QUIC_RESPONSE_BATCH_INTERVAL_US=0. These results do not cover the shipped 5 ms batching interval, GPU backends, other request shapes, or other hardware. No builds or temporary profiling ran during measurement.

Across all 18 pairs, the worst observed regressions were 3.10% in throughput, 1.60% in total p99, and 0.17% in serving CPU per completed request. All six case medians satisfy the 5% throughput/total-p99 and 10% serving-CPU limits.

Percentages in the table are the median of three paired changes (32 / 8 − 1), not ratios of independently calculated arm medians. Negative latency/CPU changes are better.

Workers	Concurrency	Requests/s, 8 / 32 medians	Paired throughput change	Total p99 ms, 8 / 32 medians	Paired total-p99 change	Paired serving CPU/request change	Requests
1	1	650.5 / 671.7	+0.03%	1.561 / 1.540	-0.76%	-0.82%	237,047
1	16	4326.3 / 4300.8	-0.04%	5.219 / 5.245	+0.04%	-0.11%	1,552,730
1	64	4516.8 / 4582.2	+1.45%	22.279 / 22.141	+0.39%	-0.47%	1,633,422
8	1	655.0 / 661.7	+1.35%	1.598 / 1.580	-1.12%	-1.14%	236,611
8	16	4121.1 / 4137.4	+0.66%	5.536 / 5.542	-0.94%	-0.83%	1,484,571
8	64	4168.3 / 4183.2	+0.25%	25.347 / 25.482	+0.53%	+0.16%	1,507,277

Costs and remaining limits

• Memory: with eight workers at concurrency 64, median frontend peak resident memory rises from 169.4 to 219.8 MiB. The three paired increases range from 19.8% to 32.9%. The change adds 24 UDP sockets per frontend response server. Quinn 0.11.12 allocates a receive buffer per endpoint; with the checked Linux GRO support and defaults, the additional buffer capacity is 69 MiB (24 × 1472 × 64 × 32 bytes). This explains additional allocation capacity, not every measured resident byte.
• UDP drops: none at concurrency 1 or 16. At concurrency 64 with one worker, totals across three runs are 1,047 for 8 endpoints / 2,488 for 32; 2,383 candidate drops occur in one run. With eight workers, totals are 7,448 / 0. Drop variation is retained and was not isolated causally in this pass. These results do not establish that 32 endpoints always reduces drops.
• Other latency tails: one TTFT-p99 pair regresses by 5.67%, and one request-average-token-time-p99 pair regresses by 8.03%. Neither repeats above 5% in the other two pairs. The corresponding largest case-median regressions are 3.87% and 1.15%. The CSV below retains every run.
• CPU accounting: serving CPU includes frontend, workers, and etcd. CPU and the completed-server-request denominator use the same steady measurement window, excluding startup and drain. Whole serving-node CPU, including kernel activity, is also recorded; paired changes range from -4.71% to +0.45%. The client is excluded from serving costs.
• Scope: the frontend uses about 0.7–0.8 cores at concurrency 1 and 3.76–3.82 cores at concurrency 16/64. Thus, the concurrency-64 cases also stress the four-core CPU limit. This pass does not establish absolute load-generator capacity.

Client “inter-token latency” is each request's average (total latency − TTFT)/(output tokens − 1). Its p99 is a percentile of request averages, not individual token-gap p99.

Source records and exclusions

Arm	Campaign source commit	Binary SHA-256
8 endpoints	901cb517bc55e06d4ddc728575fa25d2fd15b5ba	8a25728da30a31ee9a5a5b7944c040150d05b5a9b7398aa8bca19c2b58b8fd59
32 endpoints	f969ff855f91ae45d34c61ddbf60291f98c0b2f1	c7f79e12767fff20e7535d3614ec2649a74b8fb54c8e86cfa21fe339fc67903f

The source difference between these arms is the Linux SERVER_ENDPOINTS constant, 8 → 32, plus its associated comment. The shared reader change charges Tokio's cooperative budget per bounded batch of small frames (16 frames / 4 KiB thresholds), rather than per frame.

Five attempts are outside the 36-run comparison: two initial checks; one completed unpaired run retained after correcting the endpoint-order schedule; one infrastructure-interrupted run; and that run's completed partner. The interruption was an expired Kerberos ticket that prevented writing result files. Both arms were rerun together on the same four nodes after access was restored. All 40 completed runs were audited; the failed attempt and its artifacts remain retained.

Conclusion: this supports the low-concurrency performance of 32 endpoints for the tested short-response workload, subject to its memory cost. The earlier high-concurrency QUIC/TCP gaps and other qualification gaps remain. It does not qualify a default transport change.

All 36 selected runs: per-run CSV with sample counts, latency tails, resource costs, and UDP drops

serving_cpu_ms and node_cpu_ms are milliseconds per completed server request in the matching resource window. resource_window_successes can be fractional because counters are interpolated at the window boundaries. Latencies are milliseconds. frontend_rss_mib is peak resident memory within that window. repetition identifies the three paired repetitions.

workers,concurrency,endpoints,repetition,completed,errors,requests_s,output_tokens_s,request_latency_p50_ms,request_latency_p95_ms,request_latency_p99_ms,time_to_first_token_p50_ms,time_to_first_token_p95_ms,time_to_first_token_p99_ms,inter_token_latency_p50_ms,inter_token_latency_p95_ms,inter_token_latency_p99_ms,serving_cpu_ms,node_cpu_ms,frontend_rss_mib,frontend_fds,udp_drops,resource_window_start_ns,resource_window_end_ns,resource_window_successes
1,1,8,0,40301,0,672.311206,21513.958584,1.314816,1.501280,1.551775,0.702560,0.764863,0.811392,0.019755,0.025838,0.027447,3.001865,2.890298,148.187500,56,0,1789524970647513639,1789525016757260612,30999.634341
1,1,32,0,40267,0,671.745733,21495.863468,1.302560,1.488000,1.539936,0.700128,0.757868,0.797259,0.019445,0.025479,0.027123,2.979117,2.872005,146.562500,80,0,1789525037670960228,1789525083786712182,30998.911076
1,16,32,0,256448,0,4276.562195,136849.990231,3.602303,4.702996,5.274306,2.766943,3.888020,4.480336,0.025517,0.055019,0.067382,2.036449,1.997514,171.125000,187,0,1789525106113746602,1789525152238365358,197437.518299
1,16,8,0,259502,0,4326.298040,138441.537288,3.559071,4.642141,5.191230,2.633840,3.679807,4.239807,0.028940,0.057896,0.070608,2.038616,2.000395,165.750000,166,0,1789525174317828656,1789525220434865409,199735.280685
1,64,8,0,270900,0,4516.840764,144538.904450,13.887567,19.599431,22.278692,12.386603,18.139878,20.802150,0.009709,0.201083,0.278205,1.679423,1.648040,164.687500,277,38,1789525243894771257,1789525290011193244,208443.980915
1,64,32,0,274871,0,4582.152633,146628.884270,13.676829,19.590874,22.398548,11.885019,17.669115,20.372152,0.010770,0.219389,0.300558,1.671521,1.644911,170.562500,302,2383,1789525312966130233,1789525359078514759,211455.335348
8,1,32,0,39663,0,661.658387,21173.068373,1.285312,1.524736,1.578476,0.749215,0.833312,0.903898,0.017229,0.025110,0.026734,3.186790,3.101935,152.312500,108,0,1789525385117292616,1789525431228932918,30387.677396
8,1,8,0,38677,0,645.208886,20646.684361,1.299584,1.551136,1.605919,0.756992,0.837600,0.907872,0.017380,0.025709,0.027426,3.220085,3.126404,129.812500,84,0,1789525454562775261,1789525500677299696,29667.821223
8,16,8,0,247138,0,4121.144947,131876.638300,3.749264,4.958208,5.535793,3.158575,4.512517,5.096689,0.012764,0.046610,0.060100,2.836586,2.721312,158.562500,189,0,1789525521666971215,1789525567779095854,190248.120734
8,16,32,0,244968,0,4083.918142,130685.380533,3.794207,4.967295,5.542291,3.349759,4.619604,5.186207,0.008530,0.039648,0.053376,2.820904,2.704909,188.875000,216,0,1789525591318031893,1789525637414974224,188486.099893
8,64,32,0,250630,0,4178.710296,133718.729476,14.912157,22.231661,25.482072,13.045870,20.392311,23.848720,0.035061,0.191769,0.278920,2.466876,2.380057,219.750000,321,0,1789525661560566534,1789525707674020928,192608.081587
8,64,8,0,249972,0,4168.327930,133386.493761,14.984862,22.159806,25.346898,12.785951,20.067240,23.446861,0.046643,0.212636,0.296616,2.462665,2.384964,169.437500,298,1352,1789525732755701131,1789525778872724925,192300.736080
8,64,8,1,253245,0,4222.834455,135130.702564,14.698816,22.140304,25.937699,12.544800,20.055930,23.659683,0.045221,0.210811,0.302523,2.464152,2.381258,165.750000,297,4885,1789525922560536202,1789525968679523619,194855.611945
8,64,32,1,253326,0,4223.104168,135139.333377,14.740077,21.999684,25.336211,12.919038,20.173437,23.735292,0.033187,0.189772,0.270029,2.459981,2.368423,220.312500,320,0,1789525991697547465,1789526037808301911,194831.474849
8,16,32,1,250114,0,4169.777690,133432.886080,3.709952,4.854378,5.407658,3.244703,4.458294,5.008162,0.008884,0.041495,0.055708,2.802952,2.682367,206.562500,217,0,1789526061784590687,1789526107902390942,192493.403266
8,16,8,1,248481,0,4142.414403,132557.260891,3.736768,4.905343,5.459027,3.229184,4.533663,5.084434,0.010330,0.042712,0.056151,2.837454,2.722922,161.312500,191,0,1789526132994797351,1789526179114950946,191429.079378
8,1,8,1,39263,0,654.984956,20959.518594,1.297536,1.542655,1.598144,0.755392,0.833152,0.894801,0.017473,0.025484,0.027187,3.225118,3.116084,164.562500,84,0,1789526201096146197,1789526247187165877,30081.274166
8,1,32,1,39795,0,663.850242,21243.207739,1.280160,1.526889,1.580168,0.755807,0.843562,0.890279,0.016850,0.025040,0.026741,3.188375,3.074904,169.687500,108,0,1789526268780017007,1789526314889850208,30460.734749
1,64,32,1,274538,0,4577.825831,146490.426604,13.708306,19.293765,21.972770,12.307070,17.880617,20.505783,0.009221,0.191760,0.267103,1.659412,1.628712,184.625000,302,105,1789526336840429501,1789526382960702289,211347.537946
1,64,8,1,272025,0,4534.878021,145116.096681,13.860543,19.303199,21.888111,12.445120,17.915653,20.505857,0.009561,0.189069,0.264055,1.666420,1.633931,165.375000,275,20,1789526405931084367,1789526452041951087,209265.171030
1,16,8,1,259752,0,4331.652805,138612.889761,3.554431,4.646431,5.219216,2.631936,3.683118,4.256702,0.028815,0.058002,0.070938,2.019204,1.978417,165.437500,163,0,1789526476010036041,1789526522120914065,199976.520228
1,16,32,1,261023,0,4351.516771,139248.536661,3.540767,4.606080,5.160792,2.673311,3.711196,4.252056,0.026982,0.055362,0.067807,2.019060,1.977315,150.625000,186,0,1789526545092500040,1789526591206156032,200822.750462
1,1,32,1,39002,0,650.644496,20820.623863,1.273824,1.495069,1.549792,0.721824,0.791840,0.845886,0.017821,0.025076,0.026805,3.010175,2.888805,147.937500,80,0,1789526998193199566,1789527044309763637,30064.153997
1,1,8,1,38992,0,650.475336,20815.210744,1.306736,1.508750,1.561311,0.724448,0.822240,1.305579,0.018230,0.025413,0.027002,3.035208,2.916143,171.625000,56,0,1789527066217924158,1789527112343419093,30021.419966
1,1,8,2,37984,0,633.662498,20277.199945,1.319664,1.516922,1.567472,0.729088,0.789664,0.840188,0.019069,0.025469,0.027015,3.026477,2.983596,156.437500,56,0,1789527134822856736,1789527180945402012,29072.038292
1,1,32,2,40501,0,675.646467,21620.686957,1.312960,1.483168,1.533696,0.691872,0.751328,0.787072,0.020077,0.025542,0.027160,2.975070,2.843094,183.500000,80,0,1789527202854765442,1789527248979964297,31416.716741
1,16,32,2,257981,0,4300.807151,137625.828839,3.580607,4.673791,5.244589,2.764672,3.875806,4.469285,0.024809,0.054324,0.067249,2.022255,1.982738,169.562500,187,0,1789527270377336869,1789527316500389103,198556.619224
1,16,8,2,258024,0,4302.441356,137678.123389,3.581246,4.671870,5.242378,2.701312,3.758681,4.302959,0.027361,0.056237,0.069034,2.044957,2.007965,148.250000,167,0,1789527338468775932,1789527384584470228,198645.529631
1,64,8,2,266144,0,4437.824237,142010.375570,14.149773,19.937841,22.646613,12.544574,18.316874,20.997775,0.010073,0.209471,0.288597,1.701626,1.676853,154.062500,272,989,1789527406486319642,1789527452598044520,204953.179433
1,64,32,2,274944,0,4583.504132,146672.132236,13.686848,19.449773,22.140856,12.175792,17.986961,20.724008,0.009302,0.199603,0.270842,1.669362,1.636541,165.375000,296,0,1789527477240352523,1789527523356660803,211611.338672
8,1,32,2,38984,0,650.330330,20810.570564,1.303824,1.535872,1.596229,0.750080,0.826912,0.880086,0.017801,0.025363,0.027034,3.178618,3.111227,182.562500,108,0,1789527549414116300,1789527595527476906,29819.398335
8,1,8,2,40229,0,671.106516,21475.408523,1.278016,1.531091,1.593942,0.736256,0.814656,1.042342,0.017284,0.025689,0.027418,3.230697,3.109662,174.312500,84,0,1789527619113114235,1789527665224543473,30970.068348
8,16,8,2,245750,0,4096.869864,131099.835646,3.764352,5.044736,5.719106,3.206271,4.599008,5.267683,0.011700,0.046265,0.060583,2.824211,2.702484,188.000000,187,0,1789527686209400726,1789527732328565657,189126.237809
8,16,32,2,248120,0,4137.448090,132398.338865,3.729951,4.942338,5.550816,3.229855,4.522273,5.124303,0.009879,0.043277,0.057767,2.800715,2.684331,193.437500,221,0,1789527754471179452,1789527800591548422,191143.085422
8,64,32,2,250905,0,4183.217128,133862.948085,14.881435,22.272586,25.603769,12.976189,20.304303,23.940368,0.035182,0.199322,0.285460,2.479758,2.394273,219.812500,325,0,1789527825658066777,1789527871784134645,193036.548396
8,64,8,2,249199,0,4155.983807,132991.481831,15.071646,22.183645,25.257951,12.927231,20.240748,23.497567,0.044935,0.207177,0.292127,2.475827,2.383605,183.437500,304,1211,1789527896798410249,1789527942915525522,191616.682584

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Still present: the new %reason field logs a peer-controlled QUIC close reason without escaping. Embedded newlines or terminal escapes can forge physical worker-log records; record reason as a normal string field instead.
• Original discussion: Verified still present: fail_client_connection_bundle still logs the peer-influenced full QUIC close reason as %reason, which renders without string escaping in the worker warning. Record reason as a normal string field instead.
• Original discussion: Verified still present: the new full QUIC cause chain can include a peer-controlled CONNECTION_CLOSE reason, and %reason emits it unescaped in the worker warning. Newlines or ANSI escape sequences can forge log lines or alter terminal output; record reason as a normal string field instead.
• Original discussion: Verified still present: the new full cause chain can include a peer-controlled QUIC close reason, and %reason writes it unescaped to the worker log. Embedded newlines or terminal escapes can forge log records; record reason as a normal string field instead.

Questions for the author:

• Cargo.toml: For the supported QUIC response path, can you provide a baseline-versus-head measurement using this PR's exact dependency update and the shipped 5 ms batch interval? The supplied Tyche results show a higher short-response p99 but intentionally do not isolate the Quinn update or test the shipped configuration.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Still present: the new full Quinn cause chain can include a peer-controlled CONNECTION_CLOSE reason, and %reason records it without string escaping. Embedded newlines or terminal escapes can forge worker log records; record reason as a normal string field instead.
• Original discussion: The worker warning still logs the peer-influenced full QUIC close reason with %reason, which permits embedded newlines or terminal escapes to forge log records. Record reason as a normal string field.
• Original discussion: Verified still present: fail_client_connection_bundle still records the peer-influenced full QUIC close reason as %reason, which renders without string escaping in the worker warning. Record reason as a normal string field instead.
• Original discussion: Verified still present: the new full Quinn cause chain can include a peer-controlled CONNECTION_CLOSE reason, and %reason logs it without string escaping. Embedded newlines or terminal control sequences can forge worker log records; record reason as a normal string field instead.

Questions for the author:

• Cargo.toml: For the supported QUIC response path, please provide a baseline-versus-head measurement using this PR's exact Quinn update and the shipped 5 ms batch interval. The supplied comparisons use a zero batch interval and additional campaign changes, so they do not isolate the changed transport behavior or assess the observed short-response tail increase under the shipped configuration.

[jthomson04]

Review follow-up is complete in 1ec03db608: the worker log field now escapes peer-supplied control characters, and the description covers Quinn's own release changes. All 22 QUIC tests, Clippy (no warnings), formatting, diff checks, and the local log-escaping check passed.

The request for a baseline-versus-head benchmark with the shipped 5 ms batch interval is explicitly deferred for this PR. No new performance benchmark was run. The retained Tyche results use a zero batch interval and common campaign changes; they do not validate this exact PR with shipped settings or establish latency parity. The performance-validation gap remains open. TCP remains the default.

[dynamo-review-agent]

Previously reported defects still present:

• Original discussion: Verified still present: when accept_bi() returns an error, Quinn has already set close_reason, so the close_reason().is_none() guard prevents the frontend accept-loop warning from ever being emitted. Transport failures therefore remain silent at this point.