feat(runtime): add TLS support to TCP request plane by walkoss · Pull Request #10921 · ai-dynamo/dynamo

walkoss · 2026-06-24T11:41:42Z

Overview

Adds opt-in TLS encryption to the TCP request plane (frontend ↔ worker) using rustls with the ring crypto provider. When no TLS env vars are set the transport behaves exactly as before — fully backward compatible.

Details

New shared helper (lib/runtime/src/tls_utils.rs):

server_tls_config(cert, key) — builds a rustls::ServerConfig
client_tls_config(ca, insecure) — builds a rustls::ClientConfig; emits a prominent warning when insecure=true

TCP server (tcp/server.rs):

Reads DYN_TCP_TLS_CERT_PATH + DYN_TCP_TLS_KEY_PATH at startup to optionally build a TlsAcceptor
TLS handshake is spawned per-connection (non-blocking accept loop), bounded by a 10s timeout
BoxRead/BoxWrite (Box<dyn AsyncRead/AsyncWrite + Unpin + Send>) unify TLS and plaintext streams through all inner functions with no branching after accept
CallHomeHandshake read and response-stream prologue read are also bounded by 10s timeouts

TCP client (tcp/client.rs):

OnceLock<Option<TlsConnector>> — connector built once from env on first connection, Arc-cloned per connection
connect_and_split() upgrades to TLS when configured, with a 10s handshake timeout
IPv6-safe SNI parsing with DYN_TCP_TLS_SERVER_NAME override for IP-addressed servers

Environment variables (all optional, plaintext when unset):

Variable	Side	Purpose
`DYN_TCP_TLS_CERT_PATH`	Server	PEM server certificate
`DYN_TCP_TLS_KEY_PATH`	Server	PEM server private key
`DYN_TCP_TLS_CA_CERT_PATH`	Client	CA cert to verify server
`DYN_TCP_TLS_INSECURE`	Client	Skip cert verification (dev only)
`DYN_TCP_TLS_SERVER_NAME`	Client	SNI override for IP-addressed servers

Where should the reviewer start?

lib/runtime/src/tls_utils.rs — new file, self-contained rustls helpers
lib/runtime/src/pipeline/network/tcp/server.rs — build_tls_acceptor() and the accept loop spawn
lib/runtime/src/pipeline/network/tcp/client.rs — build_tls_connector_from_env() and connect_and_split()
lib/runtime/src/config/environment_names.rs — the 5 new constants under tcp_response_stream::tls

Related Issues

Relates to [CONTRIBUTION]: Add opt-in TLS and mTLS support to NATS and TCP transports #10809

Summary by CodeRabbit

New Features
- Added optional TLS support for runtime TCP connections, including both client and server modes.
- Added environment-based configuration for certificates, keys, CA roots, insecure verification, and server-name overrides.
- Improved connection handling so encrypted and unencrypted streams behave consistently.
Bug Fixes
- Added stricter timeouts for connection setup and stream handshakes to avoid indefinite waits.

copy-pr-bot · 2026-06-24T11:41:46Z

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

github-actions · 2026-06-24T11:41:50Z

🌿 Fern Docs Preview: https://nvidia-preview-019fca6e-f636-75a4-a154-b041d2a03dc9.docs.buildwithfern.com/dynamo/dev

datadog-official · 2026-06-24T11:43:32Z

Pipelines

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 43.66% (-5.50%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 5277065 | Docs | Datadog PR Page | Give us feedback!}

coderabbitai · 2026-06-24T11:49:28Z

Walkthrough

Adds optional TLS support to the Dynamo runtime's TCP networking layer. A new tls_utils module provides server_tls_config and client_tls_config builders backed by rustls. Both TcpStreamServer and TcpClient read TLS configuration from environment variables, perform handshakes with 10-second timeouts, and route streams through boxed AsyncRead/AsyncWrite abstractions so the rest of the pipeline handles plaintext and TLS uniformly.

Changes

TCP TLS Support

Layer / File(s)	Summary
TLS utility module and dependencies `lib/runtime/Cargo.toml`, `lib/runtime/src/lib.rs`, `lib/runtime/src/tls_utils.rs`	Adds `tokio-rustls`, `rustls`, and `rustls-pemfile` as dependencies; registers `tls_utils` as a public module; implements `server_tls_config` (cert/key PEM loading, ring provider, no client auth) and `client_tls_config` (optional CA cert, insecure mode via `NoVerifier`), plus the private `NoVerifier` `ServerCertVerifier`.
TLS environment variable constants `lib/runtime/src/config/environment_names.rs`	Adds the `tcp_response_stream::tls` submodule with five public env var constants (`DYN_TCP_TLS_CERT_PATH`, `DYN_TCP_TLS_KEY_PATH`, `DYN_TCP_TLS_CA_CERT_PATH`, `DYN_TCP_TLS_INSECURE`, `DYN_TCP_TLS_SERVER_NAME`) and extends the duplicate-name unit test to cover them.
TCP server TLS acceptor and boxed I/O `lib/runtime/src/pipeline/network/tcp/server.rs`	Introduces `BoxRead`/`BoxWrite` type aliases; adds `build_tls_acceptor` driven by env vars; extends `start` and `tcp_listener` to accept an optional `Arc<TlsAcceptor>`; per-connection handling spawns a task with a 10s TLS handshake timeout before boxing stream halves; all internal stream handlers (`process_stream`, `process_request_stream`, `process_response_stream`, `request_stream_send_handler`, `network_receive_handler`, `network_send_handler`) updated to use boxed framed I/O; `CallHomeHandshake` and response prologue reads bounded by 10s timeouts.
TCP client TLS connector and connect_and_split `lib/runtime/src/pipeline/network/tcp/client.rs`	Adds `BoxRead`/`BoxWrite` type aliases; introduces static `TCP_TLS_CONNECTOR` `OnceCell` with env-var-driven builder and `tls_server_name` helper; adds `connect_and_split` that retries TCP connect, optionally performs a 10s-timeout TLS handshake, and returns boxed halves; `create_response_stream` and `create_request_stream` switch to `connect_and_split`; `wait_for_connection_tasks`, `wait_for_server_shutdown`, `handle_reader`, and `handle_writer` all updated to use boxed framed I/O.
Client test harness boxing updates `lib/runtime/src/pipeline/network/tcp/client.rs`	Updates all test harnesses (`WriterHarness`, `ReaderHarness`, `RequestReaderHarness`) and their corresponding builder functions to box split TCP halves into `FramedRead<BoxRead, ...>` / `FramedWrite<BoxWrite, ...>`, aligning tests with the new boxed I/O abstraction.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly summarizes the main change: adding TLS support to the TCP request plane.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The PR description covers overview, details, reviewer start points, and a related issue, matching the template closely enough.

_{Comment @coderabbitai help to get the list of available commands.}

walkoss · 2026-06-24T12:21:22Z

@grahamking — this is PR 1 from the TLS contribution request (#10809) you approved. Scope is TCP request-plane TLS only (no mTLS, no NATS). Happy to address any feedback!

grahamking · 2026-06-24T15:07:41Z

@walkoss Great! Can you start with the red tests. Look like clippy, maybe some others. Then I'll trigger the bigger unit tests, and schedule a review.

Have you done a second round of agent work, where you ask it to simplify and remove unnecessary tests? Can usually slim down an agentic PR a fair bit like that.

walkoss · 2026-06-25T12:18:18Z

@grahamking thank you, all checks are green now.
I rebased onto the latest main which surfaced some Cargo.lock drift in the sub-workspaces (lib/bindings/python, lib/runtime/examples, lib/bindings/kvbm) - these needed syncing independently of our changes.
I also simplified the tests as you suggested, removing redundant cases and merging related assertions. Does the test coverage look good to you?

grahamking · 2026-06-25T15:41:56Z

/ok to test 69d75de

Encrypt frontend ↔ worker TCP connections using rustls (ring provider). TLS is opt-in via environment variables; existing deployments require no changes. New env vars (all optional): - DYN_TCP_TLS_CERT_PATH / DYN_TCP_TLS_KEY_PATH — server-side TLS - DYN_TCP_TLS_CA_CERT_PATH — client CA verification - DYN_TCP_TLS_INSECURE — skip cert verify (dev only) - DYN_TCP_TLS_SERVER_NAME — SNI override for IP addresses Key design decisions: - BoxRead/BoxWrite unify TLS and plaintext streams through all inner functions with zero branching after accept/connect - TLS acceptor spawned per-connection so the accept loop is never blocked; handshake timeout is 10s on both server and client - ClientConfig cached via OnceLock — built once, Arc-cloned per connection - Warn at startup when server/client TLS env vars are mismatched - Empty CA PEM detected at build time rather than failing at handshake Signed-off-by: Walid El Bouchikhi <walid.elbouchikhi@datadoghq.com>

Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>

ziqifan617

only sign off for dynamo-kv-memory-codeowners, since the related part is kvbm cargo.lock

nv-anants

toml changes look good

Per review feedback from nealvaidya: TLS configuration is not tied to Kubernetes or the operator, so it belongs alongside the other runtime reference docs instead of under the Kubernetes Operator section. Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>

The request-plane page documents all other DYN_TCP_* environment variables, so add a pointer to the TLS reference for the encryption options as well (per review feedback). Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>

nv-tusharma · 2026-08-03T22:09:35Z

/ok to test 78ba919

dynamo-ops · 2026-08-04T01:39:15Z

/ok to test 5277065

dyn-3691-extract-shared-target-pid-cuda-customstorage-operation-layer * 'main' of https://github.com/ai-dynamo/dynamo: (50 commits) docs(cli): correct removed vLLM prefill-worker flag reference (#12581) docs(operator): reserve webhook Ignore for emergencies (#12563) ci(docs): make previews and checks match what actually publishes (#12339) refactor(vllm): organize custom encoder modules (#12416) feat(llm): Select reasoning output field via env var (#11464) feat(runtime): add TLS support to TCP request plane (#10921) fix: convert conditional disagg sglang warning to httperror 400 (#12578) feat(operator): add runtime feature gates (#12421) refactor(runtime): extract PushRouter transport seam behind StreamingDispatch trait (#12447) feat(replay): add deterministic canonical offline reports (#12363) build: bump ModelExpress to 0.5.0(OPS-7978) (#12455) fix(mocker): use logical KV tokens for decode timing (#12583) fix(examples): update Triton example for CUDA 13 + fix libdcgm copy (DYN-3697) (#12577) refactor(operator): implement composition-first DGD reconciliation (#12283) feat(frontend): add basetenkenizer backend (#12376) fix(profiler): configure rapid mocker without planner (#12573) docs(vllm): correct worker-role flags and document --kv-transfer-config (#12568) ci: add Kubernetes deploy test to nightly (#12090) fix(container): reuse pinned protoc in runtime image (#12535) feat(self-host): flip DYN_SELF_HOST_METADATA default to ON (gh-8749) (#11417) ... Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

walkoss requested a review from a team June 24, 2026 11:41

pull-request-size Bot added the size/XL label Jun 24, 2026

walkoss temporarily deployed to external_collaborator June 24, 2026 11:41 — with GitHub Actions Inactive

github-actions Bot added external-contribution Pull request is from an external contributor feat labels Jun 24, 2026

This comment was marked as resolved.

Sign in to view

walkoss temporarily deployed to external_collaborator June 24, 2026 11:53 — with GitHub Actions Inactive

walkoss temporarily deployed to external_collaborator June 24, 2026 12:00 — with GitHub Actions Inactive

walkoss temporarily deployed to external_collaborator June 24, 2026 12:03 — with GitHub Actions Inactive

walkoss temporarily deployed to external_collaborator June 25, 2026 08:05 — with GitHub Actions Inactive

walkoss force-pushed the walid/tcp-tls branch from 506e0f0 to 4627de5 Compare June 25, 2026 08:12

walkoss temporarily deployed to external_collaborator June 25, 2026 08:12 — with GitHub Actions Inactive

walkoss force-pushed the walid/tcp-tls branch from 4627de5 to c0f8bcf Compare June 25, 2026 08:18

walkoss temporarily deployed to external_collaborator June 25, 2026 08:19 — with GitHub Actions Inactive

walkoss force-pushed the walid/tcp-tls branch from c0f8bcf to 69d75de Compare June 25, 2026 08:30

walkoss temporarily deployed to external_collaborator June 25, 2026 08:30 — with GitHub Actions Inactive

copy-pr-bot Bot temporarily deployed to GITLAB June 25, 2026 15:42 Inactive

walkoss force-pushed the walid/tcp-tls branch from 69d75de to fa68b7e Compare June 25, 2026 16:48

walkoss temporarily deployed to external_collaborator June 25, 2026 16:48 — with GitHub Actions Inactive

walkoss temporarily deployed to external_collaborator August 1, 2026 11:15 — with GitHub Actions Inactive

pull-request-size Bot added size/XXL and removed size/XL labels Aug 1, 2026

walkoss temporarily deployed to external_collaborator August 1, 2026 11:21 — with GitHub Actions Inactive

Merge upstream/main into walid/tcp-tls

3c42432

Signed-off-by: Walid <walid.elbouchikhi@datadoghq.com>

walkoss force-pushed the walid/tcp-tls branch from 5f91166 to 3c42432 Compare August 1, 2026 11:37

walkoss temporarily deployed to external_collaborator August 1, 2026 11:37 — with GitHub Actions Inactive

walkoss mentioned this pull request Aug 1, 2026

feat(runtime): encrypt TCP request dispatch (frontend → worker) #12533

Merged

ziqifan617 approved these changes Aug 3, 2026

View reviewed changes

nealvaidya reviewed Aug 3, 2026

View reviewed changes

Comment thread docs/fern/index.yml Outdated

nv-anants approved these changes Aug 3, 2026

View reviewed changes

walkoss temporarily deployed to external_collaborator August 3, 2026 20:16 — with GitHub Actions Inactive

walkoss requested a review from nealvaidya August 3, 2026 20:34

nealvaidya approved these changes Aug 3, 2026

View reviewed changes

walkoss temporarily deployed to external_collaborator August 3, 2026 21:18 — with GitHub Actions Inactive

Merge branch 'main' into walid/tcp-tls

5277065

GuanLuo temporarily deployed to external_collaborator August 4, 2026 01:38 — with GitHub Actions Inactive

athreesh merged commit c3f0f71 into ai-dynamo:main Aug 4, 2026
119 of 121 checks passed

walkoss mentioned this pull request Aug 4, 2026

[CONTRIBUTION]: Add opt-in TLS and mTLS support to NATS and TCP transports #10809

Open

dagil-nvidia mentioned this pull request Aug 10, 2026

docs: backport main docs tree and docs CI to release/1.4.0 #12950

Merged

walkoss mentioned this pull request Aug 12, 2026

feat(runtime): add TLS support to NATS transport #13096

Merged

This was referenced Aug 13, 2026

fix(runtime): bound TCP accept failures under file-descriptor pressure #13146

Merged

fix(runtime): bound TCP accept failures under file-descriptor pressure #12282

Closed

This was referenced Aug 19, 2026

feat(runtime): add mutual TLS (mTLS) support for TCP and NATS transports #13528

Merged

feat(operator): add TLS/mTLS auto-injection via InfrastructureConfiguration #13689

Merged

Uh oh!

Conversation

walkoss commented Jun 24, 2026 • edited by sttts Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Details

Where should the reviewer start?

Related Issues

Summary by CodeRabbit

Uh oh!

copy-pr-bot Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

datadog-official Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

coderabbitai Bot commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Estimated code review effort

Uh oh!

This comment was marked as resolved.

Uh oh!

walkoss commented Jun 24, 2026

Uh oh!

grahamking commented Jun 24, 2026

Uh oh!

walkoss commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grahamking commented Jun 25, 2026

Uh oh!

ziqifan617 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

nv-anants left a comment

Choose a reason for hiding this comment

Uh oh!

nv-tusharma commented Aug 3, 2026

Uh oh!

dynamo-ops commented Aug 4, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

10 participants

walkoss commented Jun 24, 2026 •

edited by sttts

Loading

github-actions Bot commented Jun 24, 2026 •

edited

Loading

datadog-official Bot commented Jun 24, 2026 •

edited

Loading

coderabbitai Bot commented Jun 24, 2026 •

edited

Loading

walkoss commented Jun 25, 2026 •

edited

Loading