feat(runtime): add TLS support to NATS transport

[walkoss]

Tracking issue: #10809 — [CONTRIBUTION]: Add opt-in TLS and mTLS support to NATS and TCP transports.
Part of the TLS stack: #10921 (TCP response streams, merged) → #12533 (TCP request plane, merged) → this PR (NATS) → mTLS → operator injection.

What

Adds opt-in TLS for Dynamo's NATS transport (JetStream indexer recovery/replay + audit sink), continuing the TLS stack after the TCP response streams (#10921) and TCP request plane (#12533).

• New NATS_TLS_CA_CERT_PATH / NATS_TLS_INSECURE env vars, plus --nats-tls-ca-cert-path / --nats-tls-insecure CLI flags on all backends via DynamoRuntimeArgGroup.
• NATS_SERVER now accepts the tls:// scheme in addition to nats://. A tls:// URL without explicit TLS env vars lets async-nats handle TLS natively with system roots; a NATS_TLS_CA_CERT_PATH (or insecure mode) applies a custom rustls client config and requires TLS.
• Installs the ring crypto provider as the process default so async-nats doesn't panic on CryptoProvider lookup when it sees a tls:// URL (both ring and aws-lc-rs are compiled in transitively, so rustls 0.23 can't auto-detect).
• Docs: generalizes the TLS page (title TCP TLS → TLS, intro now covers all transports), adds a NATS TLS section, and clarifies the KV/ZMQ event-plane boundary.

Notes on cert handling

Dynamo is a NATS client here: it verifies the NATS server against NATS_TLS_CA_CERT_PATH and presents no client identity (with_no_client_auth). So the server-cert hot-reload from #12533 doesn't apply to this path — there's no served leaf cert to rotate. The CA trust anchor is loaded once (rotating the CA requires a restart, same as the TCP client). Client-identity certs (and their reload) arrive with the upcoming NATS/TCP mTLS PR.

Testing

cargo test -p dynamo-runtime --lib nats — 8 pass (2 ignored; they require a live NATS server). cargo fmt --check and cargo build -p dynamo-runtime clean.

Part of #10809.

Summary by CodeRabbit

• New Features

  • Added TLS encryption support for NATS connections using tls:// URLs.
  • Added options for custom CA certificates and disabling certificate verification when needed.
  • Added runtime and environment configuration for NATS TLS settings.

• Documentation

  • Expanded TLS guidance to cover NATS connections, TCP streams, supported URL schemes, configuration options, and encrypted event-plane paths.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

🌿 Fern Docs Preview: https://nvidia-preview-01a013b3-7bab-739e-beba-485dbdd85951.docs.buildwithfern.com/dynamo/dev

[devin-ai-integration]

Devin Review found 2 potential issues.

View 1 additional finding in Devin Review.

[coderabbitai]

Walkthrough

This change adds NATS TLS runtime options, environment constants, connection handling, validation, tests, and documentation. It supports custom CA certificates, insecure verification mode, and tls:// server URLs.

Changes

NATS TLS support

Layer / File(s)	Summary
TLS configuration contracts components/src/dynamo/common/configuration/groups/runtime_args.py, lib/runtime/src/config/environment_names.rs	Runtime arguments and public environment constants define the NATS CA certificate path and certificate-verification bypass settings.
NATS TLS connection handling lib/runtime/src/transports/nats.rs	NATS client options load TLS settings, accept nats:// and tls:// URLs, apply custom TLS configuration, and validate default and environment-backed behavior with tests.
TLS behavior documentation docs/fern/pages/reference/components/tls-configuration.mdx	The reference documentation describes NATS TLS coverage, environment variables, URL schemes, and CLI options.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the primary change: adding TLS support to the NATS transport.
Description check	✅ Passed	The description explains the changes, implementation details, testing, and tracking issue, but it does not use all template headings.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/fern/pages/reference/components/tls-configuration.mdx`:
- Around line 147-177: Update the NATS documentation near the NATS TLS
configuration and the Encrypted paths table to explicitly include inference
request traffic when NATS is used as the request plane. Expand the NATS
description and its table row while retaining the existing JetStream recovery
and audit-log coverage.

In `@lib/runtime/src/transports/nats.rs`:
- Around line 294-301: The NATS configuration validation only checks the server
URL scheme and allows TLS options with nats://. Update validate_nats_server and
the pre-connection validation flow to reject nats:// when either
tls_ca_cert_path or tls_insecure is configured, returning a startup error; add
tests covering both invalid combinations.

🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 44d54a53-4745-42a9-a9f9-dd90210c2285

📥 Commits

Reviewing files that changed from the base of the PR and between 9f616ae and 392544b.

📒 Files selected for processing (4)

• components/src/dynamo/common/configuration/groups/runtime_args.py
• docs/fern/pages/reference/components/tls-configuration.mdx
• lib/runtime/src/config/environment_names.rs
• lib/runtime/src/transports/nats.rs

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 392544b33b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 50.87% (-7.84%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 33549bd | Docs | View more details | Give us feedback!}

[walkoss]

Update: addressed all automated-review feedback (Devin, CodeRabbit, Codex, dynamo-review-agent).

Latest revision folds these into the existing commits; per-thread replies have specifics.

Correctness / behavior

• Crypto provider not installed for plaintext NATS (Devin): the ring install_default() is now gated on custom_tls || tls://, so a plain nats:// connection can't clobber another component's provider choice (e.g. the HTTPS frontend's aws-lc-rs).
• Fail closed on mismatched config (CodeRabbit): custom TLS (NATS_TLS_CA_CERT_PATH / NATS_TLS_INSECURE) with a non-tls:// URL now errors at startup instead of silently forcing TLS onto nats://. Added test_connect_rejects_custom_tls_with_nats_url.
• Frontend CLI parity (Codex, P1): added --nats-tls-ca-cert-path / --nats-tls-insecure to FrontendConfig/FrontendArgGroup with env propagation, matching the --tcp-tls-* pattern (they were previously rejected by the frontend's separate parser).
• --no-nats-tls-insecure override (dynamo-review-agent): the frontend now clears NATS_TLS_INSECURE when false, so the negated flag overrides an inherited =1 before the runtime reads it.

Docs

• Tagged the NATS CLI code fence text (Devin, Codex).
• Documented NATS inference-request traffic under --request-plane nats in the NATS section and encrypted-paths table (CodeRabbit).
• Noted the frontend also accepts --nats-tls-*.

Verification: cargo fmt --check, cargo clippy -p dynamo-runtime --all-targets -- -D warnings (clean), NATS unit tests (9 pass), py_compile on the frontend changes.

[PeaBrane]

/ok to test 33549bd