[yifjiang]

Summary

Explicit native transport registration and complete rank descriptors; failed registration preserves bank owners. Requires the compatible KVCR stack.

Review slice D24 of #16748; part of #16747 and #16746. This draft targets codex/kv-reuse-d23-retained-gpu-policy for a complete incremental review diff of 400 additions plus deletions, including tests, documentation and extraction glue.

Extraction details:

• Whole endpoint/pin adapter/drain/resilience/shutdown methods land atomically, avoiding a temporarily unsafe partial native endpoint.
• Registration unit test uses a faithful geometry spec before D26 public profile admission.

Reuse remains experimental and default off. Retained inventory is advisory, complete compatible rank parts must precede readiness, and native operations retain ownership until quiescence. Supported GPU activation also requires the companion KVCR/native slices and the declared deployment profile.

Validation

All configured pre-commit hooks passed for this exact slice on allocated SLURM. Stack-level CPU regression coverage is summarized on D31; each draft still requires its own applicable CI in the Checks tab. Historical combined-head and hardware results retain their original source scope.

[devin-ai-integration]

❌ Dynamo PR CI failed — run 41803208999 (attempt 1) on ab1e14f902

Gate checks: ❌ backend-status-check · ✅ deploy-status-check · ❌ dynamo-status-check

Framework	1-GPU amd64
SGLang	❌ 1

Other	Jobs
dynamo-runtime	❌ 2 ✅ 5

Failure details

3 jobs failed: on both amd64 and arm64, the dynamo-runtime CPU tests fail the same 2 unit tests (the shared router-settings fixture uses settings kv_router_args doesn't recognize, and the reject_media_status_code default is wrong). On sglang-runtime 1-GPU amd64, the OpenResponses compliance "System Prompt" case returns incomplete (max_output_tokens). No jobs were cancelled.

❌ dynamo-runtime / test / sequential cuda13.0, amd64: 2 unit tests fail (unknown router settings; reject_media_status_code default)

Job: dynamo-runtime / test / sequential cuda13.0, amd64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126494578654 -R ai-dynamo/dynamo --log-failed

kv_router_args.py:234: in _router_settings
    raise ValueError(f"unknown router setting(s): {sorted(unknown)}")
test_frontend_parser_builder.py:62: in test_no_registered_env_var_is_read_from_os_environ
    assert vars(parser.parse_args([])) == defaults, sentinel
E     Differing items:
E     {'reject_media_status_code': 0} != {'reject_media_status_code': 415}
FAILED components/src/dynamo/common/tests/configuration/test_kv_router_args.py::test_shared_router_settings_fixture_flows_to_binding_kwargs - ValueError: unknown router setting(s): ['enable_decode_to_prefill_gpu_reuse', 'enable_decode_to_prefill_reuse', 'minimum_estimated_ttft_saving_ms', 'minimum_transfer_prefix_blocks', 'transfer_source_estimated_block_us', 'transfer_source_gpu_estimated_block_us', 'transfer_source_lookup_budget_ms', 'transfer_source_prefill_token_us']
FAILED components/src/dynamo/frontend/tests/test_frontend_parser_builder.py::test_no_registered_env_var_is_read_from_os_environ - AssertionError: 0
== 2 failed, 7784 passed, 170 skipped, 2921 deselected ==

Two tests in the pre_merge_cpu stage fail deterministically (each also failed its 3 auto-retries). test_shared_router_settings_fixture_flows_to_binding_kwargs: KvRouterArgs._router_settings rejects 8 settings from the shared router-settings fixture (enable_decode_to_prefill_reuse, transfer_source_*, minimum_*, ...) as unknown. test_no_registered_env_var_is_read_from_os_environ: with env reads replaced by a sentinel, the frontend parser default for reject_media_status_code comes out as 0 instead of 415.

❌ dynamo-runtime / test / sequential cuda13.0, arm64: 2 unit tests fail (unknown router settings; reject_media_status_code default)

Job: dynamo-runtime / test / sequential cuda13.0, arm64 · Failed step: Run CPU-only tests (parallelized) · Logs: gh run view --job 126494578702 -R ai-dynamo/dynamo --log-failed

kv_router_args.py:234: in _router_settings
    raise ValueError(f"unknown router setting(s): {sorted(unknown)}")
test_frontend_parser_builder.py:62: in test_no_registered_env_var_is_read_from_os_environ
    assert vars(parser.parse_args([])) == defaults, sentinel
E     Differing items:
E     {'reject_media_status_code': 0} != {'reject_media_status_code': 415}
FAILED components/src/dynamo/common/tests/configuration/test_kv_router_args.py::test_shared_router_settings_fixture_flows_to_binding_kwargs - ValueError: unknown router setting(s): ['enable_decode_to_prefill_gpu_reuse', 'enable_decode_to_prefill_reuse', 'minimum_estimated_ttft_saving_ms', 'minimum_transfer_prefix_blocks', 'transfer_source_estimated_block_us', 'transfer_source_gpu_estimated_block_us', 'transfer_source_lookup_budget_ms', 'transfer_source_prefill_token_us']
FAILED components/src/dynamo/frontend/tests/test_frontend_parser_builder.py::test_no_registered_env_var_is_read_from_os_environ - AssertionError: 0
== 2 failed, 7784 passed, 170 skipped, 2921 deselected ==

Two tests in the pre_merge_cpu stage fail deterministically (each also failed its 3 auto-retries). test_shared_router_settings_fixture_flows_to_binding_kwargs: KvRouterArgs._router_settings rejects 8 settings from the shared router-settings fixture (enable_decode_to_prefill_reuse, transfer_source_*, minimum_*, ...) as unknown. test_no_registered_env_var_is_read_from_os_environ: with env reads replaced by a sentinel, the frontend parser default for reject_media_status_code comes out as 0 instead of 415.

❌ sglang-runtime / Test cuda13.0, amd64: OpenResponses compliance "System Prompt" got status incomplete (max_output_tokens)

Job: sglang-runtime / Test cuda13.0, amd64 · Failed step: Run GPU tests (parallel) · Logs: gh run view --job 126498088469 -R ai-dynamo/dynamo --log-failed

[w0] E   Failed: OpenResponses compliance suite failed (exit=1).
[w0] E   Model: Qwen/Qwen3-VL-2B-Instruct
[w0] E   ✗ System Prompt (22087ms)
[w0] E     ✗ Expected status "completed" but got "incomplete"
[w0] E       "status": "incomplete",
[w0] E       "incomplete_details": {
[w0] E         "reason": "max_output_tokens"
[w0] E   Results: 5 passed, 1 failed, 6 total
[w0] FAILED tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance - Failed: OpenResponses compliance suite failed (exit=1).

test_frontend_api_surface_compliance fails (including its 3 auto-retries) because 1 of the 6 OpenResponses compliance cases fails. In the "System Prompt" case (pirate system prompt + "Say hello."), the response stops with status: incomplete and incomplete_details.reason: max_output_tokens, so it hits the output-token limit instead of completing. The other 5 cases pass. The run stopped after this first failure (stopping after 1 failures).

For agents

{"pr": 18042, "run_id": 41803208999, "run_attempt": 1, "head_sha": "ab1e14f902418a8c2d99ff97b17b54dc9558fc7f", "failures": [{"job": "dynamo-runtime / test / sequential cuda13.0, amd64", "job_id": 126494578654, "failed_step": "Run CPU-only tests (parallelized)", "signature": "ValueError: unknown router setting(s); reject_media_status_code default 0 != 415", "tests": ["components/src/dynamo/common/tests/configuration/test_kv_router_args.py::test_shared_router_settings_fixture_flows_to_binding_kwargs", "components/src/dynamo/frontend/tests/test_frontend_parser_builder.py::test_no_registered_env_var_is_read_from_os_environ"], "log_cmd": "gh run view --job 126494578654 -R ai-dynamo/dynamo --log-failed"}, {"job": "dynamo-runtime / test / sequential cuda13.0, arm64", "job_id": 126494578702, "failed_step": "Run CPU-only tests (parallelized)", "signature": "ValueError: unknown router setting(s); reject_media_status_code default 0 != 415", "tests": ["components/src/dynamo/common/tests/configuration/test_kv_router_args.py::test_shared_router_settings_fixture_flows_to_binding_kwargs", "components/src/dynamo/frontend/tests/test_frontend_parser_builder.py::test_no_registered_env_var_is_read_from_os_environ"], "log_cmd": "gh run view --job 126494578702 -R ai-dynamo/dynamo --log-failed"}, {"job": "sglang-runtime / Test cuda13.0, amd64", "job_id": 126498088469, "failed_step": "Run GPU tests (parallel)", "signature": "OpenResponses compliance System Prompt: status incomplete (max_output_tokens)", "tests": ["tests/frontend/test_frontend_api_surface_compliance.py::test_frontend_api_surface_compliance"], "log_cmd": "gh run view --job 126498088469 -R ai-dynamo/dynamo --log-failed"}]}

_{Posted automatically by Devin for run 41803208999. Updated on every full-CI run of this PR.}

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[yifjiang]

/ok to test 211b7c1

[yifjiang]

/ok to test 905e701