[yueming-yuan]

Co-authored with @yushengsu-thu.

Add the Qwen3-30B-A3B Tinker gateway launcher and an SDK marker-memorization client. The default recipe uses one eight-GPU node: four training GPUs (TP2/EP4), four sampling GPUs, and four adapter slots. The client exercises one adapter or concurrent tenant isolation through --mode single / --mode multi.

The launcher derives checkpoint storage from output_dir, with a --save-dir override, and creates its run ID per invocation. Both prepare and serve have recorded command snapshots. The SDK example uses the configured adapter layout, including unembedding; HF target detection and LoRA conversion recognize lm_head.

The gateway suite covers ordering, deduplication, batch/barrier planning, terminal-stream isolation, checkpoint ownership, lease cleanup, JSON/protobuf contracts, HTTP errors, and runtime translation. Existing sampler tests now exercise disk publication, snapshot resolution independently of the training lease, and training continuing after a sampling failure. Remove the obsolete sampler-push readiness test and backend fake method. No new test files are added.

The README and generated docs explain the shared checkpoint root, frozen base loading, immutable sampler directories, and on-demand engine loading. A sampler save does not contact inference engines or change pending gradients. Checkpoint load/save/export execution failures, including filesystem errors, terminate the shared trainer cell and server. Existing IO and gateway tests now expect propagation and dispatcher exit; obsolete continue-after-IO-failure cases are removed alongside a parameterized regression for dispatcher cancellation and background failure. Automatic training recovery/replay remains outside the current implementation.

The fixture now stops the service through run() instead of managing its private task collections. Checkpoint assertions use core/utils.py; the shutdown regression verifies that in-flight model creation and sampling do not outlive the dispatcher.

The README names the actual serve_qwen3_30b_a3b_tinker.py launcher and documents versioned checkpoint publication. Its docs mirror is regenerated. The backend fake uses the shared writer, and sampler assertions call the extracted utility. The checkpoint suite covers a process exiting before or after link replacement and a subsequent retry.

Stacked on #2845.

Validation:

• pre-commit run --all-files and incremental git diff --check passed on every updated component head. All previous heads remain ancestors; the final Unify HF snapshot and checkpoint writing #3198 tree exactly matches the reviewed implementation.
• Process-exit probes at both sides of publication lost the public path before the fix; afterward it resolves to the complete old or new version, and a subsequent save succeeds. A parameterized regression is included in the existing [multi-lora v2 7/7] Tinker gateway example and tests #2849 checkpoint test file.
• Sampler-to-training load probes with and without optimizer state now return a user failure before any backend load, while another tenant continues training. Repeated SAVE → LOAD verifies distinct retained versions and loading from a pinned version.
• Focused checkpoint continuation passed on 2 H200 GPUs (DP=2, TP/PP/EP=1), using real Megatron DDP, MultiLoRA layers, SlotOptimizer and distributed checkpoint IO. After three Adam steps, slot 0 was saved and restored into a fresh optimizer on slot 1; weights, FP32 masters, Adam moments, parameter-group metadata and step counters matched with zero tolerance before and after the next identical-gradient update. Rounding masters to BF16 and restarting from weights alone were both detected by negative controls.
• The enabled regression is tests/e2e/lora/test_slot_checkpoint_resume.py (Hopper, LoRA, 2-GPU stage). The tested final tree is Unify HF snapshot and checkpoint writing #3198 fd63c7ec8c41c2e0a61d273f0cd5a3504275ada7, with Megatron 73b54618f7e58e0f25f619bcfecbe2640765475a, Bridge 40b930897717941cfe2bd9806f417e28bf1bfa65, PyTorch 2.13.0+cu130 and Transformer Engine 2.17.0. Checkpoints were on GPFS. pre-commit run --all-files passed.
• This isolates checkpoint/Adam continuation with identical post-reduction gradients; it does not validate forward/backward numerics, topology changes, MoE/TP/PP layouts, server restart, or failure recovery. Fresh hosted CPU/NVIDIA CI is pending; broad GPU selection remains on Unify HF snapshot and checkpoint writing #3198. The Tinker cookbook and multi-tenant acceptance tests are now enabled under run-ci-multi-lora; their first GPU validation is pending.
• Launchers and command snapshots are unchanged by these fixes. The prior local launcher gate has not passed (362 passed, 30 failed, 122 errors), including macOS/Linux assumptions and existing launcher mismatches.

ci-megatron-pr: 73b54618f7e58e0f25f619bcfecbe2640765475a
ci-sglang-pr: 02b5e123c490c988d21526d0dd7837fefdd0e1da

Router dependency: install the router wheel and standalone binary from the existing cu130-torch213-x86_64 / cu130-torch213-aarch64 rolling releases in yueming-yuan/miles-wheels, using the normal /tmp/wheels installation path. Both architectures were rebuilt on CPU devboxes from radixark/sgl-router-for-miles@b6342a3515e259bbd1eb4db728035a4efbe4fdf9 and verified to forward return_indexer_topk and lora_backfill_paths. No commit-specific router release is required. The existing pr-2849 Docker image has not been rebuilt by this change.

Tinker GPU CI: add tests/e2e/lora/test_tinker_multi_tenant_gateway.py and enable the cookbook gateway test. Both carry the multi-lora domain, selected by the repository label run-ci-multi-lora, on 8 H200 GPUs. They share Qwen3-4B gateway setup (4 training GPUs, TP2/DP2; 4 sampling GPUs; rank-8 adapters). The multi-tenant client trains four distinct markers concurrently, exports each adapter, and checks loss decrease and its own sampled marker.

Validation: the real CI selector selects exactly both enabled tests for run-ci-multi-lora; mocked entrypoint checks confirm both client commands and launcher cleanup. Full pre-commit passed. GPU convergence, cookbook execution, and end-to-end tenant isolation have not yet been validated.

ci-image-tag: pr-2849

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[yueming-yuan]

/rerun-test tests/e2e/lora/test_tinker_multi_tenant_gateway.py

[yueming-yuan]

/rerun-test tests/e2e/lora/test_tinker_cookbook_gateway.py

[yueming-yuan]

/rerun-test tests/e2e/megatron/test_qwen3_5_35b_a3b_lora_ci.py

[yueming-yuan]

/rerun-test tests/e2e/megatron/model_scripts/test_nemotron_3_ultra_4layer_ci.py

[github-actions]

❌ tests/e2e/megatron/model_scripts/test_nemotron_3_ultra_4layer_ci.py failed in 48s — workflow run

The run never started: resolving the test file's execution plan failed.

[github-actions]

❌ tests/e2e/lora/test_tinker_multi_tenant_gateway.py failed in 39s — workflow run

The run never started: resolving the test file's execution plan failed.

[github-actions]

❌ tests/e2e/megatron/test_qwen3_5_35b_a3b_lora_ci.py failed in 40s — workflow run

The run never started: resolving the test file's execution plan failed.

[github-actions]

❌ tests/e2e/lora/test_tinker_cookbook_gateway.py failed in 39s — workflow run

The run never started: resolving the test file's execution plan failed.

[yueming-yuan]

/rerun-test tests/e2e/lora/test_tinker_multi_tenant_gateway.py

[yueming-yuan]

/rerun-test tests/e2e/lora/test_tinker_cookbook_gateway.py

[yueming-yuan]

/rerun-test tests/e2e/megatron/test_qwen3_5_35b_a3b_lora_ci.py

[yueming-yuan]

/rerun-test tests/e2e/megatron/model_scripts/test_nemotron_3_ultra_4layer_ci.py

[github-actions]

✅ tests/e2e/lora/test_tinker_cookbook_gateway.py passed on stage-c-8-gpu-h200 in 7m41s — workflow run

[github-actions]

✅ tests/e2e/lora/test_tinker_multi_tenant_gateway.py passed on stage-c-8-gpu-h200 in 7m18s — workflow run

[github-actions]

✅ tests/e2e/megatron/model_scripts/test_nemotron_3_ultra_4layer_ci.py passed on stage-c-8-gpu-h200 in 16m14s — workflow run

[github-actions]

✅ tests/e2e/megatron/test_qwen3_5_35b_a3b_lora_ci.py passed on stage-c-8-gpu-h100 in 24m43s — workflow run