[yueming-yuan]

Co-authored with @yushengsu-thu.

Connect the protocol service to Miles through MilesBackend and serve_tinker.py.

Training commands place a work unit in the object store, let workers split it, and reassemble per-datum outputs by sample index. Forward-only runs the requested loss without backward. Explicit actor failure results pass through to the gateway; unknown trainer failures retain the common TrainerCell boundary and terminate the current single-cell service through its supervisor.

The entry point launches the worker manager, inference controller, trainer, service loop, and HTTP server. It requires the trainer to load the same frozen HF checkpoint as the inference engines. The runtime sends sampling requests through the router with the immutable adapter name and disk backfill path, without owning an inference controller or a weight-update window. Save and export commands forward gateway metadata to the worker's directory transaction. Checkpoint execution errors propagate through the same trainer-cell and supervisor shutdown path.

The gateway exposes its port, base-model name, and checkpoint root, and derives its adapter layout from normalized HF target names. Both standard and canonical Megatron target names are supported. Per-slot optimizer ownership is established in #2840.

Stacked on #2844.

Validation:

• Reject a missing checkpoint root before worker initialization. A before/after startup probe confirmed no worker launch is reached on invalid configuration.
• pre-commit run --all-files and incremental git diff --check passed on every updated component head. All previous heads remain ancestors; the final Unify HF snapshot and checkpoint writing #3198 tree exactly matches the reviewed implementation.
• Process-exit probes at both sides of publication lost the public path before the fix; afterward it resolves to the complete old or new version, and a subsequent save succeeds. A parameterized regression is included in the existing [multi-lora v2 7/7] Tinker gateway example and tests #2849 checkpoint test file.
• Sampler-to-training load probes with and without optimizer state now return a user failure before any backend load, while another tenant continues training. Repeated SAVE → LOAD verifies distinct retained versions and loading from a pinned version.
• Focused checkpoint continuation passed on 2 H200 GPUs (DP=2, TP/PP/EP=1), using real Megatron DDP, MultiLoRA layers, SlotOptimizer and distributed checkpoint IO. After three Adam steps, slot 0 was saved and restored into a fresh optimizer on slot 1; weights, FP32 masters, Adam moments, parameter-group metadata and step counters matched with zero tolerance before and after the next identical-gradient update. Rounding masters to BF16 and restarting from weights alone were both detected by negative controls.
• The enabled regression is tests/e2e/lora/test_slot_checkpoint_resume.py (Hopper, LoRA, 2-GPU stage). The tested final tree is Unify HF snapshot and checkpoint writing #3198 fd63c7ec8c41c2e0a61d273f0cd5a3504275ada7, with Megatron 73b54618f7e58e0f25f619bcfecbe2640765475a, Bridge 40b930897717941cfe2bd9806f417e28bf1bfa65, PyTorch 2.13.0+cu130 and Transformer Engine 2.17.0. Checkpoints were on GPFS. pre-commit run --all-files passed.
• This isolates checkpoint/Adam continuation with identical post-reduction gradients; it does not validate forward/backward numerics, topology changes, MoE/TP/PP layouts, server restart, or failure recovery. Fresh hosted CPU/NVIDIA CI is pending; broad GPU selection remains on Unify HF snapshot and checkpoint writing #3198. The separate Tinker cookbook acceptance test remains disabled pending GPU validation.

ci-megatron-pr: 73b54618f7e58e0f25f619bcfecbe2640765475a
ci-sglang-pr: 02b5e123c490c988d21526d0dd7837fefdd0e1da

Controller lifecycle: match train.py by disposing the inference controller, then the trainer, after normal completion of the serving tasks. No AsyncExitStack or initialization/exception cleanup guarantees are introduced. Full pre-commit passed; no regression tests were added.

Router dependency: install the router wheel and standalone binary from the existing cu130-torch213-x86_64 / cu130-torch213-aarch64 rolling releases in yueming-yuan/miles-wheels, using the normal /tmp/wheels installation path. Both architectures were rebuilt on CPU devboxes from radixark/sgl-router-for-miles@b6342a3515e259bbd1eb4db728035a4efbe4fdf9 and verified to forward return_indexer_topk and lora_backfill_paths. No commit-specific router release is required. The existing pr-2849 Docker image has not been rebuilt by this change.

Router-pin validation: full pre-commit passed; the Docker RUN shell syntax and image-input hash change were checked. Both architectures' wheel and binary downloads match the four previously tested SHA256 digests, and the source checkout resolves to the exact pinned commit. The existing native HTTP probes passed for both artifacts on both architectures. No new tests were added. The new Miles PR image and hosted CI have not finished yet.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[nanjiangwill]

Runtime integration review: three correctness/availability concerns and two explicitly non-blocking lifecycle suggestions.

[nanjiangwill]

Addressed: service.run() and Uvicorn are now supervised together, and completion or failure of either task tears down the other.

[nanjiangwill]

TrainerController._execute_slots() calls TrainerCell.execute() with the default kill_on_failure=True; any actor exception marks the only trainer cell errored and kills its workers. TinkerService instead treats batch failures as recoverable and calls zero_grads() before continuing, so cleanup targets a dead cell and can stop dispatch. Create and load failures likewise leave the service alive with no usable trainer. Align the contracts: either stop the gateway when a trainer RPC kills the cell, or classify recoverable request errors before that boundary. Keep CUDA/NCCL failures cell-fatal.

[nanjiangwill]

_to_sequence() maps every finish reason except length to stop. SGLang returns type="abort", and miles’ normal rollout path maps that to Sample.Status.ABORTED; here it becomes a successful partial sample that the client can train on. Raise on abort—the Tinker wire enum has only stop and length—so _run_sample() settles the request as a server failure.

[nanjiangwill]

Non-blocking: compute and validate checkpoint_root before launch_worker_manager() and controller initialization. A missing --tinker-checkpoint-root/--save currently starts the worker and GPU stack before hitting this assertion.

[nanjiangwill]

Non-blocking: dispose trainer and inference_controller in this finally block. Their disposal stops watchers, tickers, and inference server clients; cancelling only service_task leaves those resources live until process exit.