[yueming-yuan]

Co-authored with @yushengsu-thu.

Add the gateway's protocol and scheduling layers under miles/tinker/core and miles/tinker/server.

• Per-model streams order and deduplicate requests. The planner packs compatible datums or merges ready optimizer barriers.
• A terminal forward/backward, optimizer-step, or load failure ends the affected model stream and fails its queued requests. Completed futures retain their results. Authenticated content-validation failures with a valid sequence settle at that position, so later training cannot skip a failed command.
• JSON and protobuf boundaries validate SDK datums and render future results. Authentication accepts X-API-Key and bearer tokens.

A sampler save exports the current adapter and returns its path after the immutable directory commits. It makes no inference-engine calls and neither applies nor discards accumulated gradients. Sampling resolves ownership and base-model compatibility from the saved metadata whether or not the training model still exists. There is no separate in-memory publication registry.

Known request-local forward-only and sampling failures leave the training stream available. Checkpoint load/save/export execution failures terminate the shared trainer cell and service; load_state rejects sampler_weights before metadata lookup or backend execution because the training loader requires Megatron DCP. weights_info remains available for both checkpoint kinds. Failed sampling does not invalidate a saved snapshot. Unknown trainer and background-task failures reach the serving supervisor. This implements failure handling and isolation; automatic recovery and stream replay are not implemented.

Model creation rechecks its record under the trainer lock. Model closure fails its creation future and only unloads initialized slots. Creation, training, and sampling duplicates whose results expired receive a stable terminal failure without replaying backend work. Unknown sampling sessions are user errors; foreign-tenant sessions remain ownership errors.

Futures, sessions, leases, deduplication, and unsaved accumulation are process-local. Saved checkpoints retain ownership and adapter metadata. These protocol layers do not import miles.ray or miles.backends; runtime wiring follows in #2845.

Request completion is centralized in _finish_request; batch and barrier execution share its failure policy. Checkpoint helpers live in core/utils.py, request validation stays in the service, and retry-result retention belongs to FutureStore. The dispatcher cancels and awaits its model-creation, sampling, and lease tasks before exiting.

Saving preserves the public checkpoint link; loading resolves it once before metadata validation and backend execution. Legacy checkpoints can be loaded, while legacy-directory overwrites are rejected before reaching the trainer.

Stacked on #2843.

Validation:

• pre-commit run --all-files and incremental git diff --check passed on every updated component head. All previous heads remain ancestors; the final Unify HF snapshot and checkpoint writing #3198 tree exactly matches the reviewed implementation.
• Process-exit probes at both sides of publication lost the public path before the fix; afterward it resolves to the complete old or new version, and a subsequent save succeeds. A parameterized regression is included in the existing [multi-lora v2 7/7] Tinker gateway example and tests #2849 checkpoint test file.
• Sampler-to-training load probes with and without optimizer state now return a user failure before any backend load, while another tenant continues training. Repeated SAVE → LOAD verifies distinct retained versions and loading from a pinned version.
• Focused checkpoint continuation passed on 2 H200 GPUs (DP=2, TP/PP/EP=1), using real Megatron DDP, MultiLoRA layers, SlotOptimizer and distributed checkpoint IO. After three Adam steps, slot 0 was saved and restored into a fresh optimizer on slot 1; weights, FP32 masters, Adam moments, parameter-group metadata and step counters matched with zero tolerance before and after the next identical-gradient update. Rounding masters to BF16 and restarting from weights alone were both detected by negative controls.
• The enabled regression is tests/e2e/lora/test_slot_checkpoint_resume.py (Hopper, LoRA, 2-GPU stage). The tested final tree is Unify HF snapshot and checkpoint writing #3198 fd63c7ec8c41c2e0a61d273f0cd5a3504275ada7, with Megatron 73b54618f7e58e0f25f619bcfecbe2640765475a, Bridge 40b930897717941cfe2bd9806f417e28bf1bfa65, PyTorch 2.13.0+cu130 and Transformer Engine 2.17.0. Checkpoints were on GPFS. pre-commit run --all-files passed.
• This isolates checkpoint/Adam continuation with identical post-reduction gradients; it does not validate forward/backward numerics, topology changes, MoE/TP/PP layouts, server restart, or failure recovery. Fresh hosted CPU/NVIDIA CI is pending; broad GPU selection remains on Unify HF snapshot and checkpoint writing #3198. The separate Tinker cookbook acceptance test remains disabled pending GPU validation.

ci-megatron-pr: 73b54618f7e58e0f25f619bcfecbe2640765475a
ci-sglang-pr: 02b5e123c490c988d21526d0dd7837fefdd0e1da

Session lease correction: models, sampling sessions, and in-flight sampling requests track their owning session. Expiry reclaims only that session’s resources; direct sessionless sampling is unaffected. The sweep decides expiry after acquiring the trainer lock and settles sampling futures before cancellation.

Validation: a real-service/fake-backend before/after probe reproduced stale model retention and confirmed selective slot reclamation. No regression tests are added for this change. Full pre-commit passed; CPU CI execution is pending.

Protobuf loss configuration: prefer v2 number/text values when present, otherwise decode the legacy numeric map. SDK 0.26.2 protobuf probes cover legacy, v2 precedence, v2-only values, zero, empty text, and malformed values. Full pre-commit passed; no regression tests were added.

[claude]

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

_{Tip: disable this comment in your organization's Code Review settings.}

[yueming-yuan]

Planner rewritten to the final architecture (no interim FCFS-passthrough): per-model streams compile command runs into commutative windows; the planner packs ready datums across models into WorkUnits by (kind, loss_fn, config) equivalence class under a token budget — cross-tenant packing and large-request splitting are the same datum-granular bin-packer; ready optim barriers of different models merge into one BarrierUnit (step_adapter_slots takes the slot dict). Completion accounting reassembles per-datum outputs back to per-request promises; a request may span units and a unit may span requests.

In-memory fake-executor smoke covers packing, barrier merge, seq ordering across barriers, out-of-order arrival, idempotent resubmit, and tenant ownership.

[nanjiangwill]

CreateModelRequest includes session_id and model_seq_id, but this method ignores both and always allocates a random model and slot. Tinker 0.26.2’s generated POST client retries up to 10 times by default with the same request body, so a lost response can run load_slot() twice and consume two slots. The model is also leased by tenant, so another live session for the same API key keeps it alive. Validate the session, key creation by (tenant, session_id, model_seq_id), store the owning session, and return the existing future/model on retry.

[nanjiangwill]

load_slot() can fail after init_adapter_slot, gradient reset, or optimizer reset has already mutated some ranks, but this exception path returns the slot directly to free_slots. The test fails before any mutation, so it does not cover that state. Run unload_slot() before reuse and return the slot only if cleanup succeeds; otherwise quarantine it.

[nanjiangwill]

CreateSamplingSessionRequest carries session_id, sampling_session_seq_id, and a base_model/model_path binding, but this stores only model_path. Later SDK sample requests normally send only sampling_session_id, so a session created with base_model="other" is accepted and then samples this gateway’s configured base model. The stable (session_id, sampling_session_seq_id) is also ignored, so a retry creates another session. Validate the parent session and model selection, store the full binding, and deduplicate creation by that pair.

[nanjiangwill]

decode_sample_request() drops seq_id, and submit_sample() always creates a new future and task. Tinker 0.26.2 calls execute_with_retries() with the same request ID, so a lost response runs generation again and returns different IDs. Preserve seq_id, deduplicate by (sampling_session_id, seq_id) before scheduling, return the retained result on retry, and fail terminally after retention expires.

[nanjiangwill]

Non-blocking: ModelStream advances only from watermark 0 to sequence 1, but submit() accepts seq_id <= 0 and non-integers. Those entries can never match watermark + 1, so their futures remain pending forever. Reject non-integers and values below 1 before creating the future; protobuf’s missing value is 0, so cover that boundary.

[nanjiangwill]

_poisoned_slots.pop() runs before zero_grads(). If zero_grads() raises, _run_barrier() fails the current optimizer request, but the poison has already been forgotten; the next optim_step can run on ambiguous gradients. Keep the poison until cleanup succeeds. If cleanup fails, retire or quarantine the model instead of continuing to serve it.

[nanjiangwill]

Non-blocking: validate the complete outcome map before settling any future. This loop can resolve and remove one stream, then hit a missing or malformed outcome for a later slot. _run_barrier() then calls _fail_barrier() for every entry, and the second stream.finish() on the already removed entry raises ValueError, terminating service.run(). A validation pass before settlement avoids the partial state.

[nanjiangwill]

Non-blocking compatibility note: this decodes only the legacy loss_fn_config. Tinker 0.26.2 dual-writes numeric values to both maps but writes strings only to loss_fn_config_v2, so string settings are silently dropped. Prefer loss_fn_config_v2 when present and fall back to the legacy map.