miles: drive the upstream-based tinker-seam interface - #15
Merged
Merged
Conversation
- backend.apply_optimizer_step: drop the non-existent set_learning_rate call; pass learning_rate into apply_optimizer_step[_and_sync](learning_rate=) to match the miles fork's RayTrainGroup interface (github.com/GavinZhu-GMI/miles main). - setup_container.sh (miles profile): overlay the fork's miles onto the base image (baked image predates the Tinker orchestration interface); default MILES_REF=main. Validated end-to-end on ns.config: create_model + 3x forward_backward + apply_optimizer_step(lr) all success=True; empty-data now correctly ValueErrors.
Omitted alpha became 0 via lora_config.get("alpha", 0), making LoRA
scaling alpha/rank = 0: gradients exactly zero, constant loss — the
grad_norm=0 seen in every LoRA round-trip. Schema already documents
alpha-defaults-to-rank. Confirmed on ns.config (specs/005 G1 spike):
with nonzero alpha, grad_norm=207, loss 11.7->3.2 over 5 steps.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
cp -r over the image's stale checkout left orphan files and a 73-file dirty git status that masqueraded as unsaved work (see specs/005 HANDOFF 2026-07-13). When the install is a git checkout, fetch+checkout the pinned ref and clean the package dir; else delete-then-copy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Targets GavinZhu-GMI/miles branch tinker-seam (6e79b54), which re-ports the
Tinker orchestration onto current upstream radixark/miles (specs/005).
- create_model: reuse upstream factories (create_placement_groups,
create_rollout_manager) and the new TinkerTrainGroup async fanout; await
group.init(); router address from args.sglang_router_{ip,port}; startup
weight sync mirrors upstream train.py's offload dance.
- forward_backward: await forward_backward_only; average pp-last metrics
across DP ranks. fb returns no per-sample logprobs on the seam (open item).
- apply_optimizer_step: await apply_optimizer_step[_and_sync](lr);
upstream RolloutManager onload_weights/onload_kv replace tag-based onload.
- forward: forward_logprobs returns client-ordered tensors; result assembled
inline (rollout_to_forward_result's per-shard juggling is obsolete here).
- save/load_checkpoint via group broadcast (load restores optimizer state).
- delete_model: _actor_handles rename; placement-group dict cleanup.
- converter: send dynamic_global_batch_size=len(batch), _loss_norm_total=1
(pure-sum gradients, G1 contract) and _loss_type_override per request.
- builder: use_dynamic_global_batch_size=True, delay_split_train_data_by_dp=True.
Not runtime-tested: needs the tinker-seam miles image (upstream stack).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Five runtime-drift fixes found running G1/G2 gates on the tinker-seam stack
(miles_tinker_seam:20260713 image):
- model_config: transformers 5.x moved rope_theta into rope_parameters;
the 10000 fallback tripped upstream's hf_validate_args.
- slime_builder: reset disable_{grad,param}_buffers_cpu_backup after forcing
offload_train=False — parse-time --offload derived them True, and actors
crashed allocating grad buffers in a TMS region without LD_PRELOAD.
- data_converter (forward path): response_length is the mask LENGTH (not its
nonzero count) + causal N-1 trim when response==total; fixes shape assert
and empty logprob slices.
- backend: run create_placement_groups / group construction off the event
loop (retrieve_future polls stalled); skip None rollout pg on delete.
- deploy script: forward MILES_REPO/MILES_REF into the container setup (the
overlay silently checked out main and reverted the baked seam).
Gate results (Qwen2.5-0.5B LoRA r32/a64, DP=4): G1 ratio EXACTLY 1.0
(grad_norm 1485.871... bit-identical for 1x8 vs 4+4 fb split — pure-sum
contract holds); G2 client-order exact; weights move; loss 11.78->5.11 over
5 steps; delete_model clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Fixes found running sl_basic (NoRobots, 30 steps) through the full stack (RolloutManager + SGLang engines + bridge-LoRA), each one layer deeper: - converter: concatenate ALL model_input chunks (chunks[0]-only silently truncated multi-chunk datums: 4-token tokens vs 438-token weights, which also masqueraded as a dynamo compile failure in fused cross-entropy) - builder: rollout_global_dataset=False (Tinker clients drive data; upstream RolloutManager otherwise requires a server-side prompt dataset file) - builder: target_modules from Tinker train_attn/train_mlp flags (upstream requires them for LoRA), megatron_to_hf_mode=bridge (raw mode silently builds full-finetune: no lora_A/B params, weight sync fails) - backend: fb metrics keys as name:mean (SDK reduction convention); fb returns per-datum logprobs in client order (SDK weights reductions by len(loss_fn_outputs); cookbook computes NLL from them; miles tinker-seam b1c2c0f computes them actor-side on pre-step weights) G3 result: 30/30 steps, per-request linear LR schedule honored, NLL stable 2.7->2.4 on an already-instruction-tuned base, checkpoint saved, clean exit. KNOWN ISSUE: multi-fb grad accumulation on bridge-LoRA shows ratio ~1/sqrt(2) (second fb appears to overwrite adapter grads) — single-fb-per-step flows (the cookbook default) are unaffected; tracked in specs/005 HANDOFF. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adapts
MilesBackendto the re-ported Tinker seam on upstream-based miles (GavinZhu-GMI/milestinker-seam, based on radixark/milesfecd5d9), replacing the old fork's Ray-orchestration interface. Design + validation record: monorepospecs/005-miles-ray-interface/.What changed
TinkerTrainGroupasync fanout (await, no more sync +to_thread); upstream factories for placement groups / RolloutManager.dynamic_global_batch_size=len(batch),_loss_norm_total=1(pure-sum loss semantics — split-invariant gradient accumulation across pipelinedforward_backwardcalls), and_loss_type_overridefor per-request loss selection.use_dynamic_global_batch_size=True,delay_split_train_data_by_dp=True; LoRA alpha now defaults to rank per the API schema (was 0 → zero gradients by construction).cp -r), forwardsMILES_REPO/MILES_REFinto the container.Validation (ns.config 4×H200, DP=4, Qwen2.5-0.5B bridge-LoRA, 2026-07-13/14)
Note the semantics shift: pure-sum grad_norm scales with total weighted tokens (~100× the old mean-normalized value); clients own normalization via weights/LR, per the Tinker contract.