[Sync] Mirror Slime through 8c17b676 (#2437/#2434)

[aoshen02]

Summary

Mirror Slime from 4c193f1f through 8c17b676: 14 applicable merged PRs;
Ascend #2424 remains explicitly excluded. Biren/SUPA is mirrored as requested.

Includes score centering, Straw-backed distributed fully-async rollout and
checkpoint continuation, PipelineRL flush, and reference full-vocabulary top-p
handling. Existing Vime-native overlays are retained separately from translations.

Engine patches

The vLLM source pin remains e9757321527ca1ecd514c07c1418dd2c53da3d19.

Patch	Purpose / PR
vllm-pull_weights.patch	Pull-checkpoint transport: aoshen02/vllm#45
vllm.patch	Weight-version propagation: vllm-project/vllm#53199; terminal speculative metrics preservation: vllm-project/vllm#55708; empty terminal token-stream events: merged vllm-project/vllm#47933
vllm-score-centering.patch	Sampling-mask transport and exact top-p extension: aoshen02/vllm#83
vllm-pd-request-metrics.patch	PD request metrics: aoshen02/vllm#29
vllm-inflight-queue-diagnostics.patch	Queue diagnostics: vllm-project/vllm#55274
vllm-aux-output-reset.patch	Backport merged vllm-project/vllm#59060, absent from the pin

Final audit

• Reviewed all 150 changed paths against the Slime tip and the pre-sync Vime baseline.
• Fixed coalesced mask IDs/logprobs and terminal speculative metrics; real-engine regressions fail before and pass after.
• Validate actual processed_logprobs server configuration, including external engines, rather than a nonexistent environment variable.
• Removed obsolete per-file R3 spill and its flags/tests; use upstream Straw references. Restore direct Straw documentation translations and omitted upstream comments.
• Audit fixes net-remove approximately 500 lines. Historical engine gaps outside this sync are not claimed closed.

Validation

• Relevant post-cleanup CPU regressions: 292 passed; baked runtime image checks: 175 passed.
• Final image smoke: 3 output-coalescing regressions + 4 documentation checks passed.
• All six patches apply sequentially to the exact pin; 2,533 patched Python files parse successfully.
• Changed-file pre-commit passes.
• Final commit: f32a36f1d8f811260f92c1901e808c35ad13c4ea.
• Experimental AMD64 image: aosheninferact/vime@sha256:d2d60d7ac5a4fc27299bee5a6ad9f24ba30508f52590dd7f10d4b66516f56cf1.
• #1445 passed 49 jobs; PipelineRL NCCL with periodic cache flush failed with Straw file-descriptor exhaustion. Exact-image sequential read checks show no FD growth; a deliberately exhausted soft limit reproduces the exception. Two CI startup lines now raise the soft FD limit to the available hard limit and log both; concurrent resource leaks are not ruled out by this check.
• H200 four-GPU control with soft FD limit1024 reproduces the exact Straw failure; raising the limit allows the normal socket concurrency (observed generation-actor FD count1614). A second high-limit run exposed an independent empty-terminal SSE omission already fixed by upstream #47933; backport that condition rather than weaken the abort assertion. Corrected actual-serving regression matrix: six failures/two passes before, eight passes after; full transport file28 passes. Final image passed two independent complete four-GPU PipelineRL runs (one trainer, three rollout engines, NCCL, NVLS0, flush interval2), including all three training rollouts, weight-version continuity, periodic abort and checkpoint assertions; both exit0 in4m03s.
• Full final CI #1454: pending; all six GPU suites must pass before merge readiness.
• Previous #1429 passed 50 jobs on the earlier revision, not evidence of final-head green.

vllm/vime:latest remains unchanged; ARM publication follows full AMD64 CI.

[read-the-docs-community]

Documentation build overview

📚 vime | 🛠️ Build #34968387 | 📁 Comparing f32a36f against latest (791f101)

  🔍 Preview build  

46 files changed · + 1 added · ± 45 modified

+ Added

• advanced/straw.html

± Modified

• genindex.html
• index.html
• search.html
• advanced/arch-support-beyond-megatron.html
• advanced/delta-weight-sync.html
• advanced/external-rollout-engines.html
• advanced/fault-tolerance.html
• advanced/low-precision.html
• advanced/megatron-config.html
• advanced/observability.html
• and 35 more...

[gemini-code-assist]

Code Review

This pull request consolidates asynchronous training by removing train_async.py and integrating fully-asynchronous rollout capabilities directly into train.py. It also introduces support for the SUPA accelerator (Biren GPUs), implements disk-spilling and prefetching for routed experts (R3) to optimize host memory usage, and adds a utility to reset the CUDA stack size after model offload. The review feedback highlights several critical robustness improvements, including safely accessing torch.version attributes to prevent AttributeError on standard PyTorch installations, defensively handling potentially None values for args.num_experts to avoid TypeError, and correcting test mocks to avoid attribute errors during unit testing.

[gemini-code-assist]

Accessing torch.version.hip directly will raise an AttributeError on standard (non-ROCm) PyTorch installations because the hip attribute is not defined on the torch.version module in standard builds. To prevent this, use getattr(torch.version, "hip", None) and getattr(torch.version, "cuda", None) to safely check the versions. Additionally, wrapping the entire body of reset_cuda_stack_size in a try...except block is highly recommended to ensure that any unexpected CUDA stack API loading or driver failures do not crash the training process.

def reset_cuda_stack_size() -> None:
    """Release an enlarged CUDA per-thread stack after model offload."""
    cuda_version = getattr(torch.version, "cuda", None)
    hip_version = getattr(torch.version, "hip", None)
    if cuda_version is None or hip_version is not None or not torch.cuda.is_initialized():
        return
    try:
        torch.cuda.synchronize()
        driver = _cuda_stack_api()
        previous = ctypes.c_size_t()
        error = driver.cuCtxGetLimit(ctypes.byref(previous), 0)  # CU_LIMIT_STACK_SIZE
        if error:
            raise RuntimeError(f"cuCtxGetLimit(CU_LIMIT_STACK_SIZE) failed: CUDA error {error}")
        if previous.value <= 1024:
            return
        error = driver.cuCtxSetLimit(0, 1024)
        if error:
            raise RuntimeError(f"cuCtxSetLimit(CU_LIMIT_STACK_SIZE) failed: CUDA error {error}")
        logger.info("Reset CUDA stack limit after offload: %d -> 1024 bytes", previous.value)
    except Exception as exc:
        logger.warning("Failed to reset CUDA stack size: %s", exc)

[gemini-code-assist]

Using monkeypatch.setattr to set cuda and hip directly on memory_utils.torch.version will raise an AttributeError on standard PyTorch installations because the hip attribute does not exist on the torch.version module. To fix this, mock the entire version attribute as a SimpleNamespace containing both cuda and hip attributes, which is consistent with the other test in this file.

Suggested change

	monkeypatch.setattr(memory_utils.torch.version, "cuda", cuda)
	monkeypatch.setattr(memory_utils.torch.version, "hip", hip)
	monkeypatch.setattr(memory_utils.torch, "version", SimpleNamespace(cuda=cuda, hip=hip))

[gemini-code-assist]

If args.num_experts is explicitly set to None, calling int(None) will raise a TypeError. It is safer to retrieve the value defensively and default to torch.iinfo(torch.int32).max if it is None or not present.

Suggested change

	num_experts = int(getattr(args, "num_experts", torch.iinfo(torch.int32).max))
	num_experts_val = getattr(args, "num_experts", None)
	num_experts = int(num_experts_val) if num_experts_val is not None else torch.iinfo(torch.int32).max

[gemini-code-assist]

If args.num_experts is None or not defined on args, accessing args.num_experts directly will raise a TypeError or AttributeError. It is safer to use getattr defensively to check if num_experts is set and is less than or equal to 256.

Suggested change

	dtype = torch.uint8 if args.num_experts <= 256 else torch.int32
	num_experts = getattr(args, "num_experts", None)
	dtype = torch.uint8 if num_experts is not None and num_experts <= 256 else torch.int32

[aoshen02]

Exact-head full CI is green: Buildkite #1385 passed all automatic CPU checks and 40/40 GPU jobs on commit 3f48e09f35740a8f4831aa8a488419fdbc986fab using experimental AMD64 image aosheninferact/vime@sha256:1ad4d005f6c5f6b0d6e38b16e821fa525021bc5d6deeeea644e3047463453113. The sole first-attempt failure was Buildkite agent loss (exit_status=-1, no test exception) on Qwen3-30B-A3B R3; one unchanged retry passed. GitHub commit status now points to #1385 and is green. Per the sync SOP, I then checked ARM64 access using ssh -o BatchMode=yes -o ConnectTimeout=5 gb200-rack1-05 'uname -m; hostname'; it timed out during SSH banner exchange. No ARM64 candidate image was built or validated, and vllm/vime:latest remains unchanged.

[aoshen02]

ARM64 follow-up: ssh -G gb200-rack1-05 reveals ProxyJump gcp-gb200-head. This alias is not an independent non-GCP route; the user has disallowed use of the GCP GB200 host. I will not build through that route. ARM64 candidate build/validation remains pending an approved accessible ARM64 host.