[nxphi47]

Summary

Remove CP-divisibility tokens and duplicated zero-loss DP microbatches from RL model inputs. Distribute the original packed row into balanced, potentially empty CP shards, and represent an inactive DP lane with a zero-token slot that still participates in distributed execution.

The implementation is generalized across the custom model families and HF inactive language-backbone execution. It does not use a Qwen-only allowlist. Both model.cp_unpadded and model.inactive_micro_batches default to true; managed RL propagates the matching settings to the orchestrator. Existing model/backend and head/group-divisibility requirements still apply.

This is a draft: two strict numerical comparisons and two integration checks remain open. Details and the distinction between passing checks, failed comparisons, and unrun checks are below.

Twenty-update real RL follow-up: warmed-up trainer throughput is essentially flat. With updates 1–10 excluded and 11–20 measured, padded versus unpadded is 6468.0 → 6486.5 original tokens/s (+0.29%) on Qwen3-30B, CP4 Ulysses / EP4, using the original torch.compile(fullgraph=False). Neither run has recompilation after update 1 or late communicator initialization; measured-half drift is −3.33% / −3.04%. The earlier three-update 11–16% trainer-throughput deficit did not persist. End-to-end RL throughput is 529.8 → 481.4 tokens/s (−9.14%), accompanied by longer rollout waits in the unpadded run. Independent sampled batches, policy trajectories and separate nodes prevent attributing either difference to padding removal. The 18.4% dense / 4.1% MoE latency gains below come from controlled DP2/CP2 benchmarks with an inactive data lane. This real CP4/EP4 topology has one data lane and removed only 102 appended tokens from about 1.96M original tokens in the measured baseline window.

Motivation and design

There are two independent sources of unnecessary model work:

1. A packed row is extended until its token count is divisible by CP. Those extra tokens pass through embeddings, decoder layers, attention and the output head even though they contribute no intended loss.
2. To align distributed microstep counts and modality order, an underfilled DP group duplicates a real microbatch and zeros its loss. A long duplicated row can cost much more than the few tokens required for CP divisibility.

The data contract should express the actual tokens and the distributed schedule separately. For example, nine tokens over four CP ranks have lengths [3, 2, 2, 2]; one token has lengths [1, 0, 0, 0]. The same partition must apply to every token-aligned stream and to reconstruction in backward.

A rank with zero local tokens cannot simply skip its model call: other ranks can still need its FSDP parameters, EP experts and collective participation. Inactive slots therefore preserve the required hooks, communication and zero gradients while forwarding no fabricated input tokens.

This work preserves loss membership and component normalization, packed sample boundaries, sampling/router replay, optimizer behavior and the configured optimization/reduction dtypes. Sampling-set -1 storage, internal kernel alignment, and SFT fixed-pack dataset padding remain separate storage contracts. No dependency, submodule, or optimization_dtype / reduce_dtype changes are included.

Implementation

Partition and collective contract

• Add immutable CPPartition metadata with balanced lengths, offsets and checked token-axis slicing.
• Use Torch's autograd-registered functional collectives for uneven sequence/head exchanges and reconstruction; preserve the equal-size gather path where applicable.
• Keep topology in the shared frozen CPContext and pass changing token/partition metadata explicitly through forwards and checkpoint recomputation.
• Specialize very short sizes during compilation so zero/unit shard strides are correct in compiled backward.
• Add a shared module-owned attention path. Ulysses redistributes actual token rows; ring gathers a combined real-token KV buffer and applies the correct causal packed boundaries. The ring buffer's memory tradeoff is measured below.

Model and kernel integration

• Update Qwen3, Llama, Qwen3-MoE, AFMoE, GLM4-MoE, MiniMax-M2, Laguna, GPT-OSS, Nemotron, Qwen3.5, GLM-DSA and DeepSeek-V4 consumers.
• Use balanced global/local offsets for rotary positions, sparse queries/keys/compression layouts, recurrent feature exchanges and vision insertion.
• Recurrent layers redistribute features or gather the actual sequence according to their kernel requirements; DeltaNet includes a gathered path when value heads cannot be evenly distributed.
• Define empty boundaries for attention, convolution/scan, normalization, sparse attention, vision and dense FP8 kernels. HF empty backbones traverse the existing embedding/decoder/norm FSDP units through language-model discovery, including nested language backbones.
• Give FA2/FA3 distinct cumulative-length tensors where required by compiled autograd. Clone DeltaNet cumulative lengths per forward so FLA metadata caching does not change selective checkpoint replay's operation ordering.

Packing, transport and training

• Split bins at whole-sample boundaries within modality groups until further alignment is impossible; reclassify text bins produced by splitting mixed bins.
• Append a wire-compatible inactive field to MicroBatch; remaining empty slots contain no input tokens, images or sample identities.
• Apply the same partition to IDs, positions, labels, temperatures, sampling masks, router replay and LoRA offsets, including one-token loss alignment.
• Preserve global RL/CE/reference-KL member counts. Inactive slots execute backward but contribute no loss members or sample annotations.
• Count actual input tokens, real microbatches, inactive slots and accepted samples; exclude empty routing groups from MoE metric means.
• Reject an entirely empty global step before offload preparation and optimizer/scheduler updates.
• Move completed-update token/sample counters before checkpoint saving.
• Update SFT's shared CP integration while preserving its dataset packing and local loss normalization; add runtime documentation and a real four-rank distributed regression harness.

What actually ran: RL versus SFT

The end-to-end jobs were RL. No end-to-end SFT training job was launched. SFT validation covered its shared CP integration, summed-CE/IGNORE_INDEX behavior, fake dataset and loss tests, and eager/compiled model gradients.

Real RL configuration

Setting	Value
Model	Qwen/Qwen3-30B-A3B-Thinking-2507, custom implementation, 48 layers
Training data/environment	Source name math, taskset ID i3_math; agent harness null, subprocess runtime
Hardware/deployment	Single node, eight H200 GPUs: four trainer GPUs + four inference GPUs
Trainer topology	CP=4, EP=4, Ulysses; the four trainer GPUs form one data lane
Attention	FlashAttention 3
Compilation	CompileConfig(fullgraph=False, mode=None) enabled; trainer log confirms all 48 layers were compiled
Activation handling	Full activation checkpointing every layer; activation offload with max_inflight_activations=5
Optimizer	AdamW, learning rate 1e-6; native full CPU optimizer/gradient offload
Numerics	Configured FP32 optimization/reduction/router policy, BF16 model compute; no numerical default changes
Length limits	seq_len=32768, inference max_model_len=32768, max_completion_tokens=31000
RL workloads	Original three-update checks and fresh twenty-update follow-up; batch_size=8, group_size=4, initial/max inflight 8
Inference topology	Tensor parallelism 4, data parallelism 1
Weight synchronization	NCCL weight broadcasts
Checkpoints	Original unpadded smoke saved each step; original resume used step 1. Twenty-update follow-up saves only the final checkpoint after the measured loop

Jobs were launched with uv run rl @ base.toml @ overlays..., using the supplied math32k_sl1node_off8.toml example plus smoke/length/unpadded/resume overlays. These were short execution/regression runs, not convergence or model-quality experiments. Evaluation and W&B were disabled for the smoke runs.

SLURM job	Run	Result
1769	Padded baseline	Completed three RL optimizer updates and weight broadcasts
1771	Unpadded	Completed three RL updates, broadcasts and final checkpoint
1777	Unpadded resume	Loaded the retained step-1 checkpoint, completed another update and broadcast
1815	Padded twenty-update follow-up	Completed 20 RL updates, broadcasts and final checkpoint; saved counters verified
1814	Unpadded twenty-update follow-up	Completed 20 RL updates, broadcasts and final checkpoint; saved counters verified

The real RL jobs establish CP4/EP4 execution, offload and resume. Because there is one trainer data lane, the separate DP2/CP2 tests below establish inactive-DP-lane correctness. The real runs consumed different sampled batches, so their throughput is not a controlled speed comparison.

Validation performed

GPU validation used interactive srun allocations and uv run/torchrun, on H200 hardware. Environment: Torch 2.13.0+cu130, CUDA 13.0, NCCL 2.29.7, FA2 2.8.3, FA3 3.0.0, FLA 0.5.2, ring-flash-attn 0.1.8, prime-kernels 0.1.0+cu130torch2.13.0, Transformers 5.6.2.

Validation was performed in targeted subsets and standalone diagnostics during development. There was no single final execution of the entire model/backend/topology cross-product. The original three-update/resume jobs predate the checkpoint-counter ordering correction; the twenty-update pair uses the final implementation and verifies the corrected saved counters.

Correctness and integration results

Area	Executed coverage and result
Unit checks	285 CPU cases + 21 GPU RL-loss cases = 306 passing cases covering configs, packing/transport, partitions and SFT/RL loss behavior
Collective semantics	Exact eager and strict dynamic-compiled exchange/reconstruction and rank-weighted backward checks for empty, short, uneven, divisible and repeated sizes
Attention	FA2/FA3 Ulysses and ring, strict compiled forward/backward; FA4 in configured compilation with disclosed graph breaks
Dense inactive lanes	DP2/CP2, FSDP, full checkpointing and strict compilation; rotating [9,5], [9,0], [0,7], [3,1]; maximum parameter-gradient discrepancy 0.01074 within the 0.015 absolute / 0.05 relative bound
Initialized MoE inactive lanes	DP2/CP2, EP1 and EP4, full checkpointing and configured compilation; same lane transitions; maximum discrepancy 0.024353 within the baseline-calibrated 0.025 / 0.05 bound
Mixed RL/CE/reference-KL	Two accumulated initialized-MoE updates with sampling/router replay; component counts and gradients pass the original 0.015 / 0.05 gradient bound. Independent-reference weight comparison has the gap below
Optimizer semantics	Given the same validated global gradients, FP32 AdamW parameters and both moment states match at 1e-7 absolute / 1e-5 relative for both updates
Full CPU + activation offload	Initialized EP4 mixed-objective updates pass the unchanged 0.0006 / 0.015 weight bound; max discrepancies 0.0003196 and 0.00035735; step counters advance once per update
HF inactive execution	Qwen2, Llama and Gemma2, compiled FA2, FSDP DP4 and full checkpointing; max gradient error below 0.006. HF Qwen3-VL nested text backbone passes 3→0→1→0 forward/backward
Real-image VLM	Qwen3.5 trainable vision encoder, actual image patches, DP2/CP2/EP4 and compiled checkpointing; rotating active lanes; max gradient errors 0.004263 / 0.005705 / 0.005063
Recurrent paths	Mamba, DeltaNet head redistribution and non-head-divisible gathered path; real distributed output/input/parameter-gradient checks including empty lengths
Other complete model checks	Laguna, MiniMax-M2, GLM4-MoE, AFMoE, Qwen3.5 and valid-group Nemotron pass compiled CP4/full-checkpointing comparisons at T9/1/0/3/10. AFMoE/Qwen3.5/Nemotron maxima: 0.01990 / 0.00372 / 0.003710
Sparse/window/sinks	GLM-DSA and GPT-OSS FA4 eager CP4/checkpointing pass. All three DeepSeek attention variants pass strict compiled CP4 at T9/1/0/3/17, max error 0.015625; full-model DeepSeek remains open
LoRA	Configured compilation, checkpointing, DP2/CP2 and two mixed-objective updates pass; gradient errors 0.004845 / 0.002197 and weight errors 0.000200 / 0.000231
Shared SFT and FP8	SFT summed-CE head/gradient comparisons pass eager/strict compiled with IGNORE_INDEX; dense FP8 empty linear forward/backward passes eager/strict compiled
Metrics and annotations	Empty-group MoE stats and distributed metric-key union pass; inactive slots carry no sample annotations; actual RL traces/annotations and broadcasts retained
Static checks	Ruff check and format check pass for all 44 changed Python files; git diff --check passes

Fixture correction: Initial tiny Qwen3-MoE probes left stacked expert parameters uninitialized. Those objective/offload/performance results were rejected and rerun with explicit nonzero expert initialization and production FP32 routers. The results above use the corrected runs. The padded initialized-MoE reference also failed the initial 0.015 absolute gradient bound (max 0.024353), which motivated the documented 0.025 absolute bound for that regression; dense and exact collective bounds were unchanged.

torch.compile and CUDA Graph behavior

• Exact functional collective tests use torch.compile(fullgraph=True, dynamic=True): seven graph variants, zero graph breaks, exact forward/backward checks.
• FA2/FA3 attention and dense CP/FSDP/SFT checks include strict fullgraph=True forward and backward with checkpointing. Zero/unit shard specialization is explicit, rather than a newly introduced eager fallback.
• Real 30B RL uses fullgraph=False, as configured in the supplied recipe. Compile enablement and successful updates do not imply the entire application is one compiled graph.
• Installed FA4/CUTLASS fake-mode/JIT calls and existing multi-LoRA tensor-dependent control flow prevent strict fullgraph tracing in those paths. Their configured fullgraph=False paths were exercised and their limitations retained.
• The captured-row MoE timing probe records existing FSDP/disabled-hook graph breaks and empty graph-capture counters. Its speedup must not be described as a fullgraph compiled MoE result.
• Strict FA3 mode="reduce-overhead" captures/replays forward and backward at isolated T33/4097/8193/32765: two CUDA Graph captures, one Dynamo graph, zero breaks per size.
• A persistent compiled function across T32/33/34/35/33/32/1/0/3/33 ends with 5–6 Dynamo variants and 16–18 CUDA Graph captures, depending on rank, with zero graph breaks. The final return to T33 adds no variants/captures. Dynamic sizes still incur cache growth and some recapture after generalization.

Performance measurements

Controlled comparisons

The dense and captured-row MoE comparisons used a separate four-H200 allocation, identical real rows/masks within each comparison, full checkpointing, AdamW, five warmups and 30 measured updates (three windows of ten). Reported latency uses the slowest rank per iteration. Startup/warmup is separate from steady state.

Workload/metric	Padded	Unpadded	Result
Dense DP2/CP2 update median	116.813 ms	95.278 ms	18.4% lower
Dense p95	125.574 ms	100.494 ms	20.0% lower
Dense useful input throughput	105,211 tok/s	128,991 tok/s	22.6% higher
Dense executed input tokens/update	24,584	12,290	Removes copied lane and divisibility tokens
Dense inactive-rank peak allocated	607.1 MB	349.8 MB	42.4% lower; active ranks stay about 607 MB
Dense inactive-rank peak reserved	813.7 MB	509.6 MB	Active ranks about 786–795 MB unpadded
Initialized MoE DP2/CP2/EP4 median	372.038 ms	356.946 ms	4.1% lower
Initialized MoE p95	461.188 ms	402.613 ms	12.7% lower
Initialized MoE useful input throughput	41,329 tok/s	43,077 tok/s	4.2% higher
Initialized MoE executed tokens/update	30,756	15,376	Same two accepted rows/masks
MoE inactive-rank allocated/reserved	3473.6 / 4869.6 MB	3406.2 / 4802.5 MB	Expert ownership and large output head dominate memory

Dense model: four layers, hidden size 1024, FFN 3072, vocabulary 4096, Q16/KV4/head64, FA3, strict compile; rows 8193 and 4097. One lane deliberately holds real data and the other is copied/inactive. Maximum five-warmup duration was 31.24 s padded versus 27.96 s unpadded; this includes initialization/compilation and is not an isolated compiler-time measurement.

MoE replay: four layers, hidden size 1024, eight experts/top-2, expert FFN 1024, vocabulary covering captured IDs, initialized experts and FP32 routers. Accepted rollout rows contain 7173/8203 tokens and 7052/8082 trained positions. Timing uses a masked logprob objective on a small model and a deliberately underfilled two-round schedule, not the original 30B weights, the full RL objective, or the packer's optimal schedule. Corrected measurements ran unpadded before padded; maximum warmup duration was 12.86 versus 12.14 s. Active-rank memory is essentially unchanged.

CP-only, ring, communication and packing

• Direct functional Ulysses call, eager CP4: T4097 median 3.050→3.152 ms (3.3% slower), T8193 3.539→2.835 ms, T32765 14.706→14.646 ms. CP-only removal can save few tokens and does not guarantee faster execution.
• Ring CP4 FA3 Q32/KV4/head128: T4097 median 5.114→2.505 ms; T32765 23.564→23.999 ms (1.8% slower). Peak allocation 98.23→106.23 MiB and 785.01→849.01 MiB respectively. The combined real KV buffer costs another 8/64 MiB in these cases.
• Isolated strict FA3 CUDA Graph medians at T33/4097/8193/32765: 1.563→1.532, 1.662→1.540, 2.052→1.841, 14.340→14.327 ms.
• Dense two-row profiler: each padded rank issues 96 CP all-to-alls with 377.61 MB of input payload; active unpadded ranks issue 100 calls with about 377.65 MB, inactive ranks 88 calls with zero input payload. FSDP still carries about 109.07 MB all-gather and 453.06 MB reduce-scatter input per rank. These are collective input-buffer bytes, not physical wire traffic.
• Packing+MessagePack for one accepted 7173-token row across four workers, five warmups/100 samples: median 8.761→0.463 ms, p95 9.210→0.501 ms; serialized bytes 1,086,536→271,640. Lane token lengths change [7176,7176,7176,7176]→[7173,0,0,0]. This measures local preparation/serialization, not network latency.

Real RL observations, not controlled speedups

Run	Step input tokens	Reported throughput by step	Peak reserved memory by step
Padded, job 1769	170304 / 203304 / 241044	2993 / 6702 / 7803 tok/s	29.66 / 31.23 / 31.09 GiB
Unpadded, job 1771	232705 / 158309 / 215196	4812 / 5938 / 6517 tok/s	30.00 / 33.22 / 30.17 GiB
Resumed, job 1777	202839 (one further update)	5781 tok/s	32.7 GiB

Rollout admission and checkpoint writes contribute substantially to wall time. Different sampled rows and checkpoint behavior prevent interpreting this table as an end-to-end performance improvement.

Twenty-step RL follow-up — measured updates 11–20

Both fresh RL runs completed 20 optimizer updates, final weight broadcasts and final checkpoints from commit 5abeb22c1: padded job 1815 on node 011 and unpadded job 1814 on node 053. Each used an exclusive eight-H200 node with four trainer GPUs and four inference GPUs. The nodes have matching H200/driver and Xeon Platinum 8468 CPU specifications; separate-node variability remains a limitation.

The workload is Qwen/Qwen3-30B-A3B-Thinking-2507, source math / taskset i3_math, CP4 Ulysses / EP4, FA3, 32K context, completion cap 31000, batch 8/group 4 and inflight 8. Both use the original torch.compile(fullgraph=False, mode=None), full activation checkpointing/offload, native full CPU AdamW offload, lr 1e-6, and unchanged numerical settings. Both start from pretrained weights. Inference uses TP4/DP1 and seed 0; filesystem rollout transport retains exact packed batches. A final checkpoint after update 20 keeps checkpoint I/O outside the measured training loop. Runtime versions: PyTorch 2.13.0+cu130, Triton 3.7.1, FA3 3.0.0, vLLM 0.30.0, Transformers 5.6.2.

Warmup protocol, fixed before launch: updates 1–5 warmup, 6–10 settling diagnostics, 11–20 primary measurement. Each trainer reports 48 compiled layers. Every trainer rank logged one shape recompilation during update 1; none occurred in updates 2–20. NCCL INIT COMPLETE records stayed at 13 per run across all trainer ranks from startup through update 20. INIT-only diagnostics avoid per-collective logging overhead. Initial jobs using intrusive NCCL TUNING logging were cancelled before their first optimizer update and contribute no measurements. This checks startup/recompilation behavior; it does not isolate individual collective kernel latency.

Primary window: ten measured updates per run

Metric, updates 11–20	Padded	Unpadded	Interpretation
Original input tokens / trainer compute second	6468.0	6486.5	+0.29%; essentially flat in this run
Loss-mask member tokens / compute second	6433.3	6453.3	Same near-flat result
Original tokens / active trainer second, including broadcasts	5453.9	5547.5	+1.72% observed
Original tokens / total RL step second, including batch waits	529.8	481.4	−9.14% observed; rollout collection/admission differs
Median / p95 trainer compute phase	30.951 / 33.036 s	32.805 / 33.446 s	Different row counts; p95 uses nearest rank on ten observations
Mean packed rows / update	7.3	7.4	Every update contains eight original training samples
Mean packed row length	26894.7 tokens	27084.5 tokens	Workloads are similar, but not identical
Total compute seconds / packed row	4.158	4.175	+0.42%; descriptive normalization, not a matched-row benchmark
Median batch wait	305.305 s	399.844 s	End-to-end throughput is dominated by rollout waiting
Median weight broadcast	5.341 s	4.979 s	Includes synchronization and cache release before broadcasting
Maximum reported rank-0 peak reserved CUDA memory	31.465 GiB	32.053 GiB	Reserved memory, not allocated memory or an all-rank maximum
Original tokens processed in window	1963214	2004252	Different sampled rollouts
Appended CP padding tokens in window	102	0	Only 0.0052% of baseline original-token count

Compute throughput is sum(original input tokens) / sum(time/forward_backward), not an average of per-step rates. Original tokens are counted from retained batches using nonempty env_names; the baseline's 102 appended tokens are excluded. time/forward_backward includes forward/backward, optimizer/offload and scheduler work. Active time is total step time minus batch wait and checkpoint time, so it also includes broadcasts, loading and metric overhead. Total RL step time includes rollout waiting. Final checkpoint time is outside all three primary-window denominators. The broadcast timer includes the subsequent CUDA synchronization, so the compute-phase timer alone is not a synchronized end-to-end update measurement.

Late-window stability

Aggregate original tokens / compute second	Updates 11–15	Updates 16–20	Change within run
Padded	6578.5	6359.7	−3.33%
Unpadded	6584.0	6383.6	−3.04%

Neither run crosses the predeclared 10% drift flag, and neither has late recompilation or communicator initialization. Per-update latency and throughput still vary with packed-row count and row length. Recorded trainer loss, entropy, mismatch-KL and optimizer statistics remain finite. The earlier three-update 11–16% trainer-throughput deficit did not persist in this warmed-up window. The longer run establishes near-equal observed trainer throughput for this CP4/EP4 workload; it does not establish a causal speedup or regression, because the runs use independent rollouts, different policy trajectories and separate nodes. The unpadded run's lower end-to-end RL throughput must also be retained in the result; longer rollout waits and admission differences prevent attributing it to padding removal.

This topology still has one data lane, so it does not exercise removal of duplicated inactive-DP rows. Its direct CP token savings are tiny. Controlled DP2/CP2 benchmark benefits above and this CP4/EP4 result answer different workload questions.

Completion and checkpoint checks: both SLURM jobs ended COMPLETED, exit 0:0, within their original allocations (2:37:20 padded; 2:33:05 unpadded). Final checkpoint saves took 4m53s and 4m34s respectively, after the measured loop. Loading only their progress fields through distributed-checkpoint I/O verified step=20, total_samples=160, and total_tokens=3866020 padded / 4036622 unpadded, exactly matching all 20 trainer updates. This verifies the corrected saved counters; a full model/optimizer resume from these new checkpoints was not run.

The final runtime audit found no active-run trainer, orchestrator, inference or environment errors. Both inference servers logged EngineDeadError during SIGTERM cleanup after training/checkpoint completion; the launcher then exited successfully. Those shutdown messages are retained in the audit and are not presented as training failures or silently omitted.

The full curves, exact per-step CSV, window summaries, runtime audit and checkpoint-counter checks are retained locally as steady20-performance.{png,pdf}, steady20-steps.csv, steady20-summary.json, steady20-runtime-audit.json and steady20-checkpoint-counters.json under outputs/remove-pad-validation/. They are not bundled into this PR. The deployment overlays and chronological progress log remain private.

Remaining gaps and limits

1. DeepSeek full-model strict gradients fail. Compiled T9 passes (max 0.021484), but T1 fails one embedding entry at 0.037109 absolute difference against the 0.025 absolute / 0.06 relative bound. Four-repeat eager checks at T1/0/3/10/9 remain finite, with exactly zero gradients at T0, but show unsharded-reference variation up to 0.125366 and CP/reference differences up to 0.123779. Mean-gradient relative L2 is up to 1.12%; some pointwise differences exceed the sampled per-entry baseline variation plus the original bound. Using the existing tests' PyTorch norm path still varies and reaches 2.35% relative L2. The cause is not fully isolated; this is not a numerical pass or proof that all differences are harmless BF16 noise. Isolated DeepSeek attention passes separately.
2. Initialized MoE independent-reference Adam weights fail the tight gate. Objectives/counts/gradients pass, but the second update has one parameter outside the 0.0003 absolute / 0.01 relative weight comparison; maximum observed difference is about 0.0003375. Near-zero rounded gradients are a plausible contributor. Matched-gradient Adam parameters/moments pass, but that narrower check does not replace end-to-end reference agreement.
3. Entirely empty global-step rejection needs a complete-loop injection test. Empty collectives and models are tested, and the loop rejects before optimizer/scheduler work by inspection; optimizer, scheduler and counters have not been asserted unchanged through a fully injected empty transaction.
4. Full resume from a checkpoint with the corrected counters remains untested. The new twenty-update checkpoints both save step/token/sample counters matching all completed updates, verified by loading their progress fields. The prior real model/optimizer resume test used an older checkpoint; a full resume from the new checkpoints has not been run.

Additional coverage limits: no end-to-end SFT run; no multi-node benchmark; no convergence/quality conclusion from these short RL validation runs; no claim that every architecture/topology/backend combination is certified. The draft retains these gaps openly and adds no model-family restriction to bypass them.

Reproduction and artifact provenance

The checked-in distributed harness is tests/unit/train/test_cp_collectives.py. Obtain an interactive four-GPU allocation with srun, then run from the checkout:

export OMP_NUM_THREADS=2
export MKL_NUM_THREADS=2
export TORCHINDUCTOR_COMPILE_THREADS=2
uv run --no-sync torchrun --standalone --nproc-per-node=4 --no-python \
  bash -c 'export TRITON_CACHE_DIR=/tmp/remove-pad-triton-$USER-$LOCAL_RANK; exec timeout --signal=TERM --kill-after=20s 1800s uv run --no-sync pytest -q -s tests/unit/train/test_cp_collectives.py'

-k cp_exchange_and_gather_gradients selects the small exact collective subset. The command above describes the full persisted matrix; executed development runs used bounded subsets and additional diagnostics, as described above.

Implementation commit: 5abeb22c1a7075da1e402d25273dff565aa87d32. Implementation and experiments started from 88f5e6def15507b42cb520f3a7bbc57f500c0c1e. The final staged files were checked against the retained validation manifest. The original validation allocations, three-update RL jobs and twenty-update follow-up are complete; both follow-up allocations were released normally.

Raw diagnostic scripts, per-rank logs, benchmark JSON, Chrome traces and captured rollout rows remain in the author's local outputs/remove-pad-validation/; real runs are in outputs/remove-pad-runs/. Deployment-specific TOML overlays and the chronological research log remain outside the PR. These local artifacts are not bundled or remotely downloadable from this PR; the checked-in tests and the detailed results above are the published evidence. Key artifact families include core-final-ranks, metrics-calibrated-ranks, optimizer-state-ranks, init-offload-ranks, persisted-final-ranks, deepseek-{attention,variability,torch-norm}-ranks, benchmark-dp-*, benchmark-replay-init-*, benchmark-ring.json, and cudagraph-changes.json.