Skip to content
Open
Show file tree
Hide file tree
Changes from 80 commits
Commits
Show all changes
195 commits
Select commit Hold shift + click to select a range
8773090
[ops, dist] fix: compute NPU expert weight gradients in FP32
YLlllllllllll Oct 29, 2026
3d94278
[ops, dist] simplify NPU expert precision validation
YLlllllllllll Oct 29, 2026
0cb2364
[dist, optim] feat: selective expert CPU offload with resident modules
YLlllllllllll Oct 29, 2026
ebbb513
[ckpt, tests] fix: preserve CPU expert checkpoint storage
YLlllllllllll Oct 29, 2026
204465c
[ckpt] fix: match full optimizer parameter namespaces
YLlllllllllll Oct 30, 2026
166c1c9
[ops] merge: align FP32 expert gradients with instance config
YLlllllllllll Oct 30, 2026
0e0e57c
[test] use complete mixed precision fixtures
YLlllllllllll Oct 30, 2026
01d29bf
[dist] fix: use CPU for default Gloo process groups
YLlllllllllll Oct 30, 2026
b334b0f
[test] cover native default process groups
YLlllllllllll Oct 30, 2026
b1bcb87
[optim] preserve sparse optimizer parameter group state
YLlllllllllll Oct 31, 2026
e9f451f
[test, optim] cover sparse groups with FSDP2 DCP
YLlllllllllll Oct 31, 2026
93c9ab5
[checkpoint] version canonical extra-parallel model layout
YLlllllllllll Oct 31, 2026
6f0e5fd
[checkpoint] merge sparse optimizer state dependency
YLlllllllllll Nov 1, 2026
adf1951
[checkpoint] prove multi-optimizer resume identity
YLlllllllllll Oct 31, 2026
b47d315
[checkpoint] coordinate strict load preflight failures
YLlllllllllll Nov 1, 2026
c36b855
[checkpoint] publish optimizer receipt after barrier
YLlllllllllll Nov 1, 2026
54e0f32
[checkpoint] invalidate file state on load barrier failure
YLlllllllllll Nov 1, 2026
f7831f6
[ci] declare CPU plans for rank-local checkpoint tests
YLlllllllllll Nov 1, 2026
3697deb
[checkpoint] reject unsafe asynchronous multi-optimizer saves early
YLlllllllllll Nov 1, 2026
139299c
[ci] allow CUDA fixture in process-group test
YLlllllllllll Nov 1, 2026
784217d
Merge commit '139299ca0bd093bf4a3d9761d42e0673a90c4980' into HEAD
YLlllllllllll Nov 1, 2026
3f1a114
[checkpoint] preserve positional configuration compatibility
YLlllllllllll Nov 1, 2026
29797c9
[checkpoint] reject nonboolean canonical layout policy
YLlllllllllll Nov 2, 2026
1b81246
[test] schedule sparse optimizer state regressions
YLlllllllllll Nov 2, 2026
ce5fecd
[checkpoint] avoid redundant layout agreement gathers
YLlllllllllll Nov 2, 2026
6a86a20
[checkpoint] sync reviewed layout and optimizer test prerequisites
YLlllllllllll Nov 2, 2026
81fd202
[checkpoint] integrate upstream expert layout provenance
YLlllllllllll Nov 2, 2026
47d438c
[integration] preserve NPU expert precision with FP8 config
YLlllllllllll Nov 2, 2026
5c4fab0
Merge upstream FP8 expert layout with CPU offload
YLlllllllllll Nov 2, 2026
c61c733
[checkpoint] remove unsupported divergent-layout test
YLlllllllllll Nov 2, 2026
17f5903
[docs] fix SeedOmni runtime source anchors
YLlllllllllll Nov 2, 2026
8884950
[checkpoint] verify legacy expert provenance across targets
YLlllllllllll Nov 2, 2026
79309ef
Refresh remaining SeedOmni documentation links
YLlllllllllll Nov 2, 2026
a2ca5eb
Fix AnyPrecision checkpoint state dtype and ownership
YLlllllllllll Nov 2, 2026
dcc195d
Correct remaining SeedOmni documentation anchors
YLlllllllllll Nov 2, 2026
5b11c02
Test asymmetric EP expert checkpoint recovery
YLlllllllllll Nov 2, 2026
924e8ec
Reject mismatched trainer and optimizer checkpoints
YLlllllllllll Nov 2, 2026
c25cf3f
Document rank-local adapter and support nested Omni hooks
YLlllllllllll Nov 2, 2026
13745a3
Ignore transient-only cache aliases in checkpoint identity
YLlllllllllll Nov 2, 2026
a38c191
Reject unproven dynamic RoPE checkpoint recovery
YLlllllllllll Nov 2, 2026
6669054
Apply canonical checkpoint layout to Omni modules
YLlllllllllll Nov 2, 2026
4ed0d45
Keep expert cold resume coverage on production checkpoint path
YLlllllllllll Nov 2, 2026
01f6728
Merge main and preserve checkpoint layout precision
YLlllllllllll Nov 2, 2026
2b7cdcd
Simplify checkpoint guards under trusted runtime contract
YLlllllllllll Nov 2, 2026
d9232af
Preserve lazy state under PyTorch DCP optimizer wrappers
YLlllllllllll Nov 2, 2026
9438e7a
Migrate optimizer CI fixtures to proven DCP recovery
YLlllllllllll Nov 2, 2026
c4050a0
[checkpoint] docs: classify multi-optimizer resume changes
YLlllllllllll Nov 2, 2026
4e40d31
[checkpoint] fix: declare scheduler for proven resume
YLlllllllllll Nov 2, 2026
09a428a
[checkpoint] fix: validate legacy expert sources
YLlllllllllll Nov 2, 2026
02f0753
[checkpoint] fix: keep export options keyword-only
YLlllllllllll Nov 2, 2026
f73d416
[checkpoint] allow trusted scheduler instrumentation
YLlllllllllll Nov 2, 2026
478ae4c
[checkpoint] prove EP1 source and simplify layout validation
YLlllllllllll Nov 3, 2026
830d625
[checkpoint] remove redundant leaf consensus checks
YLlllllllllll Nov 3, 2026
47401eb
[checkpoint] handle ENOSYS in legacy HF exports
YLlllllllllll Nov 3, 2026
1cb551d
[checkpoint] extend FUSE fallback to DCP LoRA export
YLlllllllllll Nov 3, 2026
009e1a8
[test] cover LoRA FUSE fallback failures
YLlllllllllll Nov 3, 2026
4f6a278
[docs] clarify collective legacy HF export
YLlllllllllll Nov 3, 2026
2ae3848
[checkpoint] reuse proven MultiOptimizer export save
YLlllllllllll Nov 3, 2026
462cdac
[test] preserve same-config RoPE failure coverage
YLlllllllllll Nov 3, 2026
d18fd11
Reuse same-step exports across module checkpoint owners
YLlllllllllll Nov 3, 2026
c917429
Reuse completed exports across empty epochs
YLlllllllllll Nov 3, 2026
228c0af
Declare expert-only fixture without GDN layers
YLlllllllllll Nov 3, 2026
a267fae
Preserve proven checkpoint transactions after cold resume
YLlllllllllll Nov 3, 2026
20219eb
Fail closed on checkpoint destination lookup errors
YLlllllllllll Nov 3, 2026
f3b4328
[checkpoint] remove unsupported rank-dependent fixture
YLlllllllllll Nov 3, 2026
8a4fbf0
[checkpoint] prove custom expert layouts and safe exports
YLlllllllllll Nov 3, 2026
32871dd
[checkpoint] guard sparse and legacy extra-parallel states
YLlllllllllll Nov 3, 2026
ff5108b
[checkpoint] combine expert provenance with rank-local recovery
YLlllllllllll Nov 3, 2026
9d1341d
[checkpoint] synchronize extra-parallel provenance guards
YLlllllllllll Nov 3, 2026
9385940
[checkpoint] remove counterfeit adapter fixture
YLlllllllllll Nov 3, 2026
7d7d86d
[checkpoint] use consistent replica values in test
YLlllllllllll Nov 3, 2026
b482339
Merge commit '7d7d86db1861a38ea91d1c46f2cb3c77bc5bd722' into HEAD
YLlllllllllll Nov 3, 2026
dbef082
[test] declare absent scheduler in expert cold test
YLlllllllllll Nov 3, 2026
3745392
[checkpoint] keep configuration preflight local
YLlllllllllll Nov 3, 2026
b4bc8a1
[docs] link optimizer adapter consumers and tests
YLlllllllllll Nov 3, 2026
81c8bbe
[docs] link optimizer adapter context call sites
YLlllllllllll Nov 4, 2026
50c60ff
[checkpoint] defer dynamic RoPE checks to save and load
YLlllllllllll Nov 4, 2026
98836e0
[checkpoint] reject unproven TP expert saves early
YLlllllllllll Nov 4, 2026
bf2a37f
[docs] clarify verified checkpoint migration wording
YLlllllllllll Nov 4, 2026
8feba69
[test] cover replicated EP checkpoint preflight
YLlllllllllll Nov 4, 2026
6d5f772
[docs] fix checkpoint manifest field link
YLlllllllllll Nov 4, 2026
a50fdb6
[test] keep TP checkpoint checks within model contract
YLlllllllllll Nov 4, 2026
3eef397
[checkpoint] test: enforce native preflight capacity
YLlllllllllll Nov 4, 2026
366fa23
[checkpoint] prove TP FSDP source layouts
YLlllllllllll Nov 4, 2026
e3e3350
[test] remove preflight guard-only cases
YLlllllllllll Nov 4, 2026
1768cca
[checkpoint] merge TP FSDP provenance guard
YLlllllllllll Nov 4, 2026
3e67bc4
[test] schedule TP provenance on its actual backend
YLlllllllllll Nov 4, 2026
2ba4cf6
[test] derive checkpoint launch plans from geometry
YLlllllllllll Nov 4, 2026
d199b7e
Merge branch 'review-fix-pr1396' of /tmp/veomni-review-20261104/pr139…
YLlllllllllll Nov 4, 2026
6dd9407
[test] fail assigned accelerator group shortages
YLlllllllllll Nov 4, 2026
9be7026
[test] remove accelerator skip meta tests
YLlllllllllll Nov 4, 2026
4800b5e
[test] use CPU plan for expert layout tests
YLlllllllllll Nov 4, 2026
f6a449c
[test] use CPU plan for expert layout tests
YLlllllllllll Nov 4, 2026
845ccfd
[checkpoint, model] fix: preserve safe RoPE reset and dense layouts
YLlllllllllll Nov 4, 2026
0c9002d
[checkpoint] fix: bind dynamic RoPE table bytes on resume
YLlllllllllll Nov 4, 2026
1a96e9b
[checkpoint] reject nonfinite RoPE tables without changing FP8 identity
YLlllllllllll Nov 4, 2026
80f61cf
[checkpoint] limit RoPE finiteness checks to supported types
YLlllllllllll Nov 4, 2026
f1b3e03
[checkpoint] document reserved runtime state keys
YLlllllllllll Nov 4, 2026
e4bdb9e
Merge upstream main and retain both feature paths
YLlllllllllll Nov 5, 2026
d075345
[docs] describe file-backed checkpoint abort hook
YLlllllllllll Nov 5, 2026
be924f2
Merge main and preserve extra-parallel CPU layouts
YLlllllllllll Nov 5, 2026
34aa016
Merge main and guard expert offload checkpoint layouts
YLlllllllllll Nov 5, 2026
8c97fbc
[checkpoint] unify scheduler identity validation
YLlllllllllll Nov 5, 2026
230f18c
[checkpoint] Coordinate EP optimizer type preflight
YLlllllllllll Nov 5, 2026
ed055f7
[checkpoint] Coordinate optimizer type lookup errors
YLlllllllllll Nov 5, 2026
3f7002e
[checkpoint] Coordinate EP preflight and align layouts
YLlllllllllll Nov 5, 2026
cccc1df
[checkpoint] Validate nested scheduler identity
YLlllllllllll Nov 5, 2026
ed21c45
Merge commit 'e6486d3847536d6a5cf72a433040914b69f1957d' into HEAD
YLlllllllllll Nov 5, 2026
6384c62
[checkpoint] fix standalone SwapAdamW recovery guards
YLlllllllllll Nov 5, 2026
82fbfc9
[checkpoint] constrain scheduler identity to trusted runtime
YLlllllllllll Nov 5, 2026
be814f4
[checkpoint] fix: validate versioned expert ownership
YLlllllllllll Nov 5, 2026
37b473e
[checkpoint] simplify extra-parallel optimizer preflight
YLlllllllllll Nov 6, 2026
95d0855
[checkpoint] simplify extra-parallel optimizer preflight
YLlllllllllll Nov 6, 2026
a64fd96
[checkpoint] simplify multi-optimizer leaf coordination
YLlllllllllll Nov 6, 2026
8424b85
[checkpoint] support deterministic scheduler formatting
YLlllllllllll Nov 6, 2026
ec34eef
[docs] refresh checkpoint configuration references
YLlllllllllll Nov 6, 2026
5a5520d
[test] fix SeedOmni offload config fixture
YLlllllllllll Nov 6, 2026
a656e43
[docs] fix checkpoint and offline cache anchors
YLlllllllllll Nov 6, 2026
3d0b28e
[checkpoint] integrate canonical expert provenance
YLlllllllllll Nov 6, 2026
a2708c3
[test] retain historical checkpoint rejection coverage
YLlllllllllll Nov 6, 2026
be8195a
[checkpoint] coordinate optimizer policy preflight
YLlllllllllll Nov 6, 2026
ada389b
Keep FUSE export coordination composable
YLlllllllllll Nov 6, 2026
a1ef83f
Bind FUSE adapter fallback independently of export flow
YLlllllllllll Nov 6, 2026
1fefa13
fix: compose expert precision with parallel patch paths
YLlllllllllll Nov 6, 2026
759e6fb
fix: coordinate cursor and optimizer checkpoint recovery
YLlllllllllll Nov 6, 2026
ad278e3
ci: isolate selective expert test entries
YLlllllllllll Nov 6, 2026
ed45ea5
fix: coordinate invalid checkpoint snapshots
YLlllllllllll Nov 6, 2026
9a8f1fe
fix: coordinate fatal optimizer generation failures
YLlllllllllll Nov 6, 2026
6f867fe
docs: clarify optimizer placement resume identity
YLlllllllllll Nov 6, 2026
9e07fec
fix: keep adapter save import mergeable with GDN export
YLlllllllllll Nov 6, 2026
4392697
fix: compose selective offload guard with optimizer setup
YLlllllllllll Nov 6, 2026
8f1b7d3
test: run selective optimizer guard in existing CI suite
YLlllllllllll Nov 6, 2026
d03c6c3
Merge selective expert CPU offload with FP32 expert gradients
YLlllllllllll Nov 7, 2026
2202497
Merge owner-aware checkpoint with selective expert offload
YLlllllllllll Nov 7, 2026
1b0f552
Clarify selective expert offload resume requirements
YLlllllllllll Nov 7, 2026
fa072c6
Merge offload documentation correction
YLlllllllllll Nov 7, 2026
2f0012d
Synchronize offload documentation dependency
YLlllllllllll Nov 7, 2026
6d520d9
Initialize checkpoint manager config in failure fixture
YLlllllllllll Nov 7, 2026
135f718
[docs] clarify legacy HF export collective
YLlllllllllll Nov 9, 2026
a47732c
[docs] clarify SeedOmni FP32 expert gradient scope
YLlllllllllll Nov 9, 2026
5c2837c
[docs] clarify checkpoint resume and source links
YLlllllllllll Nov 9, 2026
158d83e
[tests] fix: resolve checkpoint export CLI path
YLlllllllllll Nov 9, 2026
c051d9c
[checkpoint] clarify existing save destination
YLlllllllllll Nov 9, 2026
cfafa43
[test] guard invalid prefixed dynamic rope cache
YLlllllllllll Nov 10, 2026
f111b91
[ci, tests] require assigned NPU and fix precision fixture
YLlllllllllll Nov 10, 2026
f222247
[checkpoint] bind model storage views for resume
YLlllllllllll Nov 10, 2026
543ae79
Merge remote-tracking branch 'upstream/main' into HEAD
YLlllllllllll Nov 10, 2026
e63ffa5
Merge remote-tracking branch 'upstream/main' into HEAD
YLlllllllllll Nov 10, 2026
25a42ab
Merge upstream main
YLlllllllllll Nov 11, 2026
14e5e51
Merge upstream main
YLlllllllllll Nov 11, 2026
5926c99
[docs] fix HF export and cursor test anchors
YLlllllllllll Nov 11, 2026
1687b82
[test] exercise checkpoint storage aliases on accelerators
YLlllllllllll Nov 11, 2026
3ab98ca
Merge upstream MoE operator interface
YLlllllllllll Nov 12, 2026
efe34c2
Merge upstream checkpoint layouts and validate EP policy
YLlllllllllll Nov 12, 2026
f05a537
Normalize persistent shard axes in checkpoint preflight
YLlllllllllll Nov 12, 2026
e14b678
Check expert provenance before checkpoint overwrite
YLlllllllllll Nov 12, 2026
d3852e7
Merge split-layout checkpoint and model updates
YLlllllllllll Nov 12, 2026
81ed593
Merge verified checkpoint provenance guards
YLlllllllllll Nov 12, 2026
ca843be
Merge NPU MoE precision interface coverage
YLlllllllllll Nov 12, 2026
99ab50e
Refresh SeedOmni documentation source anchors
YLlllllllllll Nov 12, 2026
7350de0
Merge main and protect HF adapter publication
YLlllllllllll Nov 17, 2026
2ac0a21
Merge main while preserving NPU expert precision
YLlllllllllll Nov 17, 2026
1daef50
Merge HF export and preserve canonical EP PLE layouts
YLlllllllllll Nov 17, 2026
a8f8824
Merge canonical EP PLE export with proven optimizer recovery
YLlllllllllll Nov 17, 2026
f227c1d
Merge NPU expert checkpoint safety updates
YLlllllllllll Nov 17, 2026
2e2acdb
Bound merged expert weight copies during backward
YLlllllllllll Nov 17, 2026
87f806b
Reject ambiguous optimizer instance serializers
YLlllllllllll Nov 17, 2026
0fbb62b
Honor assigned NPU in expert precision tests
YLlllllllllll Nov 17, 2026
33b9364
Respect the NPU test device lease
YLlllllllllll Nov 17, 2026
2ca7be6
Merge expert rematerialization and retain device lease guards
YLlllllllllll Nov 17, 2026
a5d3c77
Check owned NPU capacity for distributed tests
YLlllllllllll Nov 17, 2026
0abc35c
Sync native expert test lease guards
YLlllllllllll Nov 17, 2026
f6d7e5a
Bind scheduler checkpoint identity to declared recipes
YLlllllllllll Nov 17, 2026
a8b8f27
Assign CPU plan to canonical PLE checkpoint tests
YLlllllllllll Nov 17, 2026
343c02c
Complete optimizer layout fixture for distributed preflight
YLlllllllllll Nov 17, 2026
61b9339
Use checkpoint converter under consistent model config
YLlllllllllll Nov 17, 2026
a0423f7
Reuse checkpoint load error coordination for row metadata
YLlllllllllll Nov 17, 2026
f333cb1
Use checkpoint converter under consistent model config
YLlllllllllll Nov 17, 2026
e88ade0
Merge upstream gated MLP with expert precision support
YLlllllllllll Nov 18, 2026
fa7ef1e
Merge upstream and align checkpoint runtime contracts
YLlllllllllll Nov 18, 2026
ae131f7
Clarify unsupported model storage views
YLlllllllllll Nov 18, 2026
dc17e6d
Reject missing scheduler sidecars during training resume
YLlllllllllll Nov 18, 2026
983faf7
Reject incomplete scheduler identity during optimizer resume
YLlllllllllll Nov 18, 2026
865d5de
test: align checkpoint failure fixtures with CI contract
YLlllllllllll Nov 18, 2026
add3d66
Fix checkpoint documentation source anchors
YLlllllllllll Nov 18, 2026
8a8ae7a
Fix checkpoint failure injection fixtures
YLlllllllllll Nov 18, 2026
cd898b2
Fix checkpoint device test and documentation links
YLlllllllllll Nov 19, 2026
6a111c7
Fix Float8 configuration documentation link
YLlllllllllll Nov 19, 2026
0986dff
[model, ckpt] fix: align Janus weight source priority
YLlllllllllll Nov 19, 2026
a46acd1
[model, ops] fix: expand unsupported Ascend GDN head ratios
YLlllllllllll Nov 19, 2026
e349e38
[checkpoint] merge: align upstream model and resume safeguards
YLlllllllllll Nov 19, 2026
a3670d7
[checkpoint] merge: align HF source priority
YLlllllllllll Nov 19, 2026
d772a03
[checkpoint] merge: retain Ascend GDN ratio fallback
YLlllllllllll Nov 19, 2026
09c471b
[docs] clarify Ascend GDN head expansion
YLlllllllllll Nov 19, 2026
96bd7c5
[docs] merge: clarify Ascend GDN head expansion
YLlllllllllll Nov 19, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .agents/knowledge/constraints.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ Core entry points:
- Token routing: `veomni/distributed/moe/moe_layer.py` — `preprocess()` computes dispatch counts, `token_pre_all2all()` / `tokens_post_all2all()` exchange tokens between EP ranks via `all_to_all` / `all_to_all_async` in `moe/comm.py`.
- Expert computation: `EPGroupGemm` runs fused expert MLP on grouped tokens per rank.
- Device mesh: `init_parallel_state_from_config()` builds `[ep × ep_fsdp]` submesh; accessed via `ParallelState.extra_parallel_mesh("ep")`, `ep_group`, `ep_rank`.
- In FSDP2: expert modules get `fully_shard()` on the `ep_fsdp` submesh with `Shard(1)` by default. `expert_fsdp_shard_dim=0` explicitly selects whole-expert FSDP shards after validating the EP plan and even local expert divisibility; `None` preserves the existing placement/Muon policy. Record the selected FSDP dimension in `SpecInfo`. If EP/FSDP split the same tensor dimension, DCP must use `_StridedShard(dim, split_factor=ep_size)` on the FSDP mesh axis: physical sharding is EP-first even though the mesh lists FSDP first. Two ordinary `Shard(0)` placements silently permute global expert IDs. VeOmni DCP save/load planners record per-tensor source EP/FSDP dimensions, sizes and expert order in `planner_data`; load compatibility depends on saved coordinates, not the target shard dimension. Unmarked legacy chunks that split only the expert axis are ambiguous and must be migrated using the original training layout; do not bypass the planners for production resumes.
- In FSDP2: expert modules get `fully_shard()` on the `ep_fsdp` submesh with `Shard(1)` by default. `expert_fsdp_shard_dim=0` explicitly selects whole-expert FSDP shards after validating the EP plan and even local expert divisibility; `None` preserves the existing placement/Muon policy. Record the selected FSDP dimension in `SpecInfo`. If EP/FSDP split the same tensor dimension on an FSDP-first mesh, DCP must use `_StridedShard(dim, split_factor=ep_size)` on the FSDP axis: physical sharding is EP-first. Two ordinary `Shard(0)` placements on that mesh silently permute global expert IDs. For the namespaced v2 layout, an EP-uneven global axis instead uses an EP-first mesh view with ordinary shards, since Torch strided metadata can give incorrect offsets for that case. VeOmni DCP save/load planners record per-tensor source EP/FSDP dimensions, sizes and expert order in `planner_data`; load compatibility depends on saved coordinates, not the target shard dimension. Unmarked legacy chunks that split only the expert axis are ambiguous and must be migrated using the original training layout; do not bypass the planners for production resumes.

- LoRA adapter export must pass both `SpecInfo.placement.dim` and `SpecInfo.fsdp_shard_dim` to `restore_extra_parallel_dim`, just like regular model checkpoints. Assuming dim-1 FSDP for whole-expert shards corrupts global adapter shapes and expert order during safetensors consolidation.

Expand Down
136 changes: 136 additions & 0 deletions docs/design/rank_local_adamw_checkpoint_adapter.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# Rank-local AdamW checkpoint adapters

`register_rank_local_adamw` is an internal extension point for an AdamW-like
optimizer whose master weights or moments must retain their own dtype or CPU
placement when restored through `MultiOptimizer` and `DistributedCheckpointer`.
It is not a generic optimizer plugin, a security boundary, or a conversion tool
for an incompatible checkpoint. Register trusted code only.

## Registration and ownership

```python
from contextlib import nullcontext

import torch

from veomni.optim.checkpoint_adapter import register_rank_local_adamw

# Illustrative callbacks for an AdamW-like optimizer that already implements
# precision-preserving state loading; MyAdamW is the application's own class.
register_rank_local_adamw(
MyAdamW,
context_factory=lambda optimizer: nullcontext(),
algorithm_identity=lambda optimizer: ("my-adamw-v1", str(optimizer.moment_dtype)),
state_dtype=lambda optimizer, param, field: (
torch.float32 if field in ("step", "master_weight") else optimizer.moment_dtype
),
state_is_offloaded=lambda optimizer, param: optimizer.is_offloaded(param),
)
```

The registry is process-local and matches `type(optimizer)` **exactly**;
register each supported concrete optimizer type before saving or loading on
every rank. Duplicate registration raises an error. Registration does not make
an arbitrary optimizer compatible or automatically preserve its precision: the
optimizer and its adapter own the correctness of their serialization and load
behavior. The in-tree checkpoint adapter tests provide an illustrative
registered implementation.

An initialized parameter entry must contain exactly `step`, `master_weight`,
`exp_avg`, and `exp_avg_sq`. The step is a rank-local plain scalar tensor, not
a scalar DTensor; the other fields have the parameter's shape. A parameter with
no optimizer state must stay absent, rather than receiving synthetic moments
or a synthetic step. The current adapter contract does not support additional
per-parameter fields.

## Callback contract

| Callback | Signature and result | Responsibility |
| --- | --- | --- |
| `context_factory` | `context_factory(optimizer) -> context manager` | Entered while the framework obtains or applies that leaf optimizer's state. It may temporarily select an optimizer's supported serialization mode, but must expose the live initialized fields and group settings without mutating training state. On restore, it must allow the optimizer's loader to install the saved precision and placement without an implicit parameter-dtype cast. `nullcontext()` is sufficient only when the optimizer already has that behavior. |
| `algorithm_identity` | `algorithm_identity(optimizer) -> immutable value` | Describes non-group settings that change updates or restore semantics. Use `None`, booleans, integers, floats, strings, or tuples recursively composed of those types; encode dtypes as strings. Include an algorithm/schema version and all such settings. The checkpoint records this value and rejects a different identity on resume. Group settings and scheduler identity are checked separately. |
| `state_dtype` | `state_dtype(optimizer, param, field) -> torch.dtype` | Returns the expected stored dtype for each of `step`, `master_weight`, `exp_avg`, and `exp_avg_sq`, including when the state is not initialized yet. `step` describes the scalar dtype, not the moment dtype. The framework records the policy and validates existing and restored tensors against it; this callback does not cast tensors. |
| `state_is_offloaded` | `state_is_offloaded(optimizer, param) -> bool` | Reports whether that parameter's master and moment state belongs on CPU. The per-parameter policy is part of checkpoint identity and selects CPU versus parameter-device allocation during restoration. The scalar step follows the optimizer group's capturable/fused placement rules separately. |

Callbacks must be deterministic for the same training configuration and
stable across the save and load. `optimizer` is the live concrete leaf, and
`param` is a parameter owned by that leaf, possibly a DTensor. A fresh process
constructs new optimizer and parameter objects; callbacks must reconstruct the
same policies for them. Parameter ownership, optimizer group
membership, scheduler identity, state dtypes and offload selection must match
the saved checkpoint. Change these only with an explicitly verified migration.

### Save and load call sites

[`_local_layout`](../../veomni/checkpoint/optimizer_resume_layout.py#L105)
records `algorithm_identity`, each parameter's `state_dtype`, and
`state_is_offloaded` in the checkpoint identity; a mismatch rejects resume
before applying state. During save,
[`_validate_source_optimizer_states`](../../veomni/checkpoint/multi_optimizer_checkpoint.py#L89)
uses `state_dtype` to check initialized fields against their declared precision.
During load,
[`_prepare_multi_optimizer_saved_states`](../../veomni/checkpoint/multi_optimizer_checkpoint.py#L408)
checks stored field dtypes and uses `state_is_offloaded` to allocate missing
destination state on CPU when the adapter specifies offload.
[`rank_local_adamw_context`](../../veomni/optim/checkpoint_adapter.py#L55)
uses `context_factory` around the optimizer's actual state read and load;
adapter implementations own correct placement and values.

The adapter contract tests include the
[`test_registered_optimizer_preserves_lazy_state_and_next_update`](../../tests/checkpoints/test_optimizer_checkpoint_adapter.py#L71)
round trip and the policy-mismatch checks
[`test_registered_optimizer_rejects_changed_algorithm_before_loading`](../../tests/checkpoints/test_optimizer_checkpoint_adapter.py#L114)
and
[`test_lazy_adapter_state_dtype_policy_is_part_of_checkpoint_identity`](../../tests/checkpoints/test_optimizer_checkpoint_adapter.py#L132).
The toy adapter in these tests reports no offload; actual device-to-CPU placement
must also be validated for an offloading implementation.

## Framework call sites and restore behavior

1. [`MultiOptimizer._sparse_state_dict()`](../../veomni/optim/optimizer.py#L563)
enters the context around PyTorch's
optimizer-state serialization. Registered adapter implementations must
preserve live state and serialize all initialized fields and group options
faithfully. State-empty leaves retain their parameter groups without
creating optimizer moments.
2. Checkpoint layout construction records the algorithm identity, parameter
ownership, dtype policy and offload policy. Save validates the source state;
load checks the checkpoint's identity, field names, dtypes and shapes before
preparing initialized state for the saved owners. Rank-local ownership and
the supported parallel geometry are not automatically resharded.
3. [`MultiOptimizer._dcp_load_state_dict()`](../../veomni/optim/optimizer.py#L601)
enters the context again while
PyTorch applies the optimizer state. The trusted adapter must restore every
validated field without changing its value or dtype. An optimizer may
implement its own precision-preserving loader; the adapter callbacks alone
do not repair a lossy one.

The currently supported exact-precision training-checkpoint path requires
`MultiOptimizer` and synchronous `DistributedCheckpointer` save/load.
`MultiOptimizer.load_state_dict()` and raw DCP `OptimizerState` reject
unproven recovery; an individual leaf's direct `load_state_dict()` cannot prove
the MultiOptimizer group, scheduler and generation identity. Unverified older
group layouts and changing the rank-local layout
require an explicit verified migration or fail closed. See
[MultiOptimizer recovery identity](../usage/checkpoint.md#multioptimizer-recovery-identity)
for the complete recovery constraints. Test an extension with initialized and
unused parameters, save, fresh-process restore, and the next update at its
actual dtype and placement; an adapter registration or a single forward pass
is not an acceptance test.

## Optional file-backed state abort hook

An optimizer that owns file-backed state may provide the zero-argument
`checkpoint_abort_file_backed_state()` hook. If the final distributed barrier
in `ModelCheckpointManager.load()` fails after loading state, the manager
calls this hook on the optimizer and on each `MultiOptimizer` leaf that defines
it. The hook owns its optimizer's cleanup or invalidation: it should make any
partially restored file-backed state unusable until the optimizer is rebuilt
or explicitly recovered. It must not perform collectives and should be safe
to call more than once. The manager ignores ordinary cleanup exceptions so the
original barrier failure is propagated; it does not report the checkpoint as
successfully loaded. Optimizers without file-backed state need no hook.

The manager calls this hook only on its final-barrier failure path; it is not a
general notification for every earlier load error. File-backed optimizers and
checkpointers remain responsible for cleanup at their own failure sites.
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,7 @@ seed_omni/mixins/offline_encoding.md
design/ci_case_scheduling.md
design/op_selection.md
design/module_config.md
design/rank_local_adamw_checkpoint_adapter.md
design/fused_moe_ops.md
design/deepgemm_fp8_moe.md
design/local_parallel_state.md
Expand Down
Loading
Loading