Repository navigation
[Dev][feat] Support CUDA Graph capture offloading modules #3219
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
lhb8125
merged 125 commits into
NVIDIA:dev
from
lhb8125:hongbinl/activation_offloading_refactor_cuda_graph
Mar 30, 2026
Merged
Changes from 101 commits
Commits
Show all changes
125 commits
Select commit
Hold shift + click to select a range
1219a26
renaming golden values
lhb8125 ce6e661
fix bug: accuracy issu because of recomputing and offloading same module
lhb8125 d04d741
Merge branch 'dev' into hongbinl/activation_offloading_fix
lhb8125 2fe4aeb
format
lhb8125 fb3f7c3
update golden values
lhb8125 5001e2b
Merge branch 'dev' into hongbinl/activation_offloading_fix
lhb8125 9937890
update golden values
lhb8125 6c83118
update model_config and golden values
lhb8125 33a38f5
format
lhb8125 6c76b07
update golden values
lhb8125 4d83f69
Merge branch 'dev' into hongbinl/activation_offloading_fix
lhb8125 8e72b44
temp save
lhb8125 1646f04
support offloading+cuda graph
lhb8125 43973a7
Merge branch 'dev' into hongbinl/activation_offloading_cuda_graph
lhb8125 a177cf5
support PP=1
lhb8125 f7cfbba
support VPP
lhb8125 6d475ad
bug fix
lhb8125 089da6c
support VPP
lhb8125 35b0f97
code refactor
lhb8125 df09b85
big code refactor and format
lhb8125 06ef4e2
Merge branch 'dev' into hongbinl/activation_offloading_cuda_graph
lhb8125 12cb8de
minor fix
lhb8125 3cf19b7
minor fix
lhb8125 d0fc888
dump offloading information
lhb8125 bc47650
Merge branch 'dev' into hongbinl/activation_offloading_cuda_graph
lhb8125 b7c0fba
fix ut
lhb8125 b18e69b
Merge branch 'hongbinl/activation_offloading_cuda_graph' of https://g…
lhb8125 ae4e2b5
format
lhb8125 b797438
fit ut
lhb8125 6cec22f
delay d2h copies until finishing cuda graph
lhb8125 5a150c7
minor fix
lhb8125 e6cf8b5
Merge branch 'dev' into hongbinl/activation_offloading_cuda_graph
lhb8125 60e3082
format
lhb8125 256d79d
fix ut
lhb8125 9475e3d
format
lhb8125 6daa2a4
Merge branch 'dev' into hongbinl/activation_offloading_cuda_graph
lhb8125 93c0827
minor fix
lhb8125 f22a194
remove changes for cuda graph
lhb8125 a9d6633
bug fix when cuda graph is disabled and fix for dumping offloading info
9d766e9
refactor and update ut
lhb8125 9d1fe34
format
lhb8125 08b46aa
fix ut
lhb8125 d33b3c4
update ut
lhb8125 884c335
update ut
lhb8125 df2e839
fix ut
lhb8125 569f347
add version check
lhb8125 7f8109a
minor refactor for fine_grained_activation_offload.py
lhb8125 19b35c3
format
lhb8125 e867835
Merge branch 'dev' into hongbinl/activation_offloading_refactor
lhb8125 dd874a9
support partial cuda graph
lhb8125 b8c0b79
fix bug when working with a2a overlap and cuda graph
lhb8125 994fc5a
support offloading less for large pp rank
lhb8125 2688c7e
fix doc
lhb8125 aab8455
code refactor
lhb8125 0a92566
remove group_start() calls
lhb8125 0df5134
add comments
lhb8125 62d36f2
fix min_offload_size and update golden values
lhb8125 17c0eb9
minor fix and format
lhb8125 d979f1e
Merge branch 'hongbinl/activation_offloading_refactor' into hongbinl/…
lhb8125 53a21e1
rename group_commit
lhb8125 600cfe7
fix for graph support
lhb8125 09956df
refine offloading strategy
lhb8125 18a2d9e
temp fix for mxfp8
lhb8125 885164b
minor fix
lhb8125 688c2ab
support offloading fraction
lhb8125 2b0276c
free input of mlp when fp8
lhb8125 0c901a8
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 e606424
minor fix and format
lhb8125 0cda334
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 0f0d1ed
fix ut
lhb8125 254a2d2
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 4ba07bd
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 e585f60
Update arguments.py
lhb8125 1b8050f
fix ut
lhb8125 410f879
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 162e388
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 504b3a4
update ut and minor refactor
lhb8125 01e2ad9
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 526fc35
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 6264494
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 7640bf3
minor refactor
lhb8125 15641c4
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 dfc8161
format
lhb8125 e362f79
fix ut
lhb8125 2da15b2
format
lhb8125 221cc16
minor fix
lhb8125 726c526
format and minor fix
lhb8125 f19c4f6
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 998d1b0
1. replace hasattr+delattr with None;
lhb8125 4bf0085
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 716e12a
format
lhb8125 cd84623
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 61b589a
bug fix
lhb8125 0200121
add flag to control flush_delayed_groups in fine_grained_callables.py
lhb8125 c8bd90d
1. move backward_record() to te_cuda_graph_capture()
lhb8125 ddd67d2
format
lhb8125 e989b95
remove the knob forward_only when executing reset()
lhb8125 b481fa9
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 19fe6b3
fix ut and reviewer's comments
lhb8125 cf04b4a
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 ce84682
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 d8da479
resolve comments
lhb8125 a6e16a9
remove redundant offloading doc
lhb8125 0095a94
add assertion for recomputing and offloading
lhb8125 2d8d03c
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 f8a6c92
fix ut
lhb8125 5932846
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 35c28eb
format and fix TE version
lhb8125 20c91f7
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 e5f154b
replace multiple imports with a _get_off_interface()
lhb8125 701f831
renaming _get_offloading_interface
lhb8125 a70f801
minor fix
lhb8125 8959afb
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 2a1afc1
fix mhc layer
lhb8125 bb4ac50
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 92ea308
fix comments
lhb8125 51ba05a
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 837dba4
fix ci
lhb8125 c56acec
fix ut
lhb8125 d78b776
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 a3b758e
refine docs and minor fix
lhb8125 b041fca
Merge branch 'hongbinl/activation_offloading_refactor_cuda_graph' of …
lhb8125 ae46402
minor fix
lhb8125 2542129
minor fix
lhb8125 0cef457
Merge branch 'dev' into hongbinl/activation_offloading_refactor_cuda_…
lhb8125 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,31 +1,141 @@ | ||
| # Fine-grained Activation Offloading (collaborated with rednote) | ||
| # Fine-Grained Activation Offloading | ||
|
|
||
| Memory capacity is more and more important with the rising of extreme sparse MoE models like DeepSeek-V3 and Qwen3-235B. Fine-grained recomputing reduces the memory footprint at the cost of extra recomputation, while offloading could utilize the host-device bandwidth to achieve nearly zero-overhead. Fine-grained Activation Offloading targets at offloading the activation at the granularity of specific modules, so that we can calibrate the amount of offloading activation to maximize the training throughput. | ||
| Fine-grained activation offloading reduces GPU memory by asynchronously transferring activations to CPU at the granularity of individual submodules within a transformer layer. Unlike layer-level offloading, it allows precise control over which activations to offload, enabling a tradeoff between memory savings and PCIe bandwidth overhead. | ||
|
|
||
| Currently, the supported offloading modules are `"attn_norm", "core_attn", "attn_proj", "mlp_norm", "expert_fc1", "moe_act"`, which could work with fine-grained recomputation to release almost all activations of a transformer layer. | ||
| ## User Guide | ||
|
|
||
| **Features** | ||
| * Support PP=1/PP/Interleaved PP | ||
| * Compatible with fine-grained recomputation | ||
| * Support FP8 | ||
| * Support MTP | ||
| * Support mixed dense & moe layer | ||
| * Support A2A Overlap | ||
| * Support CUDA Graph | ||
| * (Temporary) cuda graph scope cannot contains the offloading modules | ||
| ### Basic Usage | ||
|
|
||
| **Usage** | ||
| ```bash | ||
| # Enable fine-grained activation offloading | ||
| --fine-grained-activation-offloading | ||
|
|
||
| # Specify which modules are going to offload its input | ||
| # Choices: "attn_norm", "core_attn", "attn_proj", "mlp_norm", "expert_fc1", "moe_act". | ||
| --offload-modules expert_fc1 | ||
| # Specify which modules to offload (can combine multiple) | ||
| # Choices: attn_norm, qkv_linear, core_attn, attn_proj, mlp_norm, expert_fc1, moe_act | ||
| --offload-modules core_attn attn_proj expert_fc1 | ||
| ``` | ||
|
|
||
| ### Offloadable Modules | ||
|
|
||
| Each module offloads its **input** activation to CPU during forward and reloads it before backward: | ||
|
|
||
| | Module | Description | Notes | | ||
| |---|---|---| | ||
| | `attn_norm` | Input layernorm of attention | Skipped if using `IdentityOp` | | ||
| | `qkv_linear` | QKV linear projection | | | ||
| | `core_attn` | Core attention (softmax + matmul) | | | ||
| | `attn_proj` | Output projection of attention | Must be used together with `core_attn` | | ||
| | `mlp_norm` | Pre-MLP layernorm | Skipped if using `IdentityOp` | | ||
| | `expert_fc1` | First FC layer in MoE experts | MoE models only | | ||
| | `moe_act` | Activation function in MoE experts | MoE models only | | ||
|
|
||
| ### Tuning Parameters | ||
|
|
||
| ```bash | ||
| # Minimum tensor size (in elements) to offload. Smaller tensors are skipped. | ||
| # Default: 1048576 (1M elements) | ||
| --min-offloaded-tensor-size 1048576 | ||
|
|
||
| # Fraction of activations to offload, range [0, 1]. Default: 1.0 | ||
| # Useful for partial offloading when PCIe bandwidth is a bottleneck. | ||
| --activation-offload-fraction 0.8 | ||
|
|
||
| # Reduce offload amount on higher PP ranks (in bytes). Default: 0 | ||
| # Higher PP ranks have fewer microbatches in flight, so offloading less | ||
| # reduces overhead without increasing peak memory. | ||
| --delta-offload-bytes-across-pp-ranks 1073741824 | ||
| ``` | ||
|
|
||
| ### CUDA Graph Integration | ||
|
|
||
| Fine-grained offloading is compatible with CUDA graphs. When CUDA graph is enabled, the following constraints apply: | ||
|
|
||
| - `attn_norm` and `mlp_norm` **cannot** be offloaded (they cross CUDA graph boundaries). | ||
| - `cuda_graph_scope` must include `attn` and `moe_router`. | ||
|
lhb8125 marked this conversation as resolved.
Resolved
|
||
| - `cuda_graph_impl` must be `transformer_engine`. | ||
| - Requires `torch >= 2.9.0` and `transformer_engine >= 2.13.0`. | ||
|
|
||
| ```bash | ||
| # Delay offloading until CUDA graph launch to hide CPU overhead | ||
| --delay-offload-until-cuda-graph | ||
| ``` | ||
|
|
||
| ### Combining with Fine-Grained Recomputation | ||
|
|
||
| Offloading and recomputation are complementary: | ||
| - Use **recomputation** for lightweight modules (e.g., layernorm, activation functions) with negligible compute overhead. | ||
| - Use **offloading** for heavy modules (e.g., core_attn, expert_fc1) where recomputation would be too costly. | ||
|
|
||
| ```bash | ||
| --recompute-granularity selective | ||
| --recompute-modules layernorm moe_act | ||
| --fine-grained-activation-offloading | ||
| --offload-modules core_attn attn_proj expert_fc1 | ||
| ``` | ||
| **Compatible with Fine-grained Recomputation** | ||
| - For modules with minor perf overhead like layernorm or moe_act, use recomputing to reduce memory footprint; | ||
| - For other modules, use offloading to reduce memory footprint; | ||
| - Make sure the offloading/reloading could be overlapped with computing; | ||
|
|
||
|  | ||
|
|
||
|
|
||
| ### Compatibility | ||
|
|
||
| | Feature | Supported | | ||
| |---|---| | ||
| | PP / Interleaved PP / PP=1 | Yes | | ||
| | Fine-grained recomputation | Yes | | ||
| | FP8 training | Yes | | ||
| | MTP (Multi-Token Prediction) | Yes | | ||
| | Mixed dense & MoE layers | Yes | | ||
| | A2A overlap (EP) | Yes | | ||
| | CUDA Graph (TE impl) | Yes | | ||
|
|
||
| --- | ||
|
|
||
| ## How It Works | ||
|
|
||
| ### Architecture Overview | ||
|
|
||
| The implementation consists of three layers: | ||
|
|
||
| 1. **`PipelineOffloadManager`** (singleton): Global coordinator that manages CUDA streams, CPU tensor pools, and chunk lifecycle across pipeline stages. | ||
| 2. **`ChunkOffloadHandler`**: Per-microbatch handler that tracks tensor groups, executes D2H/H2D transfers, and decides which groups to actually offload. | ||
| 3. **`FineGrainedActivationOffloadingInterface`**: Lightweight interface used by transformer modules (attention, MoE, etc.) to mark offload boundaries. | ||
|
|
||
| ### Offload/Reload Flow | ||
|
|
||
| ``` | ||
| Forward pass (Layer N): Backward pass (Layer N): | ||
| ┌─────────────────────┐ ┌───────────────────────┐ | ||
| │ group_start(input) │─── register ──► │ │ | ||
| │ │ tensor group │ group_commit_backward │ | ||
| │ module.forward() │ │ wait H2D complete │ | ||
| │ │ │ pop tensors from │ | ||
| │ group_offload(out) │─── D2H async ──► │ CPU → GPU │ | ||
| │ on d2h_stream │ to pinned CPU │ on h2d_stream │ | ||
| └─────────────────────┘ └───────────────────────┘ | ||
| ``` | ||
|
|
||
| 1. **`group_start`**: Registers a new tensor group and hooks into `saved_tensors_hooks` to intercept `save_for_backward`. | ||
| 2. **Forward execution**: All tensors saved by autograd within the group are captured. | ||
| 3. **`group_offload`**: Triggers asynchronous D2H copy on a dedicated CUDA stream (`d2h_stream`), optionally releases GPU storage of input tensors. | ||
| 4. **Backward**: Before the group's backward, tensors are reloaded from CPU to GPU on `h2d_stream`, and the compute stream waits for the transfer to complete. | ||
|
|
||
| ### Warmup and Adaptive Offloading | ||
|
|
||
| The first training iteration serves as a **warmup phase** where the manager records tensor groups, their sizes, and the execution order. After warmup, a `post_warmup_callback` runs to: | ||
|
lhb8125 marked this conversation as resolved.
Resolved
|
||
|
|
||
| 1. **Reserve margin**: The last N groups (by deduplication count) are kept on GPU to avoid reload blocking the compute stream. | ||
| 2. **Apply PP rank delta**: Higher PP ranks offload fewer bytes (controlled by `delta_offload_bytes_across_pp_ranks`). | ||
| 3. **Apply fraction**: Only a fraction of eligible groups are actually offloaded (controlled by `activation_offload_fraction`). | ||
| 4. **Print summary table**: An ASCII table of per-rank offload bytes is printed for debugging. | ||
|
|
||
| ### CPU Tensor Pool | ||
|
lhb8125 marked this conversation as resolved.
Resolved
|
||
|
|
||
| A `GPUTensorPool` (on CPU with pinned memory) caches allocated tensors by `(shape, dtype)`. This avoids repeated `cudaMallocHost` / `cudaFreeHost` calls and reduces D2H latency after the first iteration. | ||
|
|
||
| ### CUDA Graph Support | ||
|
|
||
| When offloading modules captured inside a CUDA graph: | ||
|
|
||
| - A dedicated `cuda_graph_stream` runs the captured computation, while `d2h_stream` overlaps D2H transfers. | ||
| - During CUDA graph **warmup**, offloading is disabled (`pre_warmup_hook` / `post_warmup_hook`). | ||
| - The `delay_offload_until_cuda_graph` option defers D2H launches until graph replay, utilizing the CPU idle time during `cudaGraphLaunch` to issue offload commands with near-zero CPU overhead. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.