[Dev] Add combined 1F1B schedule-aware tensor release - #7062
Conversation
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
2a20947 to
ff310ef
Compare
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
| in 1f1b phase of pipelining or non-pipelining schedule. | ||
| """ | ||
|
|
||
| ep_overlap_use_scheduled_tensor_release: bool = False |
There was a problem hiding this comment.
What about setting to True by default? Maybe we need more benchmarking data to support this comment.
There was a problem hiding this comment.
Good point. I am keeping this opt-in for now because the current evidence covers correctness and a native-allocator microbenchmark, but not end-to-end throughput and peak memory across the supported EP-overlap configurations. This knob is only useful for the sync-free combined 1F1B path with overlap_moe_expert_parallel_comm=True; other schedules can keep the default allocator record_stream path. Also, changing this field to True alone would make the default TransformerConfig invalid because validation requires the parent overlap flag. I clarified when users should enable the knob in 188e5f5b4. We can revisit a conditional default-on behavior, with an explicit opt-out, after representative on/off benchmarks.
There was a problem hiding this comment.
This knob only helps the sync-free path, which is not widely used.
There was a problem hiding this comment.
Make sense, but for non sync-free path where CPU is faster like DSv4/3 case this toggle should also help?
Signed-off-by: hongbinl <hongbinl@nvidia.com>
Signed-off-by: hongbinl <hongbinl@nvidia.com>
|
/ok to test 930c99c |
Signed-off-by: hongbinl <hongbinl@nvidia.com>
|
/ok to test 3a8b253 |
Signed-off-by: hongbinl <hongbinl@nvidia.com>
|
/ok to test 7dc1595 |
Summary
free_input=Falseonly drops ownership metadata, while external or unregistered tensors retain the conservativerecord_streamfallback.EMPTY_STORAGEremain unambiguous.combined_1f1b_tensor_release.py, and scope implementation names to this schedule.This supersedes #6948 with the reviewed tensor-bound ownership model.
Test plan
py_compileon all 12 changed Python files.CHECK_ONLY=true BASE_REF=dev bash tools/autoformat.sh(Black, isort, Pylint, and Ruff pass).python tools/check_copyright.pyon the complete PR changeset.tests/unit_tests/a2a_overlap/test_combined_1f1b_tensor_release.py—21 passedin 4.52s.record_streamwhile the consumer is still in flight. It takes 0.12s; total file time increased from 4.41s to 4.52s.