Skip to content

Correctly detect zero_offload #2213

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
tjruwase merged 15 commits into from
Aug 22, 2022
Merged

Correctly detect zero_offload #2213

Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
573430e
Correctly detect offload configuration
tjruwase Aug 10, 2022
8678153
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
tjruwase Aug 10, 2022
83c873f
Correctly detect offload configuration
tjruwase Aug 10, 2022
02e563c
Merge branch 'olruwase/z3_bogus_cpu_offload' of github.com:microsoft/…
tjruwase Aug 10, 2022
b7fe376
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
jeffra Aug 11, 2022
8156340
Handle deprecated cpu offload setting
tjruwase Aug 11, 2022
ca15a7d
Merge branch 'olruwase/z3_bogus_cpu_offload' of github.com:microsoft/…
tjruwase Aug 11, 2022
9ae31f2
Merge branch 'master' of github.com:microsoft/DeepSpeed into olruwase…
tjruwase Aug 12, 2022
885177b
Correcly detect zero_offload setting
tjruwase Aug 12, 2022
282610a
Minor tweak
tjruwase Aug 12, 2022
39cd34d
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
jeffra Aug 12, 2022
b747cff
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
awan-10 Aug 16, 2022
a9050c1
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
tjruwase Aug 17, 2022
32de5de
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
tjruwase Aug 18, 2022
652e427
Merge branch 'master' into olruwase/z3_bogus_cpu_offload
tjruwase Aug 19, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion deepspeed/runtime/engine.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import deepspeed

from deepspeed.runtime.utils import see_memory_usage, DummyOptim
from .zero.offload_config import OffloadDeviceEnum
from deepspeed.runtime.zero.stage_1_and_2 import DeepSpeedZeroOptimizer
from deepspeed.runtime.zero.partition_parameters import ZeroParamStatus
from deepspeed.runtime.zero.utils import is_zero_supported_optimizer, ZeRORuntimeException
Expand Down Expand Up @@ -646,7 +647,9 @@ def zero_offload_param(self):
return self._config.zero_config.offload_param

def zero_cpu_offload(self):
return self._config.zero_config.offload_optimizer is not None
if self._config.zero_config.offload_optimizer is not None:
return self._config.zero_config.offload_optimizer.device == OffloadDeviceEnum.cpu
return False

def zero_sub_group_size(self):
return self._config.zero_config.sub_group_size
Expand Down