How to install smdistributed?

[rohan-varma]

What did you find confusing? Please describe.
I installed sagemaker with pip install sagemaker --update, and am attempting to use distributed model parallel with pytorch. However, I'm unable to import smdistributed.

The docs https://sagemaker.readthedocs.io/en/stable/api/training/smp_versions/v1.2.0/smd_model_parallel_pytorch.html don't have installation instructions for smdistributed. I was wondering how do I get smdistributed installed? Thank you!

I am also looking at https://docs.amazonaws.cn/en_us/sagemaker/latest/dg/model-parallel-customize-training-script-pt.html which directs me to https://sagemaker.readthedocs.io/en/stable/api/training/smp_versions/v1.2.0/smd_model_parallel_common_api.html#smp.init to initialize the sagemaker distributed environment. But again I'm not sure how to get the smdistributed library.

https://github.com/aws/amazon-sagemaker-examples has some smdistributed examples but doesn't provide any clear installation instructions. environment.yml in that repo seems to indicate all that's needed is sagemaker which I have installed.

Describe how documentation can be improved
Could not find clear installation instructions for smdistributed, would it be possible to add these?

Additional context
Add any other context or screenshots about the documentation request here.

[juliensimon]

Hi,

the easiest way is to either use or start from one of the built-in TensorFlow/PyTorch containers.

You can also build your own, starting from the SageMaker Training Toolkit at https://github.com/aws/sagemaker-training-toolkit.

More details at https://docs.aws.amazon.com/sagemaker/latest/dg/model-parallel-use-api.html
and https://docs.aws.amazon.com/sagemaker/latest/dg/data-parallel-use-api.html

[rohit901]

Hello,
could you guide me on how can I resolve my issue with smdistributed? I'm using HuggingFace container, and these are the args:

{
    'image_uri': None,
    'entry_point': 'train_vlcm_distill_lcm_wds.py',
    'source_dir': 'scripts',
    'role': 'xxxx',
    'transformers_version': '4.36.0',
    'pytorch_version': '2.1.0',
    'py_version': 'py310',
    'base_job_name': 'accelerate-sagemaker-1',
    'instance_count': 1,
    'instance_type': 'ml.p4d.24xlarge',
    'debugger_hook_config': False,
    'distribution': {
        'smdistributed': {
            'dataparallel': {
                'enabled': True
            }
        }
    },
    'environment': {
        'ACCELERATE_USE_SAGEMAKER': 'true',
        'ACCELERATE_MIXED_PRECISION': 'fp16',
        'ACCELERATE_DYNAMO_BACKEND': 'NO',
        'ACCELERATE_DYNAMO_MODE': 'default',
        'ACCELERATE_DYNAMO_USE_FULLGRAPH': 'False',
        'ACCELERATE_DYNAMO_USE_DYNAMIC': 'False',
        'ACCELERATE_SAGEMAKER_DISTRIBUTED_TYPE': 'DATA_PARALLEL'
    },
    'metric_definitions': None
}

im getting ModuleNotFoundError: No module named ‘smdistributed’

entire error logs:

ErrorMessage "ModuleNotFoundError: No module named 'smdistributed'
 /opt/conda/lib/python3.10/site-packages/diffusers/utils/outputs.py:63: UserWarning: torch.utils._pytree._register_pytree_node is deprecated. Please use torch.utils._pytree.register_pytree_node instead.
 torch.utils._pytree._register_pytree_node(
 Traceback (most recent call last)
 File "/opt/conda/lib/python3.10/runpy.py", line 196, in _run_module_as_main
 return _run_code(code, main_globals, None,
 File "/opt/conda/lib/python3.10/runpy.py", line 86, in _run_code
 exec(code, run_globals)
 File "/opt/conda/lib/python3.10/site-packages/mpi4py/__main__.py", line 7, in <module>
 main()
 File "/opt/conda/lib/python3.10/site-packages/mpi4py/run.py", line 230, in main
 run_command_line(args)
 File "/opt/conda/lib/python3.10/site-packages/mpi4py/run.py", line 47, in run_command_line
 run_path(sys.argv[0], run_name='__main__')
 File "/opt/conda/lib/python3.10/runpy.py", line 289, in run_path
 return _run_module_code(code, init_globals, run_name,
 File "/opt/conda/lib/python3.10/runpy.py", line 96, in _run_module_code
 _run_code(code, mod_globals, init_globals,
 File "train_vlcm_distill_lcm_wds.py", line 1416, in <module>
 main(args)
 File "train_vlcm_distill_lcm_wds.py", line 780, in main
 accelerator = Accelerator(
 File "/opt/conda/lib/python3.10/site-packages/accelerate/accelerator.py", line 371, in __init__
 self.state = AcceleratorState(
 File "/opt/conda/lib/python3.10/site-packages/accelerate/state.py", line 758, in __init__
 PartialState(cpu, **kwargs)
 File "/opt/conda/lib/python3.10/site-packages/accelerate/state.py", line 145, in __init__
 import smdistributed.dataparallel.torch.torch_smddp  # noqa
 --------------------------------------------------------------------------
 Primary job  terminated normally, but 1 process returned
 a non-zero exit code. Per user-direction, the job has been aborted.
 mpirun.real detected that one or more processes exited with non-zero status, thus causing
 the job to be terminated. The first process to do so was
 
 Process name: [[41100,1],1]
 Exit code:    1"

[TEChopra1000]

Note: SageMaker distributed is not currently supported in the Chinese regions. I'm adding this note because I see that you've pointed to a doc in the China region.

[rohan-varma]

Note: SageMaker distributed is not currently supported in the Chinese regions. I'm adding this note because I see that you've pointed to a doc in the China region.

I see, I'm actually not sure why I ended up at a .cn link. I think google somehow directed me to the .cn domain when I entered in my search query

[gui-miotto]

You can install it with pip install given that you have the url to the binary, e.g.:

pip install https://smdataparallel.s3.amazonaws.com/binary/pytorch/2.4.1/cu121/2024-10-09/smdistributed_dataparallel-2.5.0-cp311-cp311-linux_x86_64.whl

The available binaries (i.e. links to the .whl) can be found in https://docs.aws.amazon.com/sagemaker/latest/dg/distributed-data-parallel-support.html#distributed-data-parallel-supported-frameworks