Skip to content

contrib: use cuda-dl-base image for docker builds - #115

Merged
nv-anants merged 8 commits into
mainfrom
anants/ubuntu22-default
Apr 4, 2025
Merged

contrib: use cuda-dl-base image for docker builds#115
nv-anants merged 8 commits into
mainfrom
anants/ubuntu22-default

Conversation

@nv-anants

@nv-anants nv-anants commented Apr 2, 2025 •
edited
Loading

Copy link
Copy Markdown
Member
  • Use cuda-dl-base as default base image for docker builds
    • Ubuntu 24.04 as default
  • remove all dependencies. Nsight, UCX are already present in base image
  • Add option to pass in python-versions
  • use python 3.12 virtual env for pip installs
  • always use local for nixl repo code
    • to run a specific commit/tag, run git checkout prior to build script
  • add pip install nixl wheel to uv venv

Testing

./contrib/build-container.sh --python-versions 3.10,3.11,3.12 --os ubuntu22

./contrib/build-container.sh --python-versions 3.10,3.11,3.12

 unzip -l dist/nixl-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl 
Archive:  dist/nixl-0.1.0-cp312-cp312-manylinux_2_34_x86_64.whl
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2025-04-03 17:11   .nixl.mesonpy.libs/
   166152  2025-04-03 17:11   .nixl.mesonpy.libs/libnixl.so
   104304  2025-04-03 17:11   .nixl.mesonpy.libs/libnixl_build.so
    18528  2025-04-03 17:11   .nixl.mesonpy.libs/libserdes.so
    34440  2025-04-03 17:11   .nixl.mesonpy.libs/libstream.so
    35953  2025-04-03 17:11   .nixl.mesonpy.libs/libucx_utils.so
        0  2025-04-03 17:11   .nixl.mesonpy.libs/plugins/
   359057  2025-04-03 17:11   .nixl.mesonpy.libs/plugins/libplugin_UCX.so
   103665  2025-04-03 17:11   .nixl.mesonpy.libs/plugins/libplugin_UCX_MO.so
        0  2025-04-03 17:11   nixl/
     1103  2025-04-03 17:11   nixl/__init__.py
    17918  2025-04-03 17:11   nixl/_api.py
   626041  2025-04-03 17:11   nixl/_bindings.cpython-312-x86_64-linux-gnu.so
   155840  2025-04-03 17:11   nixl/_utils.cpython-312-x86_64-linux-gnu.so
        0  2025-04-03 17:11   nixl.libs/
 34855529  2025-04-03 17:11   nixl.libs/libcuda-c78a5b2b.so.560.35.03
   732713  2025-04-03 17:11   nixl.libs/libcudart-09529672.so.12.6.77
  3980065  2025-04-03 17:11   nixl.libs/libplugin_UCX-e092e135.so
   446689  2025-04-03 17:11   nixl.libs/libucm-4d1fd13c.so.0.0.0
  7151521  2025-04-03 17:11   nixl.libs/libucp-da29dbfa.so.0.0.0
  7889585  2025-04-03 17:11   nixl.libs/libucs-17022a28.so.0.0.0
  1270689  2025-04-03 17:11   nixl.libs/libuct-834ede91.so.0.0.0
        0  2025-04-03 17:11   nixl-0.1.0.dist-info/
    11357  2025-04-03 17:11   nixl-0.1.0.dist-info/LICENSE
    17425  2025-04-03 17:11   nixl-0.1.0.dist-info/METADATA
       99  2025-04-03 17:11   nixl-0.1.0.dist-info/WHEEL
     1983  2025-04-03 17:11   nixl-0.1.0.dist-info/RECORD
---------                     -------
 57980656                     27 files

docker run -it  docker.io/library/nixl:v0.1.0.dev.525fb9a 
source .venv/bin/activate
uv pip install torch numpy pytest

python3 examples/python/nixl_api_example.py
Using NIXL Plugins from:
/usr/local/nixl/lib/x86_64-linux-gnu/plugins
Loaded plugin GDS
Loaded plugin UCX
Loaded plugin UCX_MO
Initialized NIXL agent: target
Plugin parameters
['DRAM_SEG', 'VRAM_SEG']
{'ucx_devices': ''}

Loaded backend parameters
['DRAM_SEG', 'VRAM_SEG']
{}

Initialized NIXL agent: initiator
Loaded name from metadata: b'target'
Target done
Initiator done
Initiator done
Target done
sent notif 
b'DESCS: \x80\x04\x95\xc0\x00\x00\x00\x00\x00\x00\x00\x8c\x0enixl._bindings\x94\x8c\x0cnixlRegDList\x94\x93\x94)\x81\x94C\x96nixlSerDes|nixlDList\n\x00\x00\x00\x00\x00\x00\x00nixlSDList|t\x04\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00|s\x01\x00\x00\x00\x00\x00\x00\x00\x01|n\x08\x00\x00\x00\x00\x00\x00\x00\x02\x00\x00\x00\x00\x00\x00\x00|\x19\x00\x00\x00\x00\x00\x00\x00\x10\xfc\xd3\t\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00a|\x19\x00\x00\x00\x00\x00\x00\x00\x10\xfd\xd3\t\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00b|\x94b.'
received message from initiator
notif test complete, doing transfer 2

Transfer 2 started
Initiator done
Target done
Test Complete.

pytest test/python
=========================================================================================== test session starts ===========================================================================================
platform linux -- Python 3.12.9, pytest-8.3.5, pluggy-1.5.0
rootdir: /workspace/nixl
configfile: pyproject.toml
collected 5 items                                                                                                                                                                                         

test/python/test_nixl_api.py ..s                                                                                                                                                                    [ 60%]
test/python/test_nixl_bindings.py ..                                                                                                                                                                [100%]

====================================================================================== 4 passed, 1 skipped in 1.35s =======================================================================================
LOG: DescList of mem type 0unsorted
    LOG: Desc (1000, 105) from devID 0
    LOG: Desc (2000, 30) from devID 0
    LOG: Desc (1010, 20) from devID 0

@nv-anants nv-anants changed the title contrib: use ubuntu22 as default build and add python versions option contrib: use ubuntu22 as default build Apr 2, 2025
@nv-anants
nv-anants requested review from aranadive and tstamler April 2, 2025 17:29
@nv-anants
nv-anants marked this pull request as ready for review April 2, 2025 17:30
@tstamler
meson: meson modules for hpcx base
4501501 
Signed-off-by: Tim Stamler <tstamler@nvidia.com>
@nv-anants nv-anants changed the title contrib: use ubuntu22 as default build contrib: use cuda-dl-base image for docker builds Apr 3, 2025
tstamler
tstamler reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/Dockerfile
tstamler
tstamler reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/Dockerfile Outdated
tstamler
tstamler reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/build-container.sh Outdated
tstamler
tstamler reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/build-container.sh Outdated
@nv-anants
nv-anants requested a review from tstamler April 3, 2025 20:14
aranadive
aranadive reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/Dockerfile
aranadive
aranadive reviewed Apr 3, 2025
View reviewed changes
Comment thread contrib/Dockerfile
@sb17v

sb17v commented Apr 3, 2025

Copy link
Copy Markdown
Contributor

NIXL Benchmark data with Ubuntu-24.04:
Single node test across 2 GPUs-

root@eos0379:/workspace/home/codebase/source/nixlbench/hpcx_build# env UCX_LOG_LEVEL=ERROR UCX_TLS=^gdr_copy mpirun --allow-run-as-root -np 2 --hostfile myhostfile launch.sh nixlbench --total_buffer_size=68719476736 --max_block_size=1073741824 --start_batch_size=64 --max_batch_size=64 --seg_type="VRAM"
running ...
running ...
Loaded plugin UCX_MO
Loaded plugin UCX
Loaded plugin GDS
Loaded plugin UCX_MO
Loaded plugin UCX
Loaded plugin GDS
Block Size (B)      Batch Size     Avg Lat. (us)  B/W (MiB/Sec)  B/W (GiB/Sec)  B/W (GB/Sec)
-----------------------------------------------------------------
4096                64             3.11355        1254.6         1.22519        1.31554
8192                64             3.11092        2511.31        2.45245        2.6333
16384               64             3.11277        5019.65        4.902          5.26349
32768               64             3.11587        10029.3        9.79422        10.5165
65536               64             3.12408        20005.9        19.537         20.9777
131072              64             3.10589        40246.1        39.3028        42.2011
262144              64             3.10316        80563.1        78.6749        84.4766
524288              64             3.7178         134488         131.336        141.021
1048576             64             4.98362        200657         195.954        210.404
2097152             64             7.6798         260424         254.32         273.074
4194304             64             12.9917        307890         300.674        322.846
8388608             64             23.8646        335225         327.368        351.509
16777216            64             45.5765        351058         342.83         368.111
33554432            64             87.9193        363970         355.44         381.651
67108864            64             172.157        371753         363.04         389.811
134217728           64             340.464        375957         367.146        394.22
268435456           64             677.015        378130         369.268        396.499

2 - node test (1 GPU each node):

root@eos0379:/workspace/home/codebase/source/nixlbench/hpcx_build# env UCX_LOG_LEVEL=ERROR UCX_TLS=^gdr_copy mpirun --allow-run-as-root -np 2 --hostfile myhostfile launch.sh nixlbench --total_buffer_size=68719476736 --max_block_size=1073741824 --start_batch_size=64 --max_batch_size=64 --seg_type="VRAM"
running ...
running ...
Loaded plugin UCX_MO
Loaded plugin UCX
Loaded plugin GDS
Loaded plugin UCX_MO
Loaded plugin UCX
Loaded plugin GDS
Block Size (B)      Batch Size     Avg Lat. (us)  B/W (MiB/Sec)  B/W (GiB/Sec)  B/W (GB/Sec)
-----------------------------------------------------------------
4096                64             0.296578       13171.1        12.8624        13.8109
8192                64             0.329172       23733.8        23.1775        24.8867
16384               64             0.529703       29497.7        28.8063        30.9305
32768               64             0.993531       31453.5        30.7163        32.9813
65536               64             1.95356        31992.8        31.243         33.5469
131072              64             3.86289        32359.2        31.6008        33.9311
262144              64             7.68102        32547.8        31.7849        34.1288
524288              64             15.3144        32649          31.8838        34.2349
1048576             64             30.5824        32698.5        31.9321        34.2869
2097152             64             61.1094        32728.2        31.9611        34.318
4194304             64             122.166        32742.4        31.975         34.3329
8388608             64             244.277        32749.7        31.9821        34.3405
16777216            64             488.504        32753          31.9854        34.344
33554432            64             977.01         32753          31.9853        34.344
67108864            64             1954           32753.4        31.9857        34.3444

The data looks good, saturating both NVLink and IB B/W

@nv-anants
nv-anants merged commit 7e30652 into main Apr 4, 2025
@nv-anants
nv-anants deleted the anants/ubuntu22-default branch April 4, 2025 15:43
@coderabbitai coderabbitai Bot mentioned this pull request Aug 7, 2026
Merged
@svc-nixl svc-nixl mentioned this pull request Oct 6, 2026
Merged

This branch was previously deployed

1 inactive deployment
GITLAB — 0f7c0890 Deployed Apr 3, 2025 by nv-anants
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@aranadive aranadive aranadive left review comments
@tstamler tstamler tstamler approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@nv-anants @sb17v @tstamler @aranadive