Skip to content

AZP: CUDA compatible - #7212

Merged
yosefe merged 1 commit into
openucx:masterfrom
avildema:cuda_launch
Feb 3, 2022
Merged

AZP: CUDA compatible#7212
yosefe merged 1 commit into
openucx:masterfrom
avildema:cuda_launch

Conversation

@avildema

@avildema avildema commented Aug 9, 2021

Copy link
Copy Markdown
Contributor

launch test on official CUDA images

@avildema avildema added the WIP-DNM Work in progress / Do not review label Aug 9, 2021
@avildema
avildema requested a review from yosefe August 20, 2021 16:44
@avildema avildema removed the WIP-DNM Work in progress / Do not review label Aug 20, 2021
yosefe
yosefe reviewed Aug 21, 2021
View reviewed changes

@yosefe yosefe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in general can simplify this script since some parts are copy/paste from test_jenkins.sh but not needed here

Comment thread buildlib/azure-pipelines-pr.yml Outdated
Comment thread buildlib/azure-pipelines-pr.yml Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
Comment thread buildlib/tools/test_malloc_hook.sh Outdated
@yosefe

yosefe commented Aug 21, 2021

Copy link
Copy Markdown
Member

Azure job did not run:
image

@avildema
avildema requested a review from yosefe August 24, 2021 16:33
@yosefe

yosefe commented Aug 26, 2021

Copy link
Copy Markdown
Member

@avildema pls squash

@yosefe

yosefe commented Aug 26, 2021 •
edited
Loading

Copy link
Copy Markdown
Member

This PR creates a problem that GPU machines are running both docker and bare-metal tests. Bare-metal tests try to use temporary network devices created by docker and fail.
IMO we should move all PRs on GPU nodes to run under docker. Including:

  • Add rdma devices to docker
  • Remove ad-hoc "docker" tests - later
  • Make sure GPU-direct tests really run inside the docker
  • Go and Java tests

yosefe
yosefe requested changes Aug 26, 2021
View reviewed changes

@yosefe yosefe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

run all tests on GPU nodes under docker

yosefe
yosefe reviewed Sep 10, 2021
View reviewed changes

@yosefe yosefe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems RDMA transports are not found in the container

Comment thread buildlib/tools/test_malloc_hook.sh
Comment thread buildlib/pr/cuda.yml Outdated
@@ -0,0 +1,89 @@
jobs:
- job: source_tarball
displayName: Test autogen_cuda

@yosefe yosefe Sep 10, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Create UCX source tarball

Comment thread buildlib/pr/cuda.yml Outdated
clean: true
fetchDepth: 100
- bash: |
set -eE

@yosefe yosefe Sep 10, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set -xeE

Comment thread buildlib/pr/main.yml Outdated
parameters:
name: gpu
demands: ucx_gpu -equals yes
demands: ucx_gpu_test -equals yes

@yosefe yosefe Sep 10, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we remove this now? seems it runs on both docker and bare metal?

@avildema avildema Sep 13, 2021

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it need for test, I have one agent with one label ucx_gpu_test and this label always free

@yosefe yosefe Jan 24, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still needed?

Comment thread buildlib/tools/test_malloc_hook.sh
@@ -0,0 +1,80 @@
#!/bin/bash -eExl

@yosefe yosefe Sep 10, 2021

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

move this file under pr/.
maybe move all cuda files under buildlib/pr/cuda ?

@avildema

avildema commented Sep 17, 2021

Copy link
Copy Markdown
Contributor Author

@yosefe Could you please check result for test_jenkins.sh for GPU on docker images
https://dev.azure.com/ucfconsort/ucx/_build/results?buildId=26010&view=logs&j=466d7b3b-a401-53bf-628f-285bcb9960ef&t=1629ddf3-f393-545f-bef9-c35a9a07dc48

@yosefe yosefe added the WIP-DNM Work in progress / Do not review label Oct 20, 2021
@avildema

avildema commented Nov 18, 2021

Copy link
Copy Markdown
Contributor Author

bot:retest

@avildema avildema removed the WIP-DNM Work in progress / Do not review label Nov 22, 2021
@avildema
avildema requested a review from yosefe November 26, 2021 09:58
yosefe
yosefe reviewed Jan 24, 2022
View reviewed changes
Comment thread buildlib/pr/cuda/test_malloc_hook.sh Outdated
Comment thread buildlib/pr/main.yml Outdated
Comment thread buildlib/pr/main.yml Outdated
parameters:
name: gpu
demands: ucx_gpu -equals yes
demands: ucx_gpu_test -equals yes

@yosefe yosefe Jan 24, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still needed?

Comment thread buildlib/pr/main.yml
Comment thread buildlib/pr/cuda/cuda.yml
yosefe
yosefe reviewed Jan 24, 2022
View reviewed changes
Comment thread buildlib/pr/main.yml Outdated
@@ -1,121 +1,149 @@
variables:
DOCKER_OPT_SHARES: -v /hpc/local:/hpc/local -v /auto/sw_tools:/auto/sw_tools

@yosefe yosefe Jan 24, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

DOCKER_OPT_VOLUMES

@yosefe

yosefe commented Jan 30, 2022

Copy link
Copy Markdown
Member

@avildema pls squash

@yosefe
yosefe merged commit 72ff65d into openucx:master Feb 3, 2022
@yosefe yosefe mentioned this pull request Feb 3, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yosefe yosefe yosefe approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@avildema @yosefe