Closed
Fix Windows CUDA Plugin EP and CUDA CI azcopy authentication failures#29267
Conversation
Copilot
AI
changed the title
[WIP] Fix failing GitHub Actions job for CUDA Plugin Windows CI
Fix Windows CUDA Plugin EP test job CUDA SDK dependency
Jun 25, 2026
jiafatom
approved these changes
Jun 25, 2026
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Windows CUDA Plugin EP CI workflow to remove the Windows GPU test job’s direct CUDA SDK download dependency (which was failing due to azcopy MSI auth), by staging the required CUDA runtime files during the build job and transferring them to the test job via a dedicated artifact.
Changes:
- Stage CUDA runtime + CUPTI directories in the build job and upload them as a separate artifact.
- Download the staged CUDA runtime artifact in the test job and update
PATHto use it. - Remove the test job’s direct
azcopyCUDA SDK download step.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
… on this runner pool
…on vs2022-latest pool
Copilot
AI
changed the title
Fix Windows CUDA Plugin EP test job CUDA SDK dependency
Fix Windows CUDA Plugin EP and CUDA CI azcopy authentication failures
Jun 26, 2026
…ude dir at configure time Dawn's src/emdawnwebgpu/CMakeLists.txt copies webgpu_glfw.h and webgpu_enum_class_bitmasks.h into gen/src/emdawnwebgpu/include/webgpu/ via cmake -E copy. With -j32 parallel builds those copy commands can be triggered from multiple Makefile targets simultaneously before the destination directory is created, causing: Error copying file "...webgpu_enum_class_bitmasks.h" to "...include/webgpu/..." The existing dawn_parallel_build_fix.patch adds make_directory inside the custom command, but it may not be applied when the Dawn source is reused from a cached runner state (FetchContent skips PATCH_COMMAND when stamps indicate the source is already populated). Adding file(MAKE_DIRECTORY ...) at CMake configure time ensures the directory exists before any build job runs, regardless of patch state. Fixes wasm_Debug / build-wasm CI failure (Job ID: 83586543400).
Contributor
|
Using the system provided CUDA isn't a bad approach, but isn't a fix for azcopy auth issues. If it is expected that the agent image provide the relevant CUDA SDK then we should ditch the download entirely. Also makes completely unrelated changes. |
Contributor
|
Will use #29274 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Fixes multiple Windows CI jobs that were failing because the
AZCOPY_MSI_CLIENT_ID(63b63039-6328-442f-954b-5a64d124e5b4) was configured but not assigned to certain runner pools.Windows CUDA Plugin EP build job (
onnxruntime-github-vs2022-latest)AZCOPY_MSI_CLIENT_IDfrom the build job env; the pool lacks that user-assigned identity.Windows CUDA Plugin EP test job (
onnxruntime-github-Win2022-GPU-A10)azcopydownload of the CUDA SDK.AZCOPY_AUTO_LOGIN_TYPEandAZCOPY_MSI_CLIENT_IDfrom the test job env.Windows GPU CUDA CI Pipeline build job (
windows_cuda.yml,onnxruntime-github-vs2022-latest)AZCOPY_MSI_CLIENT_IDfrom the build job env; thevs2022-latestpool does not have that user-assigned identity.AZCOPY_AUTO_LOGIN_TYPE: MSIso azcopy uses the system-assigned identity.Win2022-GPU-A10retainsAZCOPY_MSI_CLIENT_IDas that pool still requires it.Windows GPU Doc Gen CI (
windows_gpu_doc_gen.yml)AZCOPY_MSI_CLIENT_IDfrom the job env for the same reason.Motivation and Context
Infrastructure drift caused the user-assigned managed identity to become unavailable on certain runner pools (
onnxruntime-github-vs2022-latest). Any workflow step usingAZCOPY_MSI_CLIENT_IDon those pools fails withManagedIdentityCredential authentication failed: the requested identity isn't assigned to this resource. The fix removes the hard-coded client ID from affected runner pools, allowing azcopy to fall back to the system-assigned identity, and eliminates the redundant CUDA SDK download from the GPU test runner by reusing a build artifact.