CUDA.jl CI broken again because of Enzyme

[maleadt]

As seen on CUDA.jl#master:

extensions/enzyme  (2) |         failed at 2024-08-22T19:02:35.224
Testing finished in 4 minutes, 37 seconds, 205 milliseconds
Worker 2 failed running test extensions/enzyme:
Some tests did not pass: 12 passed, 0 failed, 1 errored, 0 broken.
extensions/enzyme: Error During Test at /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/extensions/enzyme.jl:42
  Got exception outside of a @test
  Calling an intrinsic function that clashes with an existing definition: void (i64) report_exception
  Stacktrace:
    [1] error(::String, ::String, ::String, ::Symbol)
      @ Base ./error.jl:44
    [2] call!(builder::LLVM.IRBuilder, rt::GPUCompiler.Runtime.RuntimeMethodInstance, args::Vector{LLVM.ConstantExpr})
      @ GPUCompiler ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/GPUCompiler/wcqlQ/src/rtlib.jl:37
    [3] emit_exception!(builder::LLVM.IRBuilder, name::String, inst::LLVM.UnreachableInst)
      @ GPUCompiler ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/GPUCompiler/wcqlQ/src/irgen.jl:219
    [4] emit_error(B::LLVM.IRBuilder, orig::LLVM.UnreachableInst, string::String, errty::Type)
      @ Enzyme.Compiler ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:1876
    [5] emit_error
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:1861 [inlined]
    [6] #codegen#18952
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6347
    [7] codegen
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:5493 [inlined]
    [8] #83
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:801
    [9] #JuliaContext#152
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/GPUCompiler/wcqlQ/src/driver.jl:34
   [10] JuliaContext
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/GPUCompiler/wcqlQ/src/driver.jl:25 [inlined]
   [11] tape_type
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:800 [inlined]
   [12] #augmented_primal#30
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/ext/EnzymeCoreExt.jl:243
   [13] augmented_primal
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/ext/EnzymeCoreExt.jl:238 [inlined]
   [14] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/src/compiler/execution.jl:114 [inlined]
   [15] square!
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/extensions/enzyme.jl:19 [inlined]
   [16] diffejulia_square__24644wrap
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
   [17] macro expansion
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:7049 [inlined]
   [18] enzyme_call
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6658 [inlined]
   [19] CombinedAdjointThunk
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6535 [inlined]
   [20] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:320 [inlined]
   [21] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:348 [inlined]
   [22] autodiff(mode::ReverseMode{false, FFIABI, false, false}, f::typeof(square!), args::Duplicated{CuArray{Float32, 1, CUDA.DeviceMemory}})
      @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:329
   [23] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/extensions/enzyme.jl:47 [inlined]
   [24] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [25] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/extensions/enzyme.jl:43
   [26] include
      @ ./client.jl:489 [inlined]
   [27] #11
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/runtests.jl:87 [inlined]
   [28] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [29] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [30] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [31] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/src/utilities.jl:35 [inlined]
   [32] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/src/memory.jl:829 [inlined]
   [33] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/setup.jl:62
   [34] eval
      @ ./boot.jl:385 [inlined]
   [35] runtests(f::Function, name::String, time_source::Symbol)
      @ Main /var/lib/buildkite-agent/builds/gpuci-14/julialang/cuda-dot-jl/test/setup.jl:74
   [36] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
      @ Base ./essentials.jl:892
   [37] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:889
   [38] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287
   [39] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
   [40] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287

Also, as seen on JuliaGPU/CUDA.jl#2472, but probably related to Enzyme and not the PR:

extensions/enzyme  (2) |         failed at 2024-08-22T15:44:41.397
Testing finished in 3 minutes, 58 seconds, 560 milliseconds
Worker 2 failed running test extensions/enzyme:
Some tests did not pass: 9 passed, 0 failed, 2 errored, 0 broken.
extensions/enzyme: Error During Test at /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:23
  Got exception outside of a @test
  MethodError: no method matching Duplicated{CuDeviceVector{Float32, 1}}(::CuDeviceVector{Float32, 1}, ::CuDeviceVector{Float32, 1})
  Stacktrace:
    [1] forward
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/ext/EnzymeCoreExt.jl:58 [inlined]
    [2] map
      @ ./tuple.jl:291 [inlined]
    [3] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/compiler/execution.jl:110 [inlined]
    [4] square!
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:19 [inlined]
    [5] fwddiffejulia_square__8541wrap
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
    [6] macro expansion
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:7049 [inlined]
    [7] enzyme_call
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6658 [inlined]
    [8] ForwardModeThunk
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6538 [inlined]
    [9] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:437 [inlined]
   [10] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:348 [inlined]
   [11] autodiff(mode::ForwardMode{FFIABI, false}, f::typeof(square!), args::Duplicated{CuArray{Float32, 1, CUDA.DeviceMemory}})
      @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:329
   [12] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:28 [inlined]
   [13] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [14] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:24
   [15] include
      @ ./client.jl:489 [inlined]
   [16] #11
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/runtests.jl:87 [inlined]
   [17] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [18] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [19] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [20] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/utilities.jl:35 [inlined]
   [21] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/memory.jl:829 [inlined]
   [22] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:62
   [23] eval
      @ ./boot.jl:385 [inlined]
   [24] runtests(f::Function, name::String, time_source::Symbol)
      @ Main /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:74
   [25] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
      @ Base ./essentials.jl:892
   [26] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:889
   [27] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287
   [28] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
   [29] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287
extensions/enzyme: Error During Test at /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:42
  Got exception outside of a @test
  Mismatched activity for:   store i8 addrspace(1)* %bitcast_coercion13, i8 addrspace(1)** %newstruct.sroa.0.0..sroa_idx, align 8, !dbg !1302, !noalias !1305 const val:   %bitcast_coercion13 = addrspacecast i8* %34 to i8 addrspace(1)*, !dbg !1293
  Type tree: {[-1]:Pointer, [-1,-1]:Float@float}
   llvalue=  %bitcast_coercion13 = addrspacecast i8* %34 to i8 addrspace(1)*, !dbg !1293
  You may be using a constant variable as temporary storage for active memory (https://enzyme.mit.edu/julia/stable/faq/#Activity-of-temporary-storage). If not, please open an issue, and either rewrite this variable to not be conditionally active or use Enzyme.API.runtimeActivity!(true) as a workaround for now
  Stacktrace:
   [1] CuDeviceArray
     @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/device/array.jl:31
   [2] unsafe_convert
     @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/array.jl:454
  Stacktrace:
    [1] CuDeviceArray
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/device/array.jl:31 [inlined]
    [2] unsafe_convert
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/array.jl:454
    [3] adapt_storage
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/compiler/execution.jl:162 [inlined]
    [4] adapt_structure
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/Adapt/7T9au/src/Adapt.jl:57 [inlined]
    [5] adapt
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/packages/Adapt/7T9au/src/Adapt.jl:40 [inlined]
    [6] cudaconvert
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/compiler/execution.jl:205 [inlined]
    [7] map
      @ ./tuple.jl:291 [inlined]
    [8] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/compiler/execution.jl:110 [inlined]
    [9] square!
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:19 [inlined]
   [10] diffejulia_square__19124wrap
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:0
   [11] macro expansion
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:7049 [inlined]
   [12] enzyme_call
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6658 [inlined]
   [13] CombinedAdjointThunk
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/compiler.jl:6535 [inlined]
   [14] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:320 [inlined]
   [15] autodiff
      @ ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:348 [inlined]
   [16] autodiff(mode::ReverseMode{false, FFIABI, false, false}, f::typeof(square!), args::Duplicated{CuArray{Float32, 1, CUDA.DeviceMemory}})
      @ Enzyme ~/.cache/julia-buildkite-plugin/depots/3cc01fab-3357-4a7a-9294-cde2d3115a97/dev/Enzyme/src/Enzyme.jl:329
   [17] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:47 [inlined]
   [18] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [19] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/extensions/enzyme.jl:43
   [20] include
      @ ./client.jl:489 [inlined]
   [21] #11
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/runtests.jl:87 [inlined]
   [22] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [23] macro expansion
      @ ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Test/src/Test.jl:1577 [inlined]
   [24] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:63 [inlined]
   [25] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/utilities.jl:35 [inlined]
   [26] macro expansion
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/src/memory.jl:829 [inlined]
   [27] top-level scope
      @ /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:62
   [28] eval
      @ ./boot.jl:385 [inlined]
   [29] runtests(f::Function, name::String, time_source::Symbol)
      @ Main /var/lib/buildkite-agent/builds/gpuci-4/julialang/cuda-dot-jl/test/setup.jl:74
   [30] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::@Kwargs{})
      @ Base ./essentials.jl:892
   [31] invokelatest(::Any, ::Any, ::Vararg{Any})
      @ Base ./essentials.jl:889
   [32] (::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287
   [33] run_work_thunk(thunk::Distributed.var"#110#112"{Distributed.CallMsg{:call_fetch}}, print_error::Bool)
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:70
   [34] (::Distributed.var"#109#111"{Distributed.CallMsg{:call_fetch}, Distributed.MsgHeader, Sockets.TCPSocket})()
      @ Distributed ~/.cache/julia-buildkite-plugin/julia_installs/bin/linux/x64/1.10/julia-1.10-latest-linux-x86_64/share/julia/stdlib/v1.10/Distributed/src/process_messages.jl:287

TBH I'm getting kinda annoyed by the amount of breakage of the Enzyme.jl reverse CI job. Please add CUDA.jl CI here so that the Enzyme extension doesn't break that often...

[wsmoses]

So we do have a CUDA CI.....which passes......https://buildkite.com/julialang/enzyme-dot-jl/builds/3855#01917b02-f5bc-453b-a7ef-68591a5f3c89

[wsmoses]

@maleadt did GPUCompiler/etc change gpu_report_exception change names to report_exception?

That's what it looks like?

[maleadt]

No, nothing should have changed in that regard.

[wsmoses]

Well in any case, let's see if that fixes it: #1746

[maleadt]

What about the other errors? See the PR JuliaGPU/CUDA.jl#2472 -- is Enzyme so sensitive to a minor change in cudaconvert?

[wsmoses]

I think that's genuinely due to the PR. The kernel call is changed, which seems to be the issue there.

Specifically, we hook into the kernel call for a custom rule. The previous defn was

function (kernel::HostKernel)(args::Vararg{Any,N}; threads::CuDim=1, blocks::CuDim=1, kwargs...) where {N}

Now with that PR it is

@inline @generated function (kernel::AbstractKernel{F,TT})(args...; call_kwargs...) where {F,TT}

Not sure if that specifically is the issue, but I do think the issue is from that PR itself

[wsmoses]

Closing as needs resolution in cuda.jl extension and specific to PR at hand.

[maleadt]

The error I reported at the top of my issue remains:

  Calling an intrinsic function that clashes with an existing definition: void (i64) report_exception

JuliaGPU/CUDA.jl#2476

[wsmoses]

Was indeed caused by a GPUCompiler bump, resolved in JuliaGPU/GPUCompiler.jl#619