Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error when fitting with CUDA not functional (adding conditional to fix) #4

Merged
merged 6 commits into from
Apr 11, 2024

Conversation

pat-alt
Copy link
Contributor

@pat-alt pat-alt commented Apr 9, 2024

Finally having a first go at adding an extension to CounterfactualExplanations.jl and noticed that trying to fit a model when CUDA is not functional throws an error:

julia> using CounterfactualExplanations
       using MLJBase
       using NeuroTreeModels
       using TaijaData
       
       m = NeuroTreeRegressor(; depth=5, nrounds=10)
       X, y = @load_boston
       mach = machine(m, X, y) 
       MLJBase.fit!(mach)
┌ Warning: Package cuDNN not found in current path.
│ - Run `import Pkg; Pkg.add("cuDNN")` to install the cuDNN package, then restart julia.
│ - If cuDNN is not installed, some Flux functionalities will not be available when running on the GPU.
└ @ FluxCUDAExt ~/.julia/packages/Flux/Wz6D4/ext/FluxCUDAExt/FluxCUDAExt.jl:57
[ Info: Precompiling NeuroTreeModels [1db4e0a5-a364-4b0c-897c-2bd5a4a3a1f2]
┌ Warning: Package cuDNN not found in current path.
│ - Run `import Pkg; Pkg.add("cuDNN")` to install the cuDNN package, then restart julia.
│ - If cuDNN is not installed, some Flux functionalities will not be available when running on the GPU.
└ @ FluxCUDAExt ~/.julia/packages/Flux/Wz6D4/ext/FluxCUDAExt/FluxCUDAExt.jl:57
WARNING: Method definition supports_weights(Type{var"#s27"} where var"#s27"<:NeuroTreeModels.NeuroTreeRegressor) in module NeuroTreeModels at /Users/paltmeyer/.julia/packages/NeuroTreeModels/DXITM/src/MLJ.jl:71 overwritten at /Users/paltmeyer/.julia/packages/MLJModelInterface/ihpHk/src/metadata_utils.jl:62.
ERROR: Method overwriting is not permitted during Module precompilation. Use `__precompile__(false)` to opt-out of precompilation.
[ Info: Skipping precompilation since __precompile__(false). Importing NeuroTreeModels [1db4e0a5-a364-4b0c-897c-2bd5a4a3a1f2].
[ Info: Precompiling NeuroTreeExt [3b9b29e0-2f41-503c-9947-dc8035483bb2]
┌ Warning: Package cuDNN not found in current path.
│ - Run `import Pkg; Pkg.add("cuDNN")` to install the cuDNN package, then restart julia.
│ - If cuDNN is not installed, some Flux functionalities will not be available when running on the GPU.
└ @ FluxCUDAExt ~/.julia/packages/Flux/Wz6D4/ext/FluxCUDAExt/FluxCUDAExt.jl:57
┌ Warning: Module NeuroTreeModels with build ID ffffffff-ffff-ffff-0001-d43218202e04 is missing from the cache.
│ This may mean NeuroTreeModels [1db4e0a5-a364-4b0c-897c-2bd5a4a3a1f2] does not support precompilation but is imported by a module that does.
└ @ Base loading.jl:1948
[ Info: Skipping precompilation since __precompile__(false). Importing NeuroTreeExt [3b9b29e0-2f41-503c-9947-dc8035483bb2].
[ Info: Following 14 arguments were not provided and will be set to default: batchsize, MLE_tree_split, gpuID, ntrees, hidden_size, wd, actA, loss, rng, init_scale, device, outsize, lr, stack_size.
[ Info: Training machine(NeuroTreeRegressor(loss = mse, …), …).
[ Info: ["Crim", "Zn", "Indus", "NOx", "Rm", "Age", "Dis", "Rad", "Tax", "PTRatio", "Black", "LStat"]
┌ Error: Problem fitting the machine machine(NeuroTreeRegressor(loss = mse, …), …). 
└ @ MLJBase ~/.julia/packages/MLJBase/iIhiI/src/machines.jl:683
[ Info: Running type checks... 
[ Info: Type checks okay. 
ERROR: CUDA driver not found
Stacktrace:
  [1] error(s::String)
    @ Base ./error.jl:35
  [2] functional
    @ ~/.julia/packages/CUDA/htRwP/src/initialization.jl:24 [inlined]
  [3] task_local_state!()
    @ CUDA ~/.julia/packages/CUDA/htRwP/lib/cudadrv/state.jl:77
  [4] device
    @ ~/.julia/packages/CUDA/htRwP/lib/cudadrv/state.jl:191 [inlined]
  [5] reclaim(sz::Int64)
    @ CUDA ~/.julia/packages/CUDA/htRwP/src/pool.jl:532
  [6] reclaim
    @ ~/.julia/packages/CUDA/htRwP/src/pool.jl:532 [inlined]
  [7] fit_iter!(m::NeuroTreeModels.NeuroTreeModel{…}, cache::@NamedTuple{…})
    @ NeuroTreeModels ~/.julia/packages/NeuroTreeModels/DXITM/src/fit.jl:149
  [8] fit(model::NeuroTreeRegressor, verbosity::Int64, A::@NamedTuple{…}, y::SubArray{…}, w::Nothing)
    @ NeuroTreeModels ~/.julia/packages/NeuroTreeModels/DXITM/src/MLJ.jl:28
  [9] fit(model::NeuroTreeRegressor, verbosity::Int64, A::@NamedTuple{…}, y::SubArray{…})
    @ NeuroTreeModels ~/.julia/packages/NeuroTreeModels/DXITM/src/MLJ.jl:8
 [10] fit_only!(mach::Machine{NeuroTreeRegressor, true}; rows::Nothing, verbosity::Int64, force::Bool, composite::Nothing)
    @ MLJBase ~/.julia/packages/MLJBase/iIhiI/src/machines.jl:681
 [11] fit_only!
    @ ~/.julia/packages/MLJBase/iIhiI/src/machines.jl:607 [inlined]
 [12] #fit!#63
    @ ~/.julia/packages/MLJBase/iIhiI/src/machines.jl:778 [inlined]
 [13] fit!(mach::Machine{NeuroTreeRegressor, true})
    @ MLJBase ~/.julia/packages/MLJBase/iIhiI/src/machines.jl:775
 [14] top-level scope
    @ ~/code/CounterfactualExplanations.jl/test/models/neurotree.jl:9
Some type information was truncated. Use `show(err)` to see complete types.

This PR just adds a conditional statement to avoid this error.

@pat-alt pat-alt changed the title adding conditional Error when fitting with CUDA not functional (adding conditional to fix) Apr 9, 2024
@jeremiedb jeremiedb merged commit 63f30f5 into Evovest:main Apr 11, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants