You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
详细LOG:
File "/root/jiajinrang/Paddle3D/paddle3d/models/detection/bevformer/bevformer.py", line 149, in obtain_history_bev
prev_bev = self.pts_bbox_head(
File "/root/miniconda3/envs/paddle_env_1/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 948, in call
return self.forward(*inputs, **kwargs)
File "/root/jiajinrang/Paddle3D/paddle3d/models/detection/bevformer/bevformer_head.py", line 255, in forward
bev_embed = self.transformer.get_bev_features(
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/transformer.py", line 221, in get_bev_features
tmp_prev_bev = rotate(
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 217, in rotate
img = _rotate(img, matrix=matrix, interpolation=interpolation)
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 192, in _rotate
grid = _gen_affine_grid(theta, w=w, h=h, ow=ow, oh=oh)
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 127, in _gen_affine_grid
output_grid = base_grid.reshape([1, oh * ow, 3]).bmm(rescaled_theta)
File "/root/miniconda3/envs/paddle_env_1/lib/python3.8/site-packages/paddle/tensor/linalg.py", line 1730, in bmm
return _C_ops.bmm(x, y)
OSError: (External) CUBLAS error(15).
[Hint: Please search for the error code(15) on website (https://docs.nvidia.com/cuda/cublas/index.html#cublasstatus_t) to get Nvidia's official solution and advice about CUBLAS Error.] (at /paddle/paddle/phi/kernels/funcs/blas/blas_impl.cu.h:62)
I1101 12:54:37.573464 20238 tcp_store.cc:257] receive shutdown event and so quit from MasterDaemon run loop
LAUNCH INFO 2023-11-01 12:54:39,363 Exit code 1
单机环境下,无论是单卡还是多卡都报这个问题。
但是,有一个线索是,在单卡情况下,如果在train.py的main函数中加入paddle.utils.run_check(),单卡训练就能正常启动训练。但多卡情况下仍然不行。
单卡的解决方法如下所示
if name == 'main':
paddle.utils.run_check()
args = parse_args()
main(args)
Paddle3D的BevFormer训练启动时报
OSError: (External) CUBLAS error(15).
详细LOG:
File "/root/jiajinrang/Paddle3D/paddle3d/models/detection/bevformer/bevformer.py", line 149, in obtain_history_bev
prev_bev = self.pts_bbox_head(
File "/root/miniconda3/envs/paddle_env_1/lib/python3.8/site-packages/paddle/fluid/dygraph/layers.py", line 948, in call
return self.forward(*inputs, **kwargs)
File "/root/jiajinrang/Paddle3D/paddle3d/models/detection/bevformer/bevformer_head.py", line 255, in forward
bev_embed = self.transformer.get_bev_features(
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/transformer.py", line 221, in get_bev_features
tmp_prev_bev = rotate(
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 217, in rotate
img = _rotate(img, matrix=matrix, interpolation=interpolation)
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 192, in _rotate
grid = _gen_affine_grid(theta, w=w, h=h, ow=ow, oh=oh)
File "/root/jiajinrang/Paddle3D/paddle3d/models/transformers/utils.py", line 127, in _gen_affine_grid
output_grid = base_grid.reshape([1, oh * ow, 3]).bmm(rescaled_theta)
File "/root/miniconda3/envs/paddle_env_1/lib/python3.8/site-packages/paddle/tensor/linalg.py", line 1730, in bmm
return _C_ops.bmm(x, y)
OSError: (External) CUBLAS error(15).
[Hint: Please search for the error code(15) on website (https://docs.nvidia.com/cuda/cublas/index.html#cublasstatus_t) to get Nvidia's official solution and advice about CUBLAS Error.] (at /paddle/paddle/phi/kernels/funcs/blas/blas_impl.cu.h:62)
I1101 12:54:37.573464 20238 tcp_store.cc:257] receive shutdown event and so quit from MasterDaemon run loop
LAUNCH INFO 2023-11-01 12:54:39,363 Exit code 1
单机环境下,无论是单卡还是多卡都报这个问题。
但是,有一个线索是,在单卡情况下,如果在train.py的main函数中加入paddle.utils.run_check(),单卡训练就能正常启动训练。但多卡情况下仍然不行。
单卡的解决方法如下所示
if name == 'main':
paddle.utils.run_check()
args = parse_args()
main(args)
请问多卡情况下该怎么解决?
其他补充信息 Additional Supplementary Information
环境信息
Paddle version: N/A
Paddle With CUDA: N/A
OS: centos 7
GCC version: (GCC) 7.5.0
Clang version: N/A
CMake version: version 2.8.12.2
Libc version: glibc 2.17
Python version: 3.8.18
CUDA version: 11.6.112
Build cuda_11.6.r11.6/compiler.30978841_0
cuDNN version: 8.8.0
Nvidia driver version: 520.61.05
Nvidia driver List:
GPU 0: Tesla V100-SXM2-32GB
GPU 1: Tesla V100-SXM2-32GB
GPU 2: Tesla V100-SXM2-32GB
GPU 3: Tesla V100-SXM2-32GB
The text was updated successfully, but these errors were encountered: