想咨询下，你们在显存方面有哪些建议和优化

[datalee]

稍微长点的文档在32G的v100上就爆点了
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.13 GiB (GPU 0; 31.75 GiB total capacity; 18.88 GiB already allocated; 7.01 GiB free; 20.61 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

[miange91]

稍微长点的文档在32G的v100上就爆点了 torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.13 GiB (GPU 0; 31.75 GiB total capacity; 18.88 GiB already allocated; 7.01 GiB free; 20.61 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

可以用tensorRT-LLM来跑推理

[datalee]

稍微长点的文档在32G的v100上就爆点了 torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 34.13 GiB (GPU 0; 31.75 GiB total capacity; 18.88 GiB already allocated; 7.01 GiB free; 20.61 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

可以用tensorRT-LLM来跑推理

有一文档可以参考吗？是选用LLAMA2结构？