[BugFix] Fix GC bug for LLM class

[WoosukKwon]

This PR fixes a bug that LLM class is not GC-ed even after gc.collect(), which was introduced by #1804 . It turns out that the bug happens because importing punica kernels raises an exception which is not released until the user uses LoRA.

Reproducible script:

import gc
import torch
from vllm import LLM

llm = LLM("facebook/opt-125m", enforce_eager=True)
del llm

gc.collect()
torch.cuda.empty_cache()
print(f"GPU memory usage: {torch.cuda.memory_allocated() / (1024 * 1024 * 1024):.2f} GB")

[simon-mo]

Nice sleuthing! Can you add a test for this?