RuntimeError CUDA error when running Infinite Bench

[Flitieter]

Hi expert, recently I ran the code in Infinite Bench (the GPU is RTX4090 24GB, and the cuda version is 12.1.105).

However, when the code ran the dataset about kv_retrieval, it threw an exception "RuntimeError CUDA error"(the specialed error case id is 281, and the code was ok in the dataset Passkey). I checked the pips' version and found they are all consistent with what has been written in the document "ktransformers/server/requirements.txt".

So I want to know whether there are still some problems in my environment.

Here are some pips' versions in my environment:

python==3.11.0
torch==2.3.0
transformers == 4.43.2
fastapi==0.115.4
langchain==0.3.7
blessed==1.20.0
accelerate==1.1.0
sentencepiece==0.2.0

And the complete error message is: