Kohya SS Lora Training with A100 80G

[zhao-binxiang]

What's the best setup for A100 in the Accelerate Config file? I am on 80VRAM and lots of RAM running on Runpod.

I'm a tech noob. I use A100 because it processes fast with a higher batch number.

It would be hepful to see if I could use like bf16/compile/dynamo etc.

I installed TensorRT but Kohya says "W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT"

[mrmeseeks23]

Hey there! I am having the same issue. I have checked into a lot of different sources and there doesn't seem to be a straightforward answer. I am wondering if this is even a necessary component or something that doesn't really change the game for lora training specifically. Would love to see what other folks think about this.

[zhao-binxiang]

I agree. Similar observations too. I think installing it or not... doesn't make too big of a difference to the it/s. If anyone can confirm, it woul be gold.

[mrmeseeks23]

On a side note if you can try the SXM version of the A100 I have found the processing times quicker (it/s). RTX A6000 Ada still takes the gold that or the RTX 4090 in terms of it/s at least in my experience. Though the A100's are great for large batch sizes and super high res images/ large datasets. You can also use the H100 SXM5 i believe it is called. Kind of pricey but worth it in terms of speed, much quicker than A100.

[Charmandrigo]

activate venv and type
pip install torch-tensorrt