torch.compile VIT-H-14 and VIT-L-14

[dhaivatd]

I'm trying to improve inference latency for VIT-H-14 and VIT-L-14 using torch compile following @rwightman's suggestion.

The inference latency is actually worse with default torch.compile on torch 2.2 as compared to running forward pass with just torch.save'd model. Are there any compile flags or settings I can use to improve?