Add --enable-p2p-check option

[hnyls2002]

Not only 4090, there are a lot of old GPUs that cannot support p2p access and we cannot always skip the p2p checking process.

The RTX 40 hack can also be discarded as we can use --enable-p2p-check now.

[hnyls2002]

This PR is related to #531 and may probably fix #552.

@Qubitium @ZX-ModelCloud I don't know whether just checking for 4090 would cause errors or not. Is there anyone who can help me test with 4090 using --enable-p2p-checking?

[ZX-ModelCloud]

This PR is related to #531 and may probably fix #552.

@Qubitium @ZX-ModelCloud I don't know whether just checking for 4090 would cause errors or not. Is there anyone who can help me test with 4090 using --enable-p2p-checking?

When loading the model with two GPUs 4090 and --tp-size =2. Add --enable-p2p-check parameter, load model works fine and inference works fine. Without adding it, it doesn't work.

[hnyls2002]

@ZX-ModelCloud Ok, so I think making this as an option is better than the hack of skip checking only for RTX 40 series.

When people get errors, they should add this option manually.

[wavelet2008]

too slow add --enable-p2p-check