Profiling of the DeepMD-kit

[AnuragKr]

I want to do profiling of the deepmd-kit to understand the complete flow and which Cuda kernel taking more time or less.
The command I used -

1. CUDA_VISIBLE_DEVICES=1 mpirun -l -launcher=fork -hosts=localhost -np 1 nsys profile -o deepmd_report --status=true dp train --mpi-log=workers input.json

2. nsys profile -o deepmd_nsys_report --stats=true python train.py
   

Another Issue I am encountering --
I am making changes locally to the code but I am unable to see any changes. I have installed via source code.

@denghuilu Please suggest where I am going wrong. I am using nsys-cli.

[denghuilu]

I think the execution time of the profiling progress was too long. You can control the number of training steps to be about 10 steps by modifying the parameters in the training script. In fact, the DP training of each step basically goes through the same network size, so a short-time profiling still works.

[AnuragKr]

I tried to do profiling using this command by reducing the training step to 100--
Command ---
CUDA_VISIBLE_DEVICES=0,1 mpirun -l -launcher=fork -hosts=localhost -np 1 nsys profile -o single_gpu_profile_100 -t cuda,mpi,nvtx --delay 30 --duration 20 dp train --mpi-log=workers input.json
Output --
single_gpu_profile_100.nsys-rep

For GUI visualization I opened this output file using via Nsight System following output I got--

I want to know whether I am doing it correctly or What are the steps to do for analyzing via nsys.

[denghuilu]

Maybe you can profiling through the nsys command line tool and then view the relevant results locally. This can avoid some communication link problems.

[AnuragKr]

Hi @denghuilu ,

Thanks for your response to the problem mentioned above. Now I am trying to calculate FLOPS using these commands--

1. Command -- nv-nsight-cu-cli --devices 0 --query-metrics -o single_gpu_ice_triple_I dp train input.json
   Output --
   

2. Command -- nvprof --metrics flops_sp --metrics flops_sp_add --metrics flops_sp_mul --metrics flops_sp_fma --metrics flops_sp_special dp train input.json
   Output --
   

Please Suggest --

1. Which approach to use to calculate FLOP or Kernel level profiling.In research paper it is mentioned nvprof used.
2. For nvprof command anything I am doing it worong.

[AnuragKr]

@denghuilu Please guide me in resolving the FLOP count issue --
Q1.) Did you come across this issue while doing profiling -- link. Is it mandatory to give hardware access to do profiling?
Q2.) What metrics look out for evaluating performance?
My profiling window looks like this --

[AnuragKr]

Solution--

1. Issue -- ERR_NVGPUCTRPERM: Permission issue with Performance Counters
   Steps to solve (Ask System Administrator to follow these steps)--

   1. Write

      options nvidia "NVreg_RestrictProfilingToAdminUsers=0 

      into /etc/modprobe.d/nvidia-prof.conf

   2.   sudo update-initramfs -u -k all

   3. Reboot

2. Kernel level profiling and FLOP count can be done now using Nsight compute rather than nvprof.

   CUDA_VISIBLE_DEVICES=1 ncu --target-processes all -o profile mpirun -np 1 dp train  input.json

For more advanced profiling options - https://docs.nvidia.com/nsight-compute/NsightComputeCli/index.html