Query About Training & Evaluation Strategy

[AmoghTiwari]

Hello,

Thank you for your interesting work, and for making the source code public.

Could you please clarify if you use the same pretrained checkpoint to evaluate your method on all 3 datasets (3DPW, Human3.6M and MPI-INF-3DHP) or do you use different models during evaluation? As per the paper, I thought that you are using a single checkpoint for evaluations, but looking at the evaluation instructions and this issue, I understand that you are using different checkpoints for evaluating on different datasets. Hence, I am confused. Kindly clarify.

[sxl142]

hi, we follow the previous work TCMR, which uses different checkpoints for evaluating on different datasets.
We provide two checkpoints, one is for 3DPW, and the other is for the MPI and H36M.

[AmoghTiwari]

Thank you for the clarification. However, I am still slightly confused.

As per my understanding, in TCMR, the two different checkpoints are for 2 cases - (1) When they use 3DPW during training (Table-4), and (2) When they do not use 3DPW during training (Table-6). However, for table-4, they use a single checkpoint. However, in your case, there are 3 models - (1A) repr_table1_3dpw.yaml where you use 3DPW with some set of weights; (1B) repr_table1_h36m_mpii3d.yaml where you still use 3DPW, but with a different set of weights; (2) repr_table2_3dpw.yaml where you do not use 3DPW for training.

So, I understand that TCMR uses 2 different models for their table-4 and table-6. However, for table-4, they are using a single model. But in your case, for table-1 itself, there are 2 different checkpoint which you use, and then for table-2, there is a third checkpoint, right?

[sxl142]

Thank you for the clarification. However, I am still slightly confused.

As per my understanding, in TCMR, the two different checkpoints are for 2 cases - (1) When they use 3DPW during training (Table-4), and (2) When they do not use 3DPW during training (Table-6). However, for table-4, they use a single checkpoint. However, in your case, there are 3 models - (1A) repr_table1_3dpw.yaml where you use 3DPW with some set of weights; (1B) repr_table1_h36m_mpii3d.yaml where you still use 3DPW, but with a different set of weights; (2) repr_table2_3dpw.yaml where you do not use 3DPW for training.

So, I understand that TCMR uses 2 different models for their table-4 and table-6. However, for table-4, they are using a single model. But in your case, for table-1 itself, there are 2 different checkpoint which you use, and then for table-2, there is a third checkpoint, right?

Tcmr provides two configs about table 4 and the contents are different. Thus tcmr uses these two configs to train two models. In https://drive.google.com/drive/folders/1NxzmKw5QTGtOKgSQetkq66ZO-dbBSzrd, you can see that there are two checkpoints related to Table 4.

[AmoghTiwari]

Oh okay, I realise that I was mistaken about TCMR's strategy earlier. Hence the confusion. I understand it now. Thanks a lot for clarifying.

By the way, I also noticed that in the evaluate.py file, in lines 142-146, for repr_table4_h36m_mpii3d_model case h36m_test_25fps_db.pt is used. But for the repr_table6_h36m_model case, h36m_test_front_25fps_tight_db.pt is used. I understand, that this is also taken from TCMR's strategy. But do you have any insights on what is the difference between the 2 .pt files? And why two different files have been used ?