Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Query About Training & Evaluation Strategy #11

Open
AmoghTiwari opened this issue Jan 13, 2024 · 4 comments
Open

Query About Training & Evaluation Strategy #11

AmoghTiwari opened this issue Jan 13, 2024 · 4 comments

Comments

@AmoghTiwari
Copy link

AmoghTiwari commented Jan 13, 2024

Hello,

Thank you for your interesting work, and for making the source code public.

Could you please clarify if you use the same pretrained checkpoint to evaluate your method on all 3 datasets (3DPW, Human3.6M and MPI-INF-3DHP) or do you use different models during evaluation? As per the paper, I thought that you are using a single checkpoint for evaluations, but looking at the evaluation instructions and this issue, I understand that you are using different checkpoints for evaluating on different datasets. Hence, I am confused. Kindly clarify.

@sxl142
Copy link
Owner

sxl142 commented Jan 16, 2024

hi, we follow the previous work TCMR, which uses different checkpoints for evaluating on different datasets.
We provide two checkpoints, one is for 3DPW, and the other is for the MPI and H36M.

@AmoghTiwari
Copy link
Author

Thank you for the clarification. However, I am still slightly confused.

As per my understanding, in TCMR, the two different checkpoints are for 2 cases - (1) When they use 3DPW during training (Table-4), and (2) When they do not use 3DPW during training (Table-6). However, for table-4, they use a single checkpoint. However, in your case, there are 3 models - (1A) repr_table1_3dpw.yaml where you use 3DPW with some set of weights; (1B) repr_table1_h36m_mpii3d.yaml where you still use 3DPW, but with a different set of weights; (2) repr_table2_3dpw.yaml where you do not use 3DPW for training.

So, I understand that TCMR uses 2 different models for their table-4 and table-6. However, for table-4, they are using a single model. But in your case, for table-1 itself, there are 2 different checkpoint which you use, and then for table-2, there is a third checkpoint, right?

@sxl142
Copy link
Owner

sxl142 commented Jan 23, 2024

Thank you for the clarification. However, I am still slightly confused.

As per my understanding, in TCMR, the two different checkpoints are for 2 cases - (1) When they use 3DPW during training (Table-4), and (2) When they do not use 3DPW during training (Table-6). However, for table-4, they use a single checkpoint. However, in your case, there are 3 models - (1A) repr_table1_3dpw.yaml where you use 3DPW with some set of weights; (1B) repr_table1_h36m_mpii3d.yaml where you still use 3DPW, but with a different set of weights; (2) repr_table2_3dpw.yaml where you do not use 3DPW for training.

So, I understand that TCMR uses 2 different models for their table-4 and table-6. However, for table-4, they are using a single model. But in your case, for table-1 itself, there are 2 different checkpoint which you use, and then for table-2, there is a third checkpoint, right?

Tcmr provides two configs about table 4 and the contents are different. Thus tcmr uses these two configs to train two models. In https://drive.google.com/drive/folders/1NxzmKw5QTGtOKgSQetkq66ZO-dbBSzrd, you can see that there are two checkpoints related to Table 4.

@AmoghTiwari
Copy link
Author

Oh okay, I realise that I was mistaken about TCMR's strategy earlier. Hence the confusion. I understand it now. Thanks a lot for clarifying.

By the way, I also noticed that in the evaluate.py file, in lines 142-146, for repr_table4_h36m_mpii3d_model case h36m_test_25fps_db.pt is used. But for the repr_table6_h36m_model case, h36m_test_front_25fps_tight_db.pt is used. I understand, that this is also taken from TCMR's strategy. But do you have any insights on what is the difference between the 2 .pt files? And why two different files have been used ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants