Discrepancy in DAS Metric Calculation

[mrclovvn]

Hi,

Thank you for sharing your valuable work on the "SC3K" paper presented at ICCV 2023. I greatly appreciate the contributions your team has made to this field. However, I wanted to bring a potential discrepancy to your attention regarding the calculation of the DAS metric in your implementation, which raises concerns about the DAS performance reported in the paper.

The paper mentions that the DAS metric follows the approach described in the CVPR 2021 paper, "Skeleton Merger (SM)." However, after reviewing the code implementation in your repository, I noticed differences compared to the method described in the SM paper. Specifically, in SM, the DAS metric is calculated by evaluating both the forward alignment score and the backward alignment score, based on the consistency of keypoint alignments with manually annotated semantic_ids, and the final DAS is computed by averaging these two scores. This dual alignment process seems to be absent in the current implementation, which also does not use the annotations from the KeypointNet dataset for evaluating DAS.

As a result, the DAS metric output from your code appears to differ from the results in the SM implementation. Clarifying and addressing these differences would enhance the accuracy of the paper and its corresponding implementation.

Please let me know if further clarification is needed or if any additional details would be helpful.

Best regards,
Shawn.