ATL as an off-the-shelf affordance recognition module

[RomeroBarata]

Hi,

Thanks for your excellent work. Is the pre-trained ATL model suitable as an off-the-shelf affordance recognition module for object images extracted from other datasets? If so, could you please give me some pointers where to look for around in the codebase for that? Thank your for your attention.

Kind regards,
Romero

[zhihou7]

Hi,

Thanks for your interest. I think it is ok. You can find how to utilize the pre-trained ATL model for affordance recognition in https://github.com/zhihou7/HOI-CL/blob/master/GETTING_STARTED_ATL.md#3-affordance-recognition. The code is in this directory: https://github.com/zhihou7/HOI-CL/tree/master/scripts/affordance

feel free to contact me if you have further questions.

Regards,

[RomeroBarata]

Hi,

Thanks for your reply. I'll go through these links and try to set it up on my own images. I'll keep this issue open for a while in case I have further questions.

Regards,
Romero

[RomeroBarata]

Hi zhihou7,

I'm currently downloading the provided data to run the scripts, but I'm unable to download this file with the following command (from the misc/download_dataset.sh script)
python lib/ult/Download_data.py 1hIElxTyJ0HrTww_p1GpHD9KLZNw8OVJH Data/hico_20160224_det.tar.gz

I'm getting a file that is 141 bytes and cannot be extracted (so I assume it's corrupted). Could you please update the link or provide an alternative link for download? Thanks.

[zhihou7]

this url is avialable: https://drive.google.com/file/d/1QZcJmGVlF9f4h-XLWe9Gkmnmj2z1gSnk/view?usp=sharing

python lib/ult/Download_data.py 1QZcJmGVlF9f4h-XLWe9Gkmnmj2z1gSnk Data/hico_20160224_det.tar.gz

[RomeroBarata]

Hi zhihou7,

Thanks for the link! I'm now having the same issue with the following file:
python lib/ult/Download_data.py 0B1_fAEgxdnvJR1N3c1FYRGo1S1U Weights/coco_900-1190k.tgz

Could please also provide an alternative download for it? Thanks again.

[zhihou7]

here,

https://drive.google.com/file/d/0B1_fAEgxdnvJR1N3c1FYRGo1S1U/view?usp=sharing&resourcekey=0-HM6zp80Nx1O23zyMKCuU9g

[RomeroBarata]

Hi zhihou7,

I've followed the instructions from the pointers you provided me (and the set up instructions in the main repository page), but when I try to run
python scripts/affordance/extract_affordance_feature.py

I get the following error (many of them actually, I'm just pasting a truncated one here as an example)
TypeError: generator yielded an element that did not match the expected structure. The expected structure was (tf.float32, tf.int32, tf.int32, {'H_boxes': tf.float32, 'Hsp_boxes': tf.float32, 'O_boxes': tf.float32, 'gt_class_sp': tf.float32, 'gt_class_HO': tf.float32, 'gt_class_H': tf.float32, 'gt_class_C': tf.float32, 'gt_class_obj': tf.float32, 'Mask_sp': tf.float32, 'Mask_HO': tf.float32, 'Mask_H': tf.float32, 'sp': tf.float32, 'O_mask': tf.float32}), but the yielded element was (array ...

The only modification I did to the script was to change the input model to ATL_union_multi_atl_ml5_l05_t5_def2_aug5_3_new_VCOCO_test_coco_CL_21, since the default one in the script I couldn't find it for download in the pre-trained models provided.

Would have any idea of what could be the cause? It's tricky for me to debug since these errors are coming from within tensorflow (I'm using version 1.14.0). Thank you for your attention.

[zhihou7]

sorry for this issue. I think it is a bug when i prepare for the code. I will check the input again tonight，and debug it. Sorry for this issue

It is because I remove some useless input data. And the input list does not match. I will fix it.

[RomeroBarata]

No worries, thank you very much for your support and patience.

[zhihou7]

Hi, I have updated the code for affordance recognition on V-COCO.

You should first download "Test_GT_HICO_COCO_HICO.pkl" (https://cloudstor.aarnet.edu.au/plus/s/AFrv822lPC30iHt) again cause the old file is wrong.

Meanwhile, I have updated the GETTING_STARTED_ATL.md with more detailed instructions. For object prediction on HICO-DET test set (gthico) with model of V-COCO, it requires about 3-4hours to obtain the results.

feel free to post comment in this issue if you have further questions.

Regards,
Zhi Hou

[RomeroBarata]

Hi zhihou7,

Thank you again for looking into it.

I see that you updated
python scripts/affordance/extract_affordance_feature.py

to
python scripts/affordance/extract_affordance_feature.py --num_iteration 100000 --model ATL_union_multi_base_l05_t5_aug5_3_new_VCOCO_test_CL_21

I looked through the provided models but I cannot find this one for download ATL_union_multi_base_l05_t5_aug5_3_new_VCOCO_test_CL_21. Would have a link for it? Thanks.

[zhihou7]

oh, that's not necessary. I just use an example (that's the baseline)for testing. Sorry for the confusion. You can just use the released model.

#V-COCO
python scripts/affordance/extract_affordance_feature.py --num_iteration 160000 --model ATL_union_multi_atl_ml5_l05_t5_def2_aug5_3_new_VCOCO_test_coco_CL_21

python scripts/affordance/extract_obj_feature.py --type gthico --num_iteration 160000 --model ATL_union_multi_atl_ml5_l05_t5_def2_aug5_3_new_VCOCO_test_coco_CL_21

python scripts/affordance/obtain_hoi_preds.py --num_iteration 160000 --model ATL_union_multi_atl_ml5_l05_t5_def2_aug5_3_new_VCOCO_test_coco_CL_21 --dataset gthico

python scripts/affordance/stat_vcoco_affordance.py gthico ATL_union_multi_atl_ml5_l05_t5_def2_aug5_3_new_VCOCO_test_coco_CL_21

I have updated GETTING_STARTED_ATL.md again.