From 2fb620a144f3acf7da00c0f9b32b78d6c05056e5 Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Tue, 27 Feb 2024 22:47:21 +0800 Subject: [PATCH 1/8] Update train_test.md Update "submission_prefix" to "test_evaluator.submission_prefix". Update the waymo links, because the original link is 404. --- docs/en/user_guides/train_test.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en/user_guides/train_test.md b/docs/en/user_guides/train_test.md index e3a85d3aee..60d4f374de 100644 --- a/docs/en/user_guides/train_test.md +++ b/docs/en/user_guides/train_test.md @@ -89,7 +89,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \ checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results' + --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'test_evaluator.submission_prefix=./second_kitti_results' ``` The generated results be under `./second_kitti_results` directory. @@ -117,7 +117,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. + **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. 9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard. @@ -128,7 +128,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission. + **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission. ## Train predefined models on standard datasets From 42d26ee6e1b831269ea55364d769527254c38a1a Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Thu, 29 Feb 2024 12:44:26 +0800 Subject: [PATCH 2/8] Update CeterFormer README.md Update the config file's name, because the actual name is changed but the instruction isn't updated --- projects/CenterFormer/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/projects/CenterFormer/README.md b/projects/CenterFormer/README.md index f84556b695..bbff8df5ea 100644 --- a/projects/CenterFormer/README.md +++ b/projects/CenterFormer/README.md @@ -43,13 +43,13 @@ We implement CenterFormer and provide the results and checkpoints on Waymo datas In MMDetection3D's root directory, run the following command to train the model: ```bash -python tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py +python tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class.py ``` For multi-gpu training, run: ```bash -python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py +python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${NUM_GPUS} --master_port=29506 --master_addr="127.0.0.1" tools/train.py projects/CenterFormer/configs/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class.py ``` ### Testing commands @@ -57,7 +57,7 @@ python -m torch.distributed.launch --nnodes=1 --node_rank=0 --nproc_per_node=${N In MMDetection3D's root directory, run the following command to test the model: ```bash -python tools/test.py projects/CenterFormer/configs/centerformer_voxel01_second-atten_secfpn-atten_4xb4-cyclic-20e_waymoD5-3d-3class.py ${CHECKPOINT_PATH} +python tools/test.py projects/CenterFormer/configs/centerformer_voxel01_second-attn_secfpn-attn_4xb4-cyclic-20e_waymoD5-3d-3class.py ${CHECKPOINT_PATH} ``` ## Results and models From 3e2a2c8cbf06980bcd270ae4aa99127920a368c8 Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Thu, 29 Feb 2024 16:15:55 +0800 Subject: [PATCH 3/8] Update train_test.md the link about waymo --- docs/en/user_guides/train_test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/user_guides/train_test.md b/docs/en/user_guides/train_test.md index 60d4f374de..09e6799864 100644 --- a/docs/en/user_guides/train_test.md +++ b/docs/en/user_guides/train_test.md @@ -117,7 +117,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. + **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) or the [tutorial](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. 9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard. From 5473453696c060c5a5cad8ec2f1a1f0d5359730b Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Thu, 29 Feb 2024 16:19:28 +0800 Subject: [PATCH 4/8] Update train_test.md some waymo links and the test_evaluator.submission_prefix --- docs/zh_cn/user_guides/train_test.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/zh_cn/user_guides/train_test.md b/docs/zh_cn/user_guides/train_test.md index 3b475a444e..c72adae2a1 100644 --- a/docs/zh_cn/user_guides/train_test.md +++ b/docs/zh_cn/user_guides/train_test.md @@ -88,7 +88,7 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/second/second_hv_secfpn_8xb6-80e_kitti-3d-3class.py \ checkpoints/hv_second_secfpn_6x8_80e_kitti-3d-3class_20200620_230238-9208083a.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'submission_prefix=./second_kitti_results' + --cfg-options 'test_evaluator.pklfile_prefix=./second_kitti_results' 'test_evaluator.submission_prefix=./second_kitti_results' ``` 生成的结果会保存在 `./second_kitti_results` 目录。 @@ -116,7 +116,7 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在 `mmdet3d/core/evaluation/waymo_utils/`。（在使用 bazel 构建 `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。 + **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)或[教程](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在 `mmdet3d/core/evaluation/waymo_utils/`。（在使用 bazel 构建 `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。 9. 使用 8 块显卡在 waymo 数据集上测试 PointPillars，生成 bin 文件并提交到排行榜 @@ -127,7 +127,7 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **注意**：生成 bin 文件后，你可以简单地构建二进制文件 `create_submission`，并根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/master/docs/quick_start.md/)创建提交的文件。要在验证服务器上评测验证数据集，你也可以用同样的方式生成提交的文件。 + **注意**：生成 bin 文件后，你可以简单地构建二进制文件 `create_submission`，并根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)创建提交的文件。要在验证服务器上评测验证数据集，你也可以用同样的方式生成提交的文件。 ## 在标准数据集上训练预定义模型 From b7171f3c10df4087735bf9c27170789a9976d857 Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Thu, 29 Feb 2024 18:56:18 +0800 Subject: [PATCH 5/8] Update train_test.md directory of `compute_detection_metrics_main` --- docs/en/user_guides/train_test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en/user_guides/train_test.md b/docs/en/user_guides/train_test.md index 09e6799864..13fddf4c67 100644 --- a/docs/en/user_guides/train_test.md +++ b/docs/en/user_guides/train_test.md @@ -117,7 +117,7 @@ Assume that you have already downloaded the checkpoints to the directory `checkp 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) or the [tutorial](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into `mmdet3d/core/evaluation/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. + **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) or the [tutorial](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into this project's `mmdet3d/evaluation/functional/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. 9. Test PointPillars on waymo with 8 GPUs, generate the bin files and make a submission to the leaderboard. From 67c7f0693d1fa16144f65bee39641205fc33741d Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Thu, 29 Feb 2024 18:56:30 +0800 Subject: [PATCH 6/8] Update train_test.md directory of `compute_detection_metrics_main` --- docs/zh_cn/user_guides/train_test.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/zh_cn/user_guides/train_test.md b/docs/zh_cn/user_guides/train_test.md index c72adae2a1..f67d4b3770 100644 --- a/docs/zh_cn/user_guides/train_test.md +++ b/docs/zh_cn/user_guides/train_test.md @@ -116,7 +116,7 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' ``` - **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)或[教程](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在 `mmdet3d/core/evaluation/waymo_utils/`。（在使用 bazel 构建 `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。 + **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)或[教程](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在本工程 `mmdet3d/evaluation/functional/waymo_utils/`下。（在使用 bazel 构建 `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。 9. 使用 8 块显卡在 waymo 数据集上测试 PointPillars，生成 bin 文件并提交到排行榜 From 1bb4ecbc3ecaad4a1e8d4fa4fb83b1b04f7c6fb2 Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Fri, 8 Mar 2024 16:53:53 +0800 Subject: [PATCH 7/8] Update train_test.md args of `--cfg-options` when test.py on waymo --- docs/zh_cn/user_guides/train_test.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/zh_cn/user_guides/train_test.md b/docs/zh_cn/user_guides/train_test.md index f67d4b3770..cc1f052a56 100644 --- a/docs/zh_cn/user_guides/train_test.md +++ b/docs/zh_cn/user_guides/train_test.md @@ -112,8 +112,8 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py \ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \ - 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' + --cfg-options 'test_evaluator.result_prefix=results/waymo-car/kitti_results' \ + 'test_evaluator.format_only=True' ``` **注意**：对于 waymo 数据集上的评估，请根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)或[教程](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb)构建二进制文件 `compute_detection_metrics_main` 来做度量计算，并把它放在本工程 `mmdet3d/evaluation/functional/waymo_utils/`下。（在使用 bazel 构建 `compute_detection_metrics_main` 时，有时会出现 `'round' is not a member of 'std'` 的错误，我们只需要把那个文件中 `round` 前的 `std::` 去掉。）二进制文件生成时需要在 `--eval-options` 中给定 `pklfile_prefix`。对于度量方法，`waymo` 是推荐的官方评估策略，目前 `kitti` 评估是依照 KITTI 而来的，每个难度的结果和 KITTI 的定义并不完全一致。目前大多数物体都被标记为0难度，会在未来修复。它的不稳定原因包括评估的计算大、转换后的数据缺乏遮挡和截断、难度的定义不同以及平均精度的计算方法不同。 @@ -123,8 +123,8 @@ python tools/test.py ${CONFIG_FILE} ${CHECKPOINT_FILE} [--out ${RESULT_FILE}] [- ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py \ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \ - 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' + --cfg-options 'test_evaluator.result_prefix=results/waymo-car/kitti_results' \ + 'test_evaluator.format_only=True' ``` **注意**：生成 bin 文件后，你可以简单地构建二进制文件 `create_submission`，并根据[说明](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md)创建提交的文件。要在验证服务器上评测验证数据集，你也可以用同样的方式生成提交的文件。 From 5db2600ceb93b500c9a3da50ff4f13ce9f3f2f6b Mon Sep 17 00:00:00 2001 From: Yuanyuan Li <42125674+ly3106@users.noreply.github.com> Date: Fri, 8 Mar 2024 16:55:57 +0800 Subject: [PATCH 8/8] Update train_test.md args of `--cfg-options` when test.py on waymo Because the initial args of `mmdetection3d/mmdet3d/evaluation/metrics/waymo_metric.py` have been updated, but this demo instruction is not updated --- docs/en/user_guides/train_test.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/docs/en/user_guides/train_test.md b/docs/en/user_guides/train_test.md index 13fddf4c67..06cd747eba 100644 --- a/docs/en/user_guides/train_test.md +++ b/docs/en/user_guides/train_test.md @@ -113,8 +113,8 @@ Assume that you have already downloaded the checkpoints to the directory `checkp ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py \ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \ - 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' + --cfg-options 'test_evaluator.result_prefix=results/waymo-car/kitti_results' \ + 'test_evaluator.format_only=True' ``` **Notice**: For evaluation on waymo, please follow the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md) or the [tutorial](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/tutorial/tutorial.ipynb) to build the binary file `compute_detection_metrics_main` for metrics computation and put it into this project's `mmdet3d/evaluation/functional/waymo_utils/`.(Sometimes when using bazel to build `compute_detection_metrics_main`, an error `'round' is not a member of 'std'` may appear. We just need to remove the `std::` before `round` in that file.) `pklfile_prefix` should be given in the `--eval-options` for the bin file generation. For metrics, `waymo` is the recommended official evaluation prototype. Currently, evaluating with choice `kitti` is adapted from KITTI and the results for each difficulty are not exactly the same as the definition of KITTI. Instead, most of objects are marked with difficulty 0 currently, which will be fixed in the future. The reasons of its instability include the large computation for evaluation, the lack of occlusion and truncation in the converted data, different definition of difficulty and different methods of computing average precision. @@ -124,8 +124,8 @@ Assume that you have already downloaded the checkpoints to the directory `checkp ```shell ./tools/slurm_test.sh ${PARTITION} ${JOB_NAME} configs/pointpillars/pointpillars_hv_secfpn_sbn-all_16xb2-2x_waymo-3d-car.py \ checkpoints/hv_pointpillars_secfpn_sbn-2x16_2x_waymo-3d-car_latest.pth \ - --cfg-options 'test_evaluator.pklfile_prefix=results/waymo-car/kitti_results' \ - 'test_evaluator.submission_prefix=results/waymo-car/kitti_results' + --cfg-options 'test_evaluator.result_prefix=results/waymo-car/kitti_results' \ + 'test_evaluator.format_only=True' ``` **Notice**: After generating the bin file, you can simply build the binary file `create_submission` and use them to create a submission file by following the [instruction](https://github.com/waymo-research/waymo-open-dataset/blob/v1.5.0/docs/quick_start.md). For evaluation on the validation set with the eval server, you can also use the same way to generate a submission.