Skip to content

[ICRA2023] CoAlign: Robust Collaborative 3D Object Detection in Presence of Pose Errors

License

Notifications You must be signed in to change notification settings

yifanlu0227/CoAlign

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

CoAlign (ICRA2023)

Robust Collaborative 3D Object Detection in Presence of Pose Errors

Paper | VideoReadme in Feishu

Original1

Update🌟 2024.1.24

HEAL is accepted to ICLR 2024. We implement a unified and integrated multi-agent collaborative perception framework for LiDAR-based, camera-based and heterogeneous setting! See HEAL GitHub.

Update🌟 2023.7.11

Camera-based collaborative perception support!

We release the multi-agent camera-based detection code, based on Lift-Splat-Shoot. Support OPV2V, V2XSet and DAIR-V2X-C dataset.

LiDAR's feature map fusion method can seamlessly adapt to camera BEV feature. Support CoAlign's multiscale fusion, V2XViT, V2VNet, Self-Att, FCooper, DiscoNet(w.o. KD). Please feel free to browse our repo. Example yamls are listed in this folder: CoAlign/opencood/hypes_yaml/opv2v/camera_no_noise

New features (Compared with OpenCOOD):

Installation

Please visit the feishu docs CoAlign Installation Guide for details!

Or you can refer to OpenCOOD data introduction and OpenCOOD installation guide to prepare data and install CoAlign. The installation is totally the same as OpenCOOD, except some dependent packages required by CoAlign.

Data Preparation

mkdir a dataset folder under CoAlign. Put your OPV2V, V2X-Sim, V2XSet, DAIR-V2X data in this folder. You just need to put in the dataset you want to use.

CoAlign/dataset

. 
├── my_dair_v2x 
│   ├── v2x_c
│   ├── v2x_i
│   └── v2x_v
├── OPV2V
│   ├── additional
│   ├── test
│   ├── train
│   └── validate
├── V2XSET
│   ├── test
│   ├── train
│   └── validate
├── v2xsim2-complete
│   ├── lidarseg
│   ├── maps
│   ├── sweeps
│   └── v1.0-mini
└── v2xsim2_info
    ├── v2xsim_infos_test.pkl
    ├── v2xsim_infos_train.pkl
    └── v2xsim_infos_val.pkl

Note that

  1. *.pkl file in v2xsim2_info can be found in Google Drive
  2. use our complemented annotation for DAIR-V2X in my_dair_v2x

Complemented Annotations for DAIR-V2X-C 🌟

Originally DAIR-V2X only annotates 3D boxes within the range of camera's view in vehicle-side. We supplement the missing 3D box annotations to enable the 360 degree detection. With fully complemented vehicle-side labels, we regenerate the cooperative labels for users, which follow the original cooperative label format.

Original Annotations Complemented Annotations
Original1 Complemented1
Original2 Complemented2
Original3 Complemented3

Download: Google Drive

Website: Website

Checkpoints

Single detection with uncertainty

Download coalign_precalc and save it to opencood/logs

CoAlign Checkpoints

Download them and save them to opencood/logs

Citation

@inproceedings{lu2023robust,
  title={Robust collaborative 3d object detection in presence of pose errors},
  author={Lu, Yifan and Li, Quanhao and Liu, Baoan and Dianati, Mehrdad and Feng, Chen and Chen, Siheng and Wang, Yanfeng},
  booktitle={2023 IEEE International Conference on Robotics and Automation (ICRA)},
  pages={4812--4818},
  year={2023},
  organization={IEEE}
}

Acknowlege

This project is impossible without the code of OpenCOOD, g2opy and d3d!

Thanks again to @DerrickXuNu for the great code framework.

Q&A

  1. Different AP results between arxiv v2 and arxiv v3? and different from OPV2V[ICRA 22']?

    See Issue 4.

  2. How to get V2X-Sim-2.0 pickle file?

    See Issue 2.

About

[ICRA2023] CoAlign: Robust Collaborative 3D Object Detection in Presence of Pose Errors

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published