- Website: https://waterscenes.github.io
- arXiv: https://arxiv.org/abs/2307.06505
- Zhihu: https://zhuanlan.zhihu.com/p/651979604
- Video: https://www.bilibili.com/video/BV1QH4y1m72V
- 🔥 March 28, 2024: WaterScenes dataset released at Baidu Netdisk and Google Drive
- Sep 26, 2023: Video released at Website and BiliBili
- July 21, 2023: Sample dataset released at Google Drive
- July 13, 2023: Website and Devkit released at Website and GitHub
- WaterScenes, the first multi-task 4D radar-camera fusion dataset on water surfaces, which offers data from multiple sensors, including a 4D radar, monocular camera, GPS, and IMU. It can be applied in multiple tasks, such as object detection, instance segmentation, semantic segmentation, free-space segmentation, and waterline segmentation.
- Our dataset covers diverse time conditions (daytime, nightfall, night), lighting conditions (normal, dim, strong), weather conditions (sunny, overcast, rainy, snowy) and waterway conditions (river, lake, canal, moat). An information list is also offered for retrieving specific data for experiments under different conditions.
- We provide 2D box-level and pixel-level annotations for camera images, and 3D point-level annotations for radar point clouds. We also offer precise timestamps for the synchronization of different sensors, as well as intrinsic and extrinsic parameters.
- We provide a toolkit for radar point clouds that includes: pre-processing, labeling, projection and visualization, assisting researchers in processing and analyzing our dataset.
The dataset includes 54,120 sets of RGB images, radar point clouds, GPS and IMU data, covering over 200,000 objects.
| Tasks | Object Detection / Instance Segmentation / Semantic Segmentation | Free-Space Segmentation | Waterline Segmentation | Adverse Conditions | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Classes | Total | Pier | Buoy | Sailor | Ship | Boat | Vessel | Kayak | Free-Space | Waterline | Lighting | Weather |
| Frames | 54,120 | 25,787 | 3,769 | 3,613 | 19,776 | 9,106 | 9,362 | 366 | 54,057 | 53,926 | 5,604 | 10,729 |
| Objects | 202,807 | 121,827 (60.07%) | 16,538 (8.15%) | 8,036 (3.96%) | 34,121 (16.82%) | 10,819 (5.33%) | 11,092 (5.47%) | 374 (0.18%) | 54,057 | 159,901 | 30,517 (15.05%) | 46,784 (23.07%) |
WaterScenes (root)
- image # RGB images
- 000001.jpg
- radar # radar files
- 000001.csv
- radar_3_frames # 3 frames before and after current timestamp
- 000001.csv
- radar_5_frames # 5 frames before and after current timestamp
- 000001.csv
- calib # intrisic and extrisic parameters
- 000001.txt
- gps # gps file
- 000001.csv
- imu # imu file
- 000001.csv
- detection # annotation files for object detection task
- yolo
- 000001.txt
- xml
- 000001.xml
- instance # annotation files for instance segmentation task
- yolo
- 000001.txt
- labelme
- 000001.json
- SegmentationClassPNG
- 000001.png
- class_names.txt
- semantic # annotation files for semantic segmentation task
- SegmentationClass
- 000001.png
- label_mapping.txt
- free_space # annotation files for free-space segmentation task
- SegmentationClassPNG
- 000001.png
- class_names.txt
- waterline # annotation files for waterline segmentation task
- SegmentationClass
- 000001.png
| Code | Label | Note |
|---|---|---|
| -1 | no-object | No object. Only for radar point clouds |
| 0 | pier | Static object |
| 1 | buoy | Static object |
| 2 | sailor | Person on the surface vehicles |
| 3 | ship | |
| 4 | boat | |
| 5 | vessel | |
| 6 | kayak | |
| - | free-space | Only for instance/semantic/free-space segmentation |
| - | waterline | Only for waterline segmentation |
Radar point clouds are stored in csv files.
We provide radar point clouds in three flavors:
- radar (single frame of the current timestamp)
- radar_3_frames (accumulation of the last 3 radar frames)
- radar_5_frames (accumulation of the last 5 radar frames)
Each csv file contains a set of points in a specific timestamp:
| Column | Description |
|---|---|
| timestamp | timestamp of the point |
| range | radial distance to the detection (in m) |
| doppler | radial velocity measured for this point (in m/s) |
| azimuth | azimuth angle to the detection (in degree) |
| elevation | elevation angle to the detection (in degree) |
| power | reflected power value of the detection (in dB) |
| x | x value in the XYZ coordinates |
| y | y value in the XYZ coordinates |
| z | z value in the XYZ coordinates |
| comp_height | absolute height of the point (in m) |
| comp_velocity | absolute velocity of the point (in m/s) |
| u | x-axis on the image plane |
| v | y-axis on the image plane |
| label | semantic class id of the object （Refer to Labels） |
| instance | instance id of the object to which this detection belongs |
| Column | Description |
|---|---|
| timestamp | current timestamp |
| latitude | latitude |
| longitude | longitude |
| number_of_satellites | number of satellites |
| altitude | altitude |
| true_north_heading | true north heading to the earth |
| magnetic_north_heading | magnetic north heading to the earth |
| ground_speed_kn | speed (in kn) |
| ground_speed_kph | speed (in km/h) |
| Column | Description |
|---|---|
| timestamp | current timestamp |
| pitch | pitch (x-axis, right) |
| roll | roll (y-axis, front) |
| yaw | yaw (z-axis, top) |
| angular_velocity_x | angular velocity in x-axis |
| angular_velocity_y | angular velocity in y-axis |
| angular_velocity_z | angular velocity in z-axis |
| linear_velocity_x | linear velocity in x-axis |
| linear_velocity_y | linear velocity in y-axis |
| linear_velocity_z | linear velocity in z-axis |
| magnetic_field_x | magnetic field strength in x-axis |
| magnetic_field_y | magnetic field strength in y-axis |
| magnetic_field_z | magnetic field strength in z-axis |
Requirements
- Linux (tested on Ubuntu 18.04/20.04)
- Python 3.6+
-
Clone the repository:
https://github.com/WaterScenes/WaterScenes.git -
Inside the root folder, install the requirements using
pip install -r requirements.txt -
In the same terminal windows typing
jupyter notebookwill start the notebook server.
In case the interactive plots do not show up in the notebooks use: jupyter nbextension install --py --sys-prefix k3d
After fetching both the data and the devkit, please refer to these manuals for several examples of how to use them, including data loading, fetching and applying transformations, and 2D/3D visualization: Tutorial.ipynb
- We referred to the nuScenes, RadarScenes and View of Delft datasets, and are grateful to their websites and github repositories.
The WaterScenes data is published under CC BY-NC-SA 4.0 license, which means that anyone can use this dataset for non-commercial research purposes.
Please use the following citation when referencing
@misc{yao2023waterscenes,
title={WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmark for Autonomous Driving on Water Surfaces},
author={Shanliang Yao and Runwei Guan and Zhaodong Wu and Yi Ni and Zile Huang and Zixian Zhang and Yong Yue and Weiping Ding and Eng Gee Lim and Hyungjoon Seo and Ka Lok Man and Xiaohui Zhu and Yutao Yue},
year={2023},
eprint={2307.06505},
archivePrefix={arXiv},
primaryClass={cs.CV}
}