Skip to content

Novicevillageplayer/WaterScenes

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

32 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

A Multi-Task 4D Radar-Camera Fusion Dataset for Autonomous Driving on Water Surfaces

Overview

Changelog

Dataset

Introduction

  • WaterScenes, the first multi-task 4D radar-camera fusion dataset on water surfaces, which offers data from multiple sensors, including a 4D radar, monocular camera, GPS, and IMU. It can be applied in multiple tasks, such as object detection, instance segmentation, semantic segmentation, free-space segmentation, and waterline segmentation.
  • Our dataset covers diverse time conditions (daytime, nightfall, night), lighting conditions (normal, dim, strong), weather conditions (sunny, overcast, rainy, snowy) and waterway conditions (river, lake, canal, moat). An information list is also offered for retrieving specific data for experiments under different conditions.
  • We provide 2D box-level and pixel-level annotations for camera images, and 3D point-level annotations for radar point clouds. We also offer precise timestamps for the synchronization of different sensors, as well as intrinsic and extrinsic parameters.
  • We provide a toolkit for radar point clouds that includes: pre-processing, labeling, projection and visualization, assisting researchers in processing and analyzing our dataset.

USV Setup

Data Collection

Dataset Statistics

The dataset includes 54,120 sets of RGB images, radar point clouds, GPS and IMU data, covering over 200,000 objects.

Tasks Object Detection / Instance Segmentation / Semantic Segmentation Free-Space Segmentation Waterline Segmentation Adverse Conditions
Classes Total Pier Buoy Sailor Ship Boat Vessel Kayak Free-Space Waterline Lighting Weather
Frames 54,120 25,787 3,769 3,613 19,776 9,106 9,362 366 54,057 53,926 5,604 10,729
Objects 202,807 121,827 (60.07%) 16,538 (8.15%) 8,036 (3.96%) 34,121 (16.82%) 10,819 (5.33%) 11,092 (5.47%) 374 (0.18%) 54,057 159,901 30,517 (15.05%) 46,784 (23.07%)

Dataset Structure

WaterScenes (root)
  - image # RGB images
    - 000001.jpg
  - radar # radar files
    - 000001.csv
  - radar_3_frames # 3 frames before and after current timestamp
    - 000001.csv
  - radar_5_frames # 5 frames before and after current timestamp
    - 000001.csv
  - calib # intrisic and extrisic parameters
    - 000001.txt
  - gps # gps file
    - 000001.csv
  - imu # imu file
    - 000001.csv
  - detection # annotation files for object detection task
    - yolo 
      - 000001.txt
    - xml 
      - 000001.xml
  - instance # annotation files for instance segmentation task
    - yolo 
      - 000001.txt
    - labelme 
      - 000001.json
    - SegmentationClassPNG 
      - 000001.png
    - class_names.txt
  - semantic # annotation files for semantic segmentation task
    - SegmentationClass 
      - 000001.png
    - label_mapping.txt
  - free_space # annotation files for free-space segmentation task
    - SegmentationClassPNG 
      - 000001.png
    - class_names.txt
  - waterline # annotation files for waterline segmentation task
    - SegmentationClass 
      - 000001.png

Labels

Code Label Note
-1 no-object No object. Only for radar point clouds
0 pier Static object
1 buoy Static object
2 sailor Person on the surface vehicles
3 ship
4 boat
5 vessel
6 kayak
- free-space Only for instance/semantic/free-space segmentation
- waterline Only for waterline segmentation

Radar Format

Radar point clouds are stored in csv files.

We provide radar point clouds in three flavors:

  • radar (single frame of the current timestamp)
  • radar_3_frames (accumulation of the last 3 radar frames)
  • radar_5_frames (accumulation of the last 5 radar frames)

Each csv file contains a set of points in a specific timestamp:

Column Description
timestamp timestamp of the point
range radial distance to the detection (in m)
doppler radial velocity measured for this point (in m/s)
azimuth azimuth angle to the detection (in degree)
elevation elevation angle to the detection (in degree)
power reflected power value of the detection (in dB)
x x value in the XYZ coordinates
y y value in the XYZ coordinates
z z value in the XYZ coordinates
comp_height absolute height of the point (in m)
comp_velocity absolute velocity of the point (in m/s)
u x-axis on the image plane
v y-axis on the image plane
label semantic class id of the object (Refer to Labels
instance instance id of the object to which this detection belongs

GPS Format

Column Description
timestamp current timestamp
latitude latitude
longitude longitude
number_of_satellites number of satellites
altitude altitude
true_north_heading true north heading to the earth
magnetic_north_heading magnetic north heading to the earth
ground_speed_kn speed (in kn)
ground_speed_kph speed (in km/h)

IMU Format

Column Description
timestamp current timestamp
pitch pitch (x-axis, right)
roll roll (y-axis, front)
yaw yaw (z-axis, top)
angular_velocity_x angular velocity in x-axis
angular_velocity_y angular velocity in y-axis
angular_velocity_z angular velocity in z-axis
linear_velocity_x linear velocity in x-axis
linear_velocity_y linear velocity in y-axis
linear_velocity_z linear velocity in z-axis
magnetic_field_x magnetic field strength in x-axis
magnetic_field_y magnetic field strength in y-axis
magnetic_field_z magnetic field strength in z-axis

Devkit

Installation

Requirements

  • Linux (tested on Ubuntu 18.04/20.04)
  • Python 3.6+
  1. Clone the repository: https://github.com/WaterScenes/WaterScenes.git

  2. Inside the root folder, install the requirements using pip install -r requirements.txt

  3. In the same terminal windows typing jupyter notebook will start the notebook server.

In case the interactive plots do not show up in the notebooks use: jupyter nbextension install --py --sys-prefix k3d

Example

After fetching both the data and the devkit, please refer to these manuals for several examples of how to use them, including data loading, fetching and applying transformations, and 2D/3D visualization: Tutorial.ipynb

Acknowledgement

Citation

The WaterScenes data is published under CC BY-NC-SA 4.0 license, which means that anyone can use this dataset for non-commercial research purposes.

Please use the following citation when referencing

@misc{yao2023waterscenes,
      title={WaterScenes: A Multi-Task 4D Radar-Camera Fusion Dataset and Benchmark for Autonomous Driving on Water Surfaces}, 
      author={Shanliang Yao and Runwei Guan and Zhaodong Wu and Yi Ni and Zile Huang and Zixian Zhang and Yong Yue and Weiping Ding and Eng Gee Lim and Hyungjoon Seo and Ka Lok Man and Xiaohui Zhu and Yutao Yue},
      year={2023},
      eprint={2307.06505},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}

About

Official repository for WaterScenes dataset

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 77.5%
  • Jupyter Notebook 22.5%