We introduce a object detection benchmark in challenging adverse weather conditions covering 12000 samples in real world driving scenes and 1500 samples in controlled weather conditions within a fog chamber. The dataset covers different weather conditions as Fog, Snow and Rain and was acquired by over 10,000 km of driving in northern Europe. The driven route with cities along the route are shown below. In total 100k Objekts where labeled with accurate 2D and 3D bounding boxes. Main contributions of this dataset are:
- We provide a proving ground for a broad range of algorithms covering signal enhancement, domain adaptation, object detection or multi-modal sensor fusion focusing on the learning of robust redundancies in between sensor especially if they fail asymmetrically in different weather conditions.
- The dataset was created with the initial intention to showcase methods learning robust redundancies in between sensor enabling raw data sensor fusion in case of asymmetric sensor failure induced through adverse weather effects. In our case we departed from proposal level fusion and applied a adaptive fusion driven by measurement entropy enabling the detection also in case of unknown adverse weather effects. This method outperforms other reference fusion methods, which even drop in below single image methods.
- Furthermore, we compared the method to variety of related methods as domain adaptation, signal enhancement and adverse weather simulation. Please see paper for more information.
Clone the benchmark code.
git clone https://github.com/princeton-computational-imaging/SeeingThroughFog
cd SeeingThroughFog
For running the evaluation and visualization code you need Python with the following packages:
- tensorflow
- numpy
- cv2
- matplotlib
- qt
- pyqtgraph
- pyquaternion
We provide a conda environment to run our code.
conda env create -f environment.yaml
Activate the conda environment.
conda activate LabelTool
Download the SeeingThroughFog dataset from the DENSE dataset webpage. Please download the zip files per sensor and copy all sensorfolders into one common dataset folder.
After unzipping the files, your directory should look like this:
.
|-- SeeingThroughFogData
|-- cam_stereo_left
|-- *.tiff
|-- ...
|-- cam_stereo_left_lut
|-- *.png
|-- ...
|-- lidar_hdl64_last
|-- *.bin
|-- ...
|-- lidar_hdl64_strongest
|-- *.bin
|-- ...
|-- ...
Each folder has following meaning:
Folder | Extrinsic Coordinate System | Necessary for DataViewer | Description |
---|---|---|---|
cam_stereo_left | cam_stereo_left_optical | - | 12-bit RGB images captured from the left stereo camera. |
cam_stereo_left_lut | cam_stereo_left_optical | x | 8-bit RGB left stereo camera images used for annotating the dataset, with custom tone mapping presented in tools/Raw2LUTImages/main.py. |
cam_stereo_right | cam_stereo_right_optical | - | 12-bit RGB images captured from the right stereo camera. |
cam_stereo_right_lut | cam_stereo_right_optical | x | 8-bit RGB right stereo camera images with custom tone mapping. |
cam_stereo_sgm | - | - | Stereo disparity calculated using Stereo Global Matching (SGM). |
cam_stereo_left_raw_history_<d> | cam_stereo_left_optical | - | Temporal History data for <d> in [-5,3] which represents the relativ offset to the annotated samples in cam_stereo_left. |
fir_axis | fir_axis_roof_optical | - | 8-bit FIR reference camera. |
gated<d>_raw | bwv_cam_optical | - | 10-bit NIR gated camera image for slice index <d>. |
gated<d>_rect | bwv_cam_optical | x | 10-bit rectified NIR gated camera image for slice index <d>. |
gated<d>_8bit | bwv_cam_optical | - | 8-bit rectified bitshifted NIR gated camera image for slice index <d> used for dataset annotation. |
gated_full_rect | bwv_cam_optical | x | 10-bit NIR gated camera image with joint slices from a single capture. |
gated_full_rect8 | bwv_cam_optical | - | 8-bit rectified bitshifted NIR gated camera image with joint slices from a single capture. |
gated_full_acc_rect | bwv_cam_optical | - | 10-bit NIR gated camera image with overlayed gated slices from gated<d>_raw. |
gated_full_acc_rect8 | bwv_cam_optical | x | 8-bit rectified bitshifted NIR gated camera image with overlayed gated slices from gated<d>_8bit. |
radar_targets | radar | - | Radar target pointcloud. |
lidar_hdl64_last | lidar_hdl64_s3_roof | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the last echo. |
lidar_hdl64_last_gated | bwv_cam_optical | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the last echo projected into gated camera. |
lidar_hdl64_last_stereo_left | cam_stereo_left_optical | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the last echo projected into left stereo image. |
lidar_hdl64_strongest | lidar_hdl64_s3_roof | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the strongest echo. |
lidar_hdl64_strongest_gated | bwv_cam_optical | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the strongest echo projected into gated camera. |
lidar_hdl64_strongest_stereo_left | cam_stereo_left_optical | x | Lidar pointcloud acquired from the Velodyne HDL64-S3D containing the strongest echo projected into left stereo image. |
lidar_vlp32_last | lidar_vlp32_roof | - | Lidar pointcloud acquired from the Velodyne VLP32 containing the last echo. |
lidar_vlp32_strongest | lidar_vlp32_roof | - | Lidar pointcloud acquired from the Velodyne VLP32 containing the strongest echo. |
velodyne_planes | - | - | Lidar groundplanes estimated from the lidar_hdl64_strongest measurements. |
road_friction | - | x | Road friction measurements. |
weather_station | - | x | Ambient temperature, humidity, dew point, ... |
labeltool_labels | - | x | Meta labels on ambient illumination, drivable patch, scene setting and weather. |
filtered_relevant_can_data | - | x | Vehicle data on wiper state, speed, ... |
gt_labels | - | x | 3D annotation labels following the Kitti annotation layout. cam_left_labels_TMP contains the labels inside the stereo camera left coordinate system and gated_labels_TMP in the gated camera coordinate system. |
The extrinsic coordinate transormations in the third column are given in tools/DatasetViewer/calibs/calib_tf_tree_full.json. It follows the general tansformation logic from ROS.
For an initial reasoning about the dataset the DataViwer in tools/DatasetViewer/DataViewer_V2.py can be used. The needed data is marked in the table above.
To total we have equipped a test vehicle with sensors covering the visible, mm-wave, NIR and FIR band, see Figure below. We measure intensity, depth and weather condition. For more information please refer to our dataset paper.
The final dataset provides about 12000 samples from an extensive data collection campaing with in total 1.4 mio samples. The subsampling process is described in depth within the supplemental material. Which can be found here. We provide 2d and 3d annotations for 4 main classes PassengerCars, Pedestrians, RidableVehicles, LargeVehicles. If PassengerCars, RidableVehicles and LargeVehicles can not be differentiated the Objects are Labeled within the fallback class Vehicle. In case Vehicle and Pedestrians can not be differentiated Objects are labeled as Obstacles. Below you can find the illustrated LabelDefinition:
The Annotation format follows the well known Kitti Format with added coloums for 3d Box Rotation angles. You can find the loading functionalities in
./tools/DatasetViewer/lib/read.py
Recomended data splits can be found in
./splits
Tools can be found in
"./SeeingThroughFog/tools/".
The tools help to viszualize the dataset (DatasetViewer), create a TFRecords dataset (CreateTFRecords), create simple fog simulations for lidar and rgb data (DatasetFoggification) and to quickly reason about the dataset statistics (DatasetStatisticsTool).
If you find our work on object detection in adverse weather useful in your research, please consider citing our paper:
@InProceedings{Bijelic_2020_STF,
author={Bijelic, Mario and Gruber, Tobias and Mannan, Fahim and Kraus, Florian and Ritter, Werner and Dietmayer, Klaus and Heide, Felix},
title={Seeing Through Fog Without Seeing Fog:
Deep Multimodal Sensor Fusion in Unseen Adverse Weather},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
These works have received funding from the European Union under the H2020 ECSEL Programme as part of the DENSE project, contract number 692449.
Feedback? Questions? Any problems/errors? Do not hesitate to contact us!