Multisource Data Reconstruction-based Deep Unsupervised Hashing for Unisource Remote Sensing Image Retrieval

Abstract

Unsupervised hashing for remote sensing (RS) image retrieval first extracts image features and then use these features to construct supervised information (e.g., pseudo-labels) to train hashing networks. Existing methods usually regard RS images as natural images to extract unisource features. However, these features only contain partial information about ground objects and cannot produce reliable pseudo-labels. In addition, existing methods only generate a pseudo single-label to annotate each RS image, which cannot accurately represent multiple scenes in a RS image. To address these drawbacks, this paper proposes a new Multisource data reconstruction-based deep unsupervised hashing method, called MrHash, which explores the characteristics of RS images to construct reliable pseudo-labels. In particular, we first use geographic coordinates to obtain different satellite images and develop a novel autoencoder network to extract multisource features from these images. Then pseudo multi-labels are designed to deal with the coexistence of multiple scenes in a single image. These labels are generated by a custom probability function with extracted multisource features. Finally, we propose a novel multi-semantic hash loss by using the Kullback–Leibler (KL) divergence to preserve the semantic similarity of these pseudo multi-labels in Hamming space. Our newly developed MrHash only uses multisource images to construct supervised information, and hash code generation still relies on a unisource input image. Experiments on benchmark datasets clearly show the superiority of the proposed method over state-of-the-art baselines.

Dataset

Update (2023-04-11): The train.txt, test.txt, and database.txt of DSRSID and the extracted images from the original Matlab file. Goolge Drive

Update (2023-04-04): the extracted features (e.g., all_features_resnet50_without_l2_msi.pickle) by a pretrained ResNet50 can be obtained from Google Drive. Note: the shared features are optional and you can extract your features using existing pretrained models.

We conduct experiments on two public remote sensing image retrieval datasets.

• EuroSAT is a geo-referenced benchmark dataset. The dataset can be accessed from https://github.com/phelber/eurosat. In addition, we use VHR images as complementary multisource images collected from Google Earth imagery and ArcGIS maps. These images can be obtained from Google Drive.

• DSRSID is a public dataset for remote sensing image retrieval, which contains multispectral images and panchromatic images. In our experiments, we treat the panchromatic images as complementary multisource images for our method. The dataset can be accessed from https://pan.baidu.com/s/15ZWaZ2yArnvwcwtead_rpQ.

Train

1. Firstly, train myautoencoder to get a pretrained myautoencoder model.

cd myautoencoder; python autoencoder.py

2. Secondly, train MrHash with the pretrained myautoencoder.

python train.py --hash_bit 32 --pretrained_dp_path your_trained_ae_path

Test

python test.py --hash_bit 32 --pretrain_path your_trained_mrhash

Citation

@ARTICLE{sun10001754,
  title={Multisource Data Reconstruction-based Deep Unsupervised Hashing for Unisource Remote Sensing Image Retrieval}, 
  author={Yuxi Sun, Yunming Ye, Jian Kang, Ruben Fernandez-Beltran, Yifang Ban, Xutao Li, Bowen Zhang, and Antonio Plaza},
  journal={IEEE Transactions on Geoscience and Remote Sensing}, 
  year={2022}
  doi={10.1109/TGRS.2022.3231215}
}