Skip to content

px39n/Awesome-Data-Fusion-for-Remote-Sensing

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

14 Commits
 
 
 
 

Repository files navigation

Awesome Data Fusion Awesome

List of reference,algorithms, applications in RS data fusions (contribution are welcome) drawing

Table of Contents

Overview of Data Fusion

Trends

There are popular topics and review literatures in different periods.

1992~2000: Data fusion, spatial resolution

  • Data Fusion Subpanel of the Joint Directors of Laboratories (JDL. 1991)

  • Fusion of satellite images of different spatial resolutions(Linas et al.1997)

  • Multisensor Image Fusion in Remote Sensing: Concepts, Methods and Applications(Pohl et al.1997)

  • Some terms of reference in data fusion(Wald,1999)

2001~2010: classification, unsupervision, change detection, model, multi-resolution, quality, feature extraction Wavelet transform

  • #Handbook of Multisensor Data Fusion(Hall et al.2001)
  • Mathematical Techniques in Multisensor Data Fusion (Hall et al. 2004)
  • Multi-Sensor Data Fusion: An Introduction (Mitchell 2007)
  • Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics(Thomas et al.2008)
  • Decision Fusion for the Classification of Hyperspectral Data: Outcome of the 2008 GRS-S Data Fusion Contest (Licciardi et al.2009)
  • Advances in Multi-Sensor Data Fusion: Algorithms and Applications (Dong et al.2009)
  • Multi-source remote sensing data fusion: status and trends (Zhang et al.2010)

2011~2021:hyperspectum, deep learning, Heterogeneous, fusion framework, sparse expressions.

Basic Concept

DOD: Data fusion is a multilevel, multifaceted process dealing with the automatic detection, association, correlation, estimation, and combination of data and information from multiple source. Community: Data fusion is a formal framework in which are expressed means and tools for the alliance of data originating from different sources. It aims at obtaining information of greater quality; the exact definition of ‘greater quality’ will depend upon the application

Sensors

Satellite Sensor Spatial Resolution Band Revisit Cycle
NOAA AVHRR 1.1km VIS,NIR,TIR 12h
Terra/Aqua MODIS 250m,500m,1000m 36bands 1d
Terra ASTER 15m,30m,90m 14bands(VIS-TIR) 16d
Lansat MSS 79m VNIR 18d
Terra MSS+TM 30m,120m VNIR,TIR 16d
Terra ETM+ 30m,60m VNIR,TIR 16d
Terra OLI 30m,100m VNIR,TIR 16d
OrbView-2 Sealifts 1km VIR,NIR,pan 1d
SPOT HRV 20m 3VNIR 26d
SPOT VGT 1.15km 3VNIR+SWIR 26d
SPOT HRG/HRS/VGT 10m 26d
ENVISAT MERIS 300m 15(390-1040nm) 35d
Sentinel-2 MSI 10m,20m,60m VNIR, SWIR 5days
Sentinel-1 SAR >5m C band 12
Sentinel-3 SLSTR,OLCI,SRAL,DORIS ~300M optical, micro, 1d
HJ-1A/1B UPDATING 30m 31d
TH-1 UPDATING 10m 58d
BJ-1 UPDATING 32m
CBERS-01/02 UPDATING 20m-150m 26d
ZY-1 02B UPDATING 20m 26d
ZY-2 02C UPDATING 10m 55d
SJ-9A UPDATING 10m 69d
IRS-P3 WIFS/MOS 188m 5d
updating

Focus, Taxonomy

Research Focus

The application of data fusion in remote sensing is mainly divided in two scenarios:

1. Resolution enhancement:

It aims to provide higher resolution by combining multi modal data. The results are more like to be transition data, base map, or continuous time series for applications need high temporal and spatial resolution.

They are mostly:

  • Based on pixel level.
  • Focus on Robustness, temporal continuous

For examples:

Sub Topic Enhancement Applications
Super Resolution Spatial Transition for registration, base map
pan-sharpening Spatial base map
spatio-temporal fusion Temporal and Spatial base map, time series

Typical applications: Plant detection, weather detection, ecology, change detection, land cover, etc.

2. Feature/object detection:

It aims to improve accuracy and precision of specific task, based on complementary property of heterogeneous information in multimodality. Often, temporal continuity can be sacrificed.

They are mostly:

  • End to End
  • based on pixel level or decision level.
  • Focus on accuracy of task (instead of fused images)

For examples:

Sub Topic Enhancement Application
Change detection Accuracy of Change map Land cover, Building..
Object detection Accuracy of Detection Car, Building
Segmentation Accuracy of Classification Land Cover, Forest

Typical Applications: change detection, land change coverage, land classification, disaster monitoring, building recognition, vehicle recognition, etc.

3. Data Alignment.

It aims to find and adjust the unlinear connections between different sensors, It could be divided into matching and co-registration, Furthermore, the problems need to be resolved could be regarded as topological and radiation issues. They are mostly:

  • Necessary for all multi-modal applications
  • part of preprocess
  • can be selectively ignored or modified(such as adaptive registration)

For examples:

Sub Topic Description
Radiometric Correction Atmospheric effect. etc
Geometric Correction Camera, Solar Angle etc
Band Adjustment Function based on prior knowledge
BRDF Adjustment Function based on vegetation
Registration of Different Modal SAR-Optical, Multi-Temporal..
Adaptive Registration Domain, Manifold, Attention Mechanics, Tensor

Taxonomy

Depending on the main problem solved by the model/algorithm, the development of data fusion can be divided into many areas

  • According to process levels: Pixel, Attribute, Decision
  • According to Principle: Weighted function based, Decomposition based, Learning based (including sparse dictionary, DL), Bayesian based, Hybrid based
  • According to modal: Homogeneous fusion, Heterogeneous fusion (NIR,-VIS, Optical-SAR, Optical-Thermal, hyper-Multi etc.,), Remote sensing site fusion, Remote sensing non-observation fusion (Data assimilation)
  • According to mechanisms: Competitive integration, Complementary integration(like temporal and spatial), Cooperative integration(like 3D reconstruction)
  • According to Dimension: Spatial Dimension Enhancement, Spectral Dimension Enhancement, Time dimension enhancement, End to End.
  • According to Applications: spatial resolution enhancement, Matching and co-registration of multisource data, Change detection, Object recognition, Agriculture, Ecology etc.,
  • According to procedure: Matching, Co-registration, Process, Quality Assessment.

Current Challengings

In almost every small direction (as shown in the previous subsection). There are a number of issues so authors only list some most important challenges to demonstrate.

As mentioned before, spatial enhancement and fusion of complementary information are main scientific focus. Apart from these, there are:

1. Issues Related to Radiation

Issues Popular Solutions Description
Atmospheric effect Radiometric Correction Commonsense
Solar azimuth and elevation Radiometric Correction Commonsense
Band pass Adjustment linear regression The small differences between MSI and OLI equivalent spectral bands need to be adjusted.
Bidirectional Reflectance Distribution effect BRD Function (BRDF) The BRDF is needed in remote sensing for the correction of view and illumination angle effects (for example in image standardization and mosaicking)

2. Issues related to Topology

Issues Popular Solutions Description
Lens Distortion (Camera Calibration in) Geometric Correction Commonsense
Error caused by Elevation(SAR and optical) Co-Registration Please see figure1 and figure2
Mis-matching in pixel Co-Registration in Multi-Temporal, Multi-Sensor, Multi-Angle
Mis-matching in Feature Co-Registration , Domain Adaptation adaptive alignment as part of model
Mis-matching in Object Co-Registration , Attention Mechanics adaptive alignment as part of model

Figure1. Error due to deviation in DEM, Relationship between error and Angle

enter image description here

3. Issues Related to Data

Noise:

Issues Popular Solutions Description
Noise in SAR (Camera Calibration in) Geometric Correction Speckle
Cloud mask, super resolution, reconstruction, interpolation

Sample:

Issues Popular Solutions Description
Spectral/Index Colinearity, Similarity PCA, Correlation Analysis Huges Phenomenon, Statistics
Lack of Samples Data Augmentation, Semi-supervised, GANs Overfitting, low generalization
Unbalanced Samples Loss functions, updating... weak train in small class
Gap in Resolution Transition(Super resolution) For 1:2 or higher resolution ratio in multi modal, It will increase the difficulty in data fusion

4. Issues Related to Method and Process

Issues Popular Solutions Description
Computation Speed Alternative method, cloud service, Parallel computing
Issues related to training AI tools like overfitting, vanished gradient etc.,
Hard to fuse heterogeneity Pixel(like unmixing), feature(like feature layers), decision level(like DT) multi-modal
low Generalization Data augmentation, Domain Adaptive, Pre-train transferability
lack of Interpretability Combination with prior knowledge(Branch, Attention, feature layers, data assimilation ) Physics

5. Issues related to Physics

Issues Popular Solutions Description
Intra-class Variation (updating) Small difference between different class
Inter-class variation (updating) Large difference in same class
Landscape Heterogeneity Learning based method,(updating) Variations in high resolution pixel.
Change due to Landcover Learning based method,(updating)
Abrupt Change Function related to time(updating) disaster etc
Seasonal Change Function related to time(updating) vegetation

Algorithms

There are currently more than 200 spatio-temporal models, so only part of baseline models or popular papers are included.

1.Spatial Resolution Enhancement Algorithms

Multisensor image fusion for spatial resolution enhancement such as pan-sharpening, multi/hyperspectral image fusion, and downscaling of multiresolution imagery

Purpose Principle Method Paper Code Features
Spatial Component (+) Principle Component Analysis(PCA) Shettigara et al.1992 Updating
Spatial + Intensy-Hue-Saturation(IHS) Tu et al.2001 Updating
Spatial + Brovey Transform(BT) Tu et al.2005 Updating
Spatial + Gram-Smidt(GS) Aiazzi et al.2007 Updating
Spatial + GS adaptive(GSA) Aiazzi et al.2007 Updating
Spatial + GIHS adaptive(GIHSA) Aiazzi et al.2007 Updating
Spatial Unmaxing MMT (Zhukov et al.1999) Updating
Spatial + MMT(MERIS,Lansat) (Milla et al.2008) Code constraints, Positive of End Member
Spatial + LAC-GAC NDVI (MAselli, 2011) Code
Spatial Baysian BME (Li et al.2013) Code
Spatial Hybrid Updating
Spectral Linear Wavelet Transform Nunez et al.1999 Updating
Spectral + High-pass filtering Chavez et al.1991 Updating
Spectral + Curvelet Transform Nencini et al.2007 Updating
Spectral + Contour Transform do and Vetterli 2005 Updating
Spectral + Laplacian Pyramid Schmitt and Zhu, 2005 Updating
Spectral + Smoothing Filter-based intensity modulation Liu,2000 Updating
Spectral Unmixing Spectral Unmixing Bendoumi et al.2014
Spectral + Nonnegative Matrix Unmixing Huang et al.2008 Updating
Spectral + Coupled Nonnegative Matrix Unmaxing Yokoya et al.2012 Updating
Spectral Bayesian Maximum a posteriori Hardie et al.2004 Updating
Spectral Learning Sparse Representation
Spectral + Analysis Sparse Model
Spectral + MRA DNN Azarang et al.2017 Code
Spectral + PNN Li et al.2012 Updating
Spectral + DRPNN Wei et al.2017 Matlab Residual Network
Spectral + MSDCNN Zhou et al.2019 Python
Spectral Hybrid

2. Spatiotemporal Fusion by Traditional Methods

(Normally Prediction by fusing two high temporal and high spatial resolution sensors with correction,Multisensor and multimodal data fusion using a variety of sensors such as optical imaging, SAR, and LiDAR)

Sensor Principle Method Paper Code Features Registration
Landsat&MODIS Weighted Function STARFM Gao et al.2006) Python
+ + STAARCH (Hilker et al.2009) Code Cloud and Snow Cover
+ + ESTARFM (Zhu et al.2010) IDL,Python Enhancement in Heterogeneous region
MODIS&landsat + RWSTFM (wang et al.2017) Code kriging Based
1MODIS,2landsat + Prediction Smooth Method (Zhong et al.2018) Code abrupt change, phenology Manual
MODIS&landsat + SADFAT Weng et al.2014 Code Consider Annual temperature Cycle
Unlimited + STITFM Wu et al.2014 Code Multi-Sensor LST
+ STVIFM Liao et al.2017 Code growth stages
Unmixing ESTDFM (Zhang et al.2013)
+ MSTDFA Wu et al.2015 Code sensor adjustment by linear model
+ OB-STVIUM (Lu et al.2016) Code Consideration of Phenology Change
Temporal Bayesian NDVI-BSFM
Learning SPSTFM (Huang et al.2012) Code sparse representation
+ One-pair image learning method (Song et al.2012) Code sparse representation
Landsat&AHVRR + EBSCDL Wu et al.2015
Landsat&MODIS + ELM (Liu et al.2016 ) Code Matlab
Landsat&MODIS + CSSF (Wei et al.2017) Code Compression downsampling
Landsat&MODIS + WAIFA Moosavi et al.2015 wavelet&ANN
2Landsat&1MODIS + STFDCNN Song et al.2018 Code Transition Image
+ DCSFTN Code
Hybrid FSDAF (Zhu et al.2016) Abrupt Change
+ NDVI-LMGM (yu et al.2015) linear growth&unmixing
Two Time series Others STAIR (Luo et al.2015) Difference, Cloud
Multi Time series Others STAIR2 (Luo et al.2020)

3. Spatio-Temporal Fusion by Neural Network

Registration Modal Process Level Method Paper Code Features
Traditional A(T1,T3), B(T2) pixel StfNet(Two Branch) (Liu,2019) Code Features
Traditional A1-3,B1,3 pixel Super Resolution, weighted function (SONG, 2019) Code Features
Traditional A1-3,B1,3 pixelLevel Super Resolution, weighted function Li,2019 Code Features
Traditional A12,B1 PIXEL DMnet, concatenate lI 2020 Code Features
Traditional A12,B1 PIXEL CNN, concatenate [Yin 2020]( Traditional A1,B1

4. SAR to Optical

Task Source Method Paper Code Features
Pan-Sharpening Low&High CNN jin et al.2016 Code Resolution enhancedment of MSS
Pan-Sharpening Low&High CNN G Masi et al.2016 Code End-to-End
SAR 2 Optical SAR GANs Reyes et al.2018 Code Feature Level, two Stream, semi-auto- Label
SAR 2 Optical(Cloud removal) SAR DRN Meraner et al.2018 Code cloud removal
SAR 2 Optical SAR sar2opt Toriya et al.2019 Code
Growth Prediction Multi CNN Scarpa et al.2018 Code pixel level, Sentinel,NDVI fusion and pixel fusion
Growth Prediction SAR&VNIR Random Forest hECKEL et al.2020 Code pixel level, Sentinel

5.Registration

Learning Modal Process Level Method Paper Code Features
Registration and SR Low&High Deep Neural Network Y. Qu et al.2018 Code
Registration SAR&Optical Deep Neural Network Mou 2017 et al.2018 Patch-based
Recognition Hyper&SAR Dual DCNN Lagrange et al.2018 Code Feature Level, two Stream
Recognition Multi&SAR Multi-TaskUNet JIAN et al.2019 Code Multi-Task(Edge, Biniary),xception
Registration Optical&SAR Siamese Mou et al.2018 Code Feature Level, two Stream, semi-auto- Label
Registration Optical&SAR 3 DNN Hughes et al.2020 Code hot map, goodness
Registration Optical&SAR Siamese& Gaussian pyramid coupling quadtree He et al.2018 Code Coarse-finer
Registration Optical&SAR Pseudo-Siamese CNN Hughes et al.2018 Code
Supervised Optical&SAR Siamese CNN Merk et al.2017 Code

6. Super Resolution Enhancement

Registration Modal Process Level Method Paper Code Features
CNN Yuan et al.2016 Code End-to-End,Transfer Model from CV, hyperspectrum
Low&High Deep Residual Convolutional Neural Network Wang et al 2017 Code End-to-End,Transfer Model from CV, hyperspectrum
Low&High SSF-CNN X. Han et al 2018 Code End-to-End,Transfer Model from CV, hyperspectrum
Low&High Deep Neural Network R. Dian et al.2017 Code
Low&High Sparse Dirichlet-Net Y. Qu et al.2018 Code
Low&High Deep Neural Network Y. Qu et al.2018 Code
Low&High Deep Neural Network Y. Qu et al.2020 Code Chen
Mannual HSI-MSI patch-based,concatenation Super Resolution Low&High Deep Neural Network Yang et al.2018

7. Classification

Registration Modal Process Level Method Paper Code Features
Classification Multi CNN Lagrange et al.2018 Code Comparision of existing
Classification Multi FusioNet Hu et al.2017 Code Feature Level, two Stream
Multi DeepNetsForEO Audebert et al.2017 # Semantic Segmentation of Earth Observation Data Using Multimodal and) Code Feature Level, two Stream
Classification VNIR-DSM DeepUNet Audebert et al.2017 Code Channel Packing
Manual Sentinel2,Lansat8, OSM,etc,. Pixel(Concatenation) ResNet Qiu et al.2018 LCZ maps

8.Change Detction

Modal Method Paper Code Features
SAR1 (T1, T2) Ratioing/log Ratioing Papers
SAR1 (T1, T2) Small wavelet transform (Bovolo,2005) Unsupervised
SAR1 (T1, T2),SAR2 (T1, T2) Markov (Solarna,2018) Unsupervised
More Change detection research please refer to Awesome Change Detection

Quality Assessment

1.Assessment Index With Reference Images.

Index Description Dimension Reference
Spectral angle updating spectral Updating
General image quality index updating spectral Updating
Root Mean Square Error updating spectral&temporal Updating
Relative mean spectral error updating spectral&temporal
Signal-to-noise ratio updating spatial Updating
Peak signal-to-noise ratio updating spatial Updating
Correlation coefficient updating spatial&temporal Updating
Structural similarity coefficient spatial&temporal Updating
Global integrated error index updating spatial&spectral Updating
Average error updating spatial Updating

2.Assessment Index Without Reference Images.

Index Description Dimension Reference
Average value updating spatial,spectral
Variance updating spatial,temporal Updating
Standard deviation updating spatial,temporal Updating
Information entropy updating spatial,temporal Updating
Mean gradient updating spatial,temporal Updating

Community

IEEE GRSS data fusion contest(Link)

Year: 2020 Title: Global Land Cover Mapping with Weak Supervision Data: MSS, SENTINEL MSS

Track1:

Main Author Approach Code
Robinson A combination of iterative clustering and epitome representations Code
Yu Xia Multi-branch fusion of unsupervised multi-resolution segmentation, random forest classification of remote sensing indexes, and convolutional neural network predictions with post-processing based on expert priors WHU_YuXia
Daniele Cerra Automated label pre-processing, a Gaussian Naive Bayes classifier trained on cluster centroids, and classes obtained by k-means clustering and random forests with bag of words features, followed by classification refinement designed for specific classes Pineapples
Track 2:
Main Author Approach Code
-- -- --
Huijun Chen An ensemble of random forests trained on refined labels Antonia
Daniele Cerra As Track 1 third, but random forests trained on high-resolution labels for validation data Pineapples
Shuting Yin A combination of random forests, k-means, and DeepLabv3++ with postprocessing and retraining dfchen

Year: 2019 Title: Reconstruct both a 3D geometric model and a segmentation of semantic classes for an urban scene Data: WorldView-3, VIR, NIR, LiDAR

Track 1: Single-view semantic 3D

Main Author Approach Code
Saket Kunwar An ensemble of random forests trained on refined labels nest
Zhuo Zheng A pyramid on pyramid network based on an encoder-dual decoder framework RSIDEA-WHU
Track 2: Pairwise semantic stereo
Main Author Approach Code
-- -- --
Hongyu Chen A modified version of Pyramid Stereo Matching Network (PSMNet) and Disparity Fusion Segmentation Net (DFSN) BurningAllthing
Rongjun Qin U-Net and Pyramid Stereo Matching Network (PSMNet) qin.324

Track 3: Multi-view semantic stereo

Main Author Approach Code
Pablo d’Angelo Semi-global matching and an ensemble of CNN classifiers with ad hoc detectors Panoptes
Rongjun Qin Semi-global matching and U-Net qin.324
Track 4: 3D point cloud classification
Main Author Approach Code
-- -- --
Lian Yanchao An ensemble of random forests trained on refined labels nest
Jia Meixia Attention-SIFT Net (AS net) based on Pointnet++ and PointSIFT aijinli0613

Year: 2018 Title: urban land use and land cover classification Data: Hyper, Multi, Lidar, RGB(5cm) updating

Main Author Approach Code
Yonghao Xu Fully convolutional networks and post-classification with topological relationships among different objects Gaussian
Daniele Cerra Deep convolutional and shallow neural networks on a simplified set of classes, completed by a series of specific detectors and ad hoc classifiers dlrpba
Sergey Sukhanov Ensemble learning based on several classifiers, including convolutional neural networks, gradient boosting machines, and random forests, followed by post-processing techniques AGTDA

2017,2016,2015 are updating!

Discussion

Software and Open Source Tools

We will focus on cloud computing and some important machine learning libraries

Cloud Services

  • Google Earth Engine
  • AWS
  • Updating

Useful Library

  • Updating

Database

TERMS

There are terms which are slightly different from those in other areas.

Measurements(SIGNAL/image): Primarily the outputs of a sensor,represent the raw information, normally in format of singal, images. The elementary support of the measurement is a pixel in the case of an image, and is called a sample in the general case

Object: It is defined by its properties, e.g., its color, its materials, its shapes, its neighborhood, etc. It can be a field, a building, the edge of a road, a cloud, an oceanic eddy, etc.

attribute(Feature): It is a property of an object. Mathematical attribute: aggregation of measurements made for each of the elements of the object Modality: It refers to the raw input used by the sensors.

Spatial context of a pixel, computed by local variance, or structure function or any spatial operator. This operation can be extended to time context in the case of time-series of measurements. Equivalent terms are local variability, local fluctuations, spatial or time texture, or pattern.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published