Add Structural Similarity Loss/Metric #172
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Differentiable structural similarity index that works on Chainer! Repository at https://github.com/higumachan/ssim-chainer.
weiji14
added
enhancement ✨
DeepCode Report (#d24094)DeepCode analyzed this pull request. |
Incorporating a Structural Similarity (SSIM) Index based loss function into out adapted ESRGAN's Generator Network's loss function. Currently set with a weighting of 1e-2 that matches the L1 content loss weighting. Properly creating a unit-tested function that wraps around the differentiable [SSIM](https://github.com/higumachan/ssim-chainer) chainer module, in case things change down the line. Also flipped the y_true/y_pred kwarg positioning in psnr() to ease my OCD, and correctly renamed d_train_loss to d_dev_loss (not major).
For better structural reconstruction of our DEM, we revise our SSIM Loss weighting from 1e-2 up to 5.25e-2. Based on [Zhao et al. 2017](https://doi.org/10.1109/TCI.2016.2644865)'s paper which empirically weighted MS-SSIM loss at 0.84 and L1 loss at 0.16 (1-0.84) which is therefore 5.25x. Yes, it's only an empirical setting, but too lazy to tune those weightings (though someone probably should in the future). Current best SSIM score we get is ~0.25 which is a ways off from perfect at 1.00, so setting a higher structural weighting should encourage our model to produce more structurally similar images to the groundtruth. Even though an RMSE_test of 1655.87 isn't so great, nor is the actual SSIM score of 0.1885, qualitative 3D evaluation of the result at https://www.comet.ml/weiji14/deepbedmap/88b073324a644fd695aecf47109dd2bc does show a pretty nice terrain. Tempted to use SSIM as 'the' tuning metric instead of RMSE_test now, but we'll see.
weiji14
added a commit
that referenced
this pull request
Sep 10, 2019
Closes #172 Add Structural Similarity Loss/Metric.
weiji14
added a commit
that referenced
this pull request
Sep 13, 2019
Adjust our Content, Adversarial, Topographic and Structural Loss weightings to be on a more equal footing, with priority towards better SSIM scores (patching #172). Content and Topographic Losses (~35) was overpowering the Adversarial and Structural Losses (~0.1-10) by an order of magnitude (i.e. wasn't really any adversarial impact/structural improvement)! Loss weighting changes are as follows: Loss: Content Adversarial Topographic Structural Old 1e-2 5e-3 5e-3 5.25e-3 New 1e-2 2e-2 2e-3 5.25e-0 This is all to do with our domain specific (DEM generation) task. Ideally we would scale our images to lie in the range of 0-1 like those out in the computer vision world, which can be easily done by converting metres to kilometres (divide by 1000). Workaround instead is to scale down the content and topographic losses relative to the adversarial and structural loss. Also because we are recording too much metric information that results in I/O errors when writing to the sqlite database, I've made code changes so that our 2 Tesla V100 GPUs and 2 Tesla P100 GPUs write their training results to separate databases named based on the hostname. Best tuned score had RMSE_test of 215.60 and SSIM of 0.6195 at https://www.comet.ml/weiji14/deepbedmap/699ecfa6f14448c09cf0e450edf64f30. The results aren't so good when ran on the whole Pine Island Glacier area (2007tx, 2010tr, istarxx), but they're not as bad compared to the others, and you can really see the realistic DEM textures now!
10 tasks
weiji14
added a commit
that referenced
this pull request
Apr 15, 2020
Lots of edits to bring the paper up to speed on what our DeepBedMap v0.9.4 model's training details, architecture and perceptual loss function looks like. Updated training details to reference quad GPU training and new hyperparameters. Mention how our Generator model architecture now uses Deformable Convolution in the last 2 layers (#171), and remove mention that we used a residual scaling factor of 0.1 and PixelShuffle upsampling layers (#160). Added proper equations on topographic loss (added in d599ee8) and structural loss (#172) functions we've introduced, with some tweaks to the Loss Functions subsection as well. Also made sure to cite Chainer and Optuna properly using their KDD19 paper, plus some others of course.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Discovered this ssim-chainer repository that does Differentiable Structural Similarity (SSIM) index which works on Chainer! Workaround until chainer/chainer#2503 is properly solved.
Adding this to our adapted ESRGAN's generator loss function. So the total
Generator Lossfunction (which we minimize) is nowContent Loss+Adversarial Loss+Topographic Loss+Structural Lossthat abbreviates nicely toCATS!Code Implementations:
References:
TODO:
1-SSIM) in our generator loss (8fe9e03)