Bump REMA surface DEM from v1.0 to v1.1, 200m to 100m resolution #64
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Increasing spatial resolution of our Antarctic surface digital elevation model! There is finally a 100m mosaic product for the whole continent. Note that the dataset doi (currently linking to Harvard Dataverse) is still at v1.0. From the 'release notes' at https://www.pgc.umn.edu/data/rema/: REMA Release 1.1 corrects 2 minor issues in the mosaic tiles: (1) A vertical offset of 39 cm was added to the 1086 8-m mosaic tiles registered to Cryosat-2 altimetry, as well as Antarctica-wide mosaics, to correct for biases induced by surface penetration and surface re-tracking. This correction was determined through comparison to ICESat-1 altimetry. (2) Incorrect No Data values were replaced in 288 of the 8-m mosaic tiles and the Antarctica-wide mosaics. Supersedes #8.
weiji14
added
enhancement ✨
Update tables with new metadata for REMA_v1.1. Markdown table updated in misc/README.md with new filename and resolution (data_list.yml patched with correct 100m resolution...). Also updated the rendered tables in jupyter notebook. Pandas queries modernized to use `.query` instead of `.loc`. Tables now have consistent table ids with changes to the pprint_table lambda function.
Truly adding geopandas now (not as in 9a932bf). Using latest from github master with a fiona deprecation warning fix geopandas/geopandas#854. Descartes also being installed for plotting purposes.
There are some data gaps in the 100m spatial resolution mosaic, so getting the 200m resolution (v1.1 this time) to gap fill those holes.
Better visibility of the exact areas of the tiles we use for training. Saved in Antarctic Polar Stereographic Projection (EPSG:3031) and WGS84 (EPSG:4326) for easy rendering in Github. Plots of the tiles uses geopandas instead of shapely now in the jupyter notebook. Also update some README.md files describing some of the new stuff.
Ensures that the selective_tiling function works. Not a very well written integration test (there are hard coded elements) but it does capture an important part of the data processing pipeline. The test retrieves the bounding boxes directly from the tiles_3031.geojson file, meaning it excludes the get_window_bounds function (which is itself captured in a unit test only).
Core part of #64. Higher resolution tiles of Antarctica's surface elevation! The 100m Reference Elevation Model of Antarctica v1.1 has some data gaps/holes, so we're filling it in with the 200m resolution mosaic (resampled to 100m). Tiles are still 8km by 8km in shape, but we now have 80pixel by 80pixel REMA tiles, up from the 40pixel by 40pixel tiles previously. Note that the gapfill part of the selective_tile function is not covered by any unit or integration tests.
Retrain Super Resolution Generative Adversarial Network on the new Reference Elevation Model of Antarctica v1.1 100m resolution (gapfilled) tiles from cd72a30. Hard coded input shape, filter kernel size and strides for REMA have to change to the new 80pixel by 80pixel tile shape. Trained for 200 epochs instead of 50 epochs before. Visual inspection of results appear to show a minor improvement over previous training in 33666cc using 200m resolution REMA. Somewhat smoother looking results, but still have checkerboard artifacts because tile borders are not well resolved.
weiji14
changed the title
weiji14
added a commit
that referenced
this pull request
Aug 30, 2019
Write script for gapfilling the 100m REMA Ice Surface Elevation DEM with the seamless 200m version (bilinear interpolated), i.e. a more proper version of cd72a30 of #64. Based on the old selective_tiling code's gapfill_raster section that we deprecated in 690c365. I've experimented with alternative methods such as making a virtual (.vrt) GeoTIFF, mosaicking using pure GDAL and rasterio's merge tool, even considered GMT's grdblend, but nothing really merges the two together (with REMA_100 as highest priority, then REMA_200) properly the way I want it, in a reasonable-ish amount of time. Might be good to actually output this homemade 100m gapfilled REMA to a Cloud-Optimized GeoTIFF, NetCDF or Zarr, but we'll stick to good ol' GeoTIFF for now, even if it is 9.9GB.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Increasing spatial resolution of our Antarctic surface digital elevation model! There is finally a 100m mosaic product for the whole continent. Note that the dataset doi (currently linking to Harvard Dataverse) is still at v1.0.
From the 'release notes' at https://www.pgc.umn.edu/data/rema/:
REMA Release 1.1 corrects 2 minor issues in the mosaic tiles:
Supersedes #8.
TODO: