Re-grid and re-tile groundtruth data with coordinates rounded to 250 #155
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Creating new groundtruth NetCDF grids using GMT surface, replacing the ones last created in b90bd74 in #112. Besides having updated to the GMT 6.0.0rc1 tagged release, the main change here is with using nicely rounded bounds (to 250 units in EPSG:3031) instead of arbitrary decimal points. This will really help resolve some of the problems with points not being included in our RMSE_test calculations near the grid's edges (see #152), and integer coordinates are just nicer to debug won't you say? Specifically, the data_prep.get_region was refactored to use `gmt info -I xxx.csv` instead of pure pandas, returning an `xmin/xmax/ymin/ymax` string that has an extended region optimized for `gmt surface`. There is a "surface [WARNING]: Your grid dimensions are mutually prime. Convergence is very unlikely" which I'm just gonna ignore for now. Note that data_prep.ascii_to_xyz was one-line patched to drop NaNs as there were some points in the WISE_ISODYN_RadarByFlight.XYZ file with missing elevation (z) values (since #112...) that was messing up gmt.info in the refactored data_prep.get_region. Unit tests have been modified accordingly, and the grids in the integration tests are now downloaded/created in folder /tmp to avoid messing with the actual files in highres. Matplotlib plots of the grids in data_prep.ipynb have been updated, and the new grids will be released in v0.9.2.
weiji14
added
enhancement ✨
|
Check out this pull request on ReviewNB: https://app.reviewnb.com/weiji14/deepbedmap/pull/155 You'll be able to see visual diffs and write comments on notebook cells. Powered by ReviewNB. |
Chipping out new training tiles from the '250 rounded' new grids. We are down from 2347 to 2275 tiles now... The GeoJSON bounding box polygons have been updated, and new quilt data hash to use now is 0734959aa4f4903a17ed2acdfd53b3c0c826aadfc718e5fdd3c1b04963e1206e.
Recalculating the RMSE_test value of our v0.9.0 trained model (at https://www.comet.ml/weiji14/deepbedmap/0b9b232394da42e394998b112f628696) on the new 250 rounded grids produced in 0681bf3. Basically comparing against the results reported in 75266fc. We now calculate elevation errors on 2007tx.nc using the full 42995 groundtruth points instead of just 37829 before, and the RMSE_test is now 44.34 which is up from 43.58 before. Good to see that results are comparable, but it does highlight how the border predictions need more optimization work. Also added a neat new functionality to the _download_deepbedmap_model_weights_from_comet function, downloading any trained .npz model weights via an experiment_key!
weiji14
added a commit
that referenced
this pull request
Jun 21, 2019
Closes #155 Re-grid and re-tile groundtruth data with coordinates rounded to 250.
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
So that we align better with BEDMAP2 and other raster grids out there, we are re-processing new groundtruth NetCDF grids (produced by data_prep.ipynb) with coordinates (in EPSG:3031) rounded to 250 units/metres. This means no more or less crazy decimal coordinates that should make our various scripts (especially those dependent on bounding boxes) play a lot more nicely together.
Fixes #152.
TODO: