Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve geotop functionality #196

Merged
merged 25 commits into from
Jul 20, 2023
Merged

Improve geotop functionality #196

merged 25 commits into from
Jul 20, 2023

Conversation

dbrakenhoff
Copy link
Collaborator

@dbrakenhoff dbrakenhoff commented Jun 21, 2023 •
edited
Loading

See #193 .

Introduces:

  • Restructure geotop reading to be more similar to regis (e.g. get_geotop now downloads raw data)
  • Use dask.delayed to download geotop from URL
  • Add support in caching for dask.delayed datasets
  • rewrite memory intensive operations to be more memory efficient

Warning: this PR introduces some breaking changes:

  • nlmod.read.geotop.get_geotop() now loads raw geotop data (what used to be performed by get_geotop_raw_within_extent(). Using this old method will raise a DeprecationWarning).
  • to reproduce the old get_geotop(), which converts geotop data to a layer model:
geotop_ds = nlmod.read.geotop.get_geotop(extent)  # load raw data
layer_ds = nlmod.read.geotop.to_model_layers(geotop_ds)  # convert to layer model

dbrakenhoff and others added 23 commits June 16, 2023 11:13
@dbrakenhoff
improve get_geotop
2c918b4 
- use chunks and dask delayed to keep memory usage efficient
- get_geotop now gets raw data
- add deprecationwarning to get_geotop_raw_within_extent()
@dbrakenhoff
old get_geotop is renamed geotop_to_layermodel:
456b983 
- now takes geotop_ds as input to convert to layers (not extent)
@dbrakenhoff
clean up old code
84593ce
@dbrakenhoff
improve get_regis:
62a9861 
- do not use sortby but check diff of y dimension and flip if necessary
@dbrakenhoff
add cache support for chunked datasets
56a264e
@dbrakenhoff
add log msg to split_layers
68a4c1f
@OnnoEbbens
remove nan layer (value -127) from geo_eenheden
4ed3976
@dbrakenhoff
use values instead of data to deal with potential dask array
25f4f72
@dbrakenhoff
improve geotop:
f9cc9d7 
- use .values instead of .data to load data into memory if using dask
- undo onno changes for now w missing strat code
@dbrakenhoff
if caching with dask, close and reopen ds to ensure ds points to file
34dfdcf
@dbrakenhoff
fix DeprecationWarning
de54e3d
@dbrakenhoff
fix tests
87e7afe
@dbrakenhoff
fix geotop_lithok_in_cross_section
a6a26c5 
- use .values to deal with dask arrays
- use get_geotop
- update docstring
@rubencalje
Rename geotop_to_layermodel to to_model_layers and remove convert_geo…
471e724 
…top_to_ml_layers
@rubencalje
Some minor changes
26db2e8
@rubencalje
Fix combination of regis and geotop
a551f55
@rubencalje
Improve geotop.to_model_layers and add method insert_layer
fd588c7
@rubencalje
Add extra legends for layer models
776a654 
Add
@rubencalje
Add Formatie van Nieuwkoop, Laagpakket van Nij Beets in geotop
ecbe766
@rubencalje
Add test_insert_layer
e016683
@rubencalje
Fix tests and codacy issues
25ad000
@rubencalje
Set the default of method_gullies to "add_to_layer_below"
c033401
@rubencalje
Rename gullies to geulen and gully to geul
c45d9a0
@rubencalje rubencalje marked this pull request as ready for review July 17, 2023 13:39
dbrakenhoff
dbrakenhoff commented Jul 17, 2023
View reviewed changes
nlmod/dims/layers.py Outdated Show resolved Hide resolved
dbrakenhoff
dbrakenhoff commented Jul 17, 2023
View reviewed changes
nlmod/dims/layers.py Show resolved Hide resolved
nlmod/dims/layers.py Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
nlmod/dims/layers.py Outdated Show resolved Hide resolved
@dbrakenhoff
resolve my own comments
b4140db
@dbrakenhoff
please codacy
28ee878 
- replace Exception with more specific types
- add LayerError and MissingValueError
- fix some typos
@dbrakenhoff dbrakenhoff merged commit 787e12c into dev Jul 20, 2023
1 of 2 checks passed
@dbrakenhoff dbrakenhoff deleted the 193-geotop branch July 20, 2023 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@dbrakenhoff @OnnoEbbens @rubencalje