Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Get uxarray running with FVCOM model output #385

Open
kthyng opened this issue Jul 18, 2023 · 12 comments
Open

Get uxarray running with FVCOM model output #385

kthyng opened this issue Jul 18, 2023 · 12 comments
Assignees
Labels
bug Something isn't working
Milestone

Comments

@kthyng
Copy link
Contributor

kthyng commented Jul 18, 2023

FVCOM is a commonly-used ocean model with an unstructured triangular mesh. The community would like to be able to use uxarray with this model output, as well as other unstructured ocean models, but this is a good one to start with.

To Reproduce
With a small change implemented in PR #384, here is the status of reading in an example FVCOM file. (Note that the date will have to be moved forward at some point as these files do not stay around forever. If needed, bump up the date in two places to today's date.)

import uxarray as ux
url = "https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2023/07/18/nos.sfbofs.fields.f000.20230718.t03z.nc"
uxds = ux.open_dataset(url, url, drop_variables=["siglay","siglev"])

Currently (with the PR) this returns:

RuntimeError: Could not recognize dataset format.

Full stack trace:

--------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) Cell In[3], line 2 1 url = "https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2023/07/18/nos.sfbofs.fields.f000.20230718.t03z.nc" ----> 2 uxds_single = uxr.open_dataset(url, url, drop_variables=["siglay","siglev"])

File ~/packages/uxarray/uxarray/core/api.py:165, in open_dataset(grid_filename_or_obj, filename_or_obj, gridspec, vertices, islatlon, isconcave, use_dual, **kwargs)
109 """Wraps xarray.open_dataset(), given a grid topology definition with a
110 single dataset file or object with corresponding data.
111
(...)
161 >>> ux_ds = ux.open_dataset("grid_filename.g", "grid_filename_vortex.nc")
162 """
164 ## Grid definition
--> 165 uxgrid = open_grid(grid_filename_or_obj,
166 gridspec=gridspec,
167 vertices=vertices,
168 islatlon=islatlon,
169 isconcave=isconcave,
170 use_dual=use_dual,
171 **kwargs)
173 ## UxDataset
174 ds = xr.open_dataset(filename_or_obj, decode_times=False,
175 **kwargs) # type: ignore

File ~/packages/uxarray/uxarray/core/api.py:90, in open_grid(grid_filename_or_obj, gridspec, vertices, islatlon, isconcave, use_dual, **kwargs)
85 else:
86 grid_ds = xr.open_dataset(grid_filename_or_obj,
87 decode_times=False,
88 **kwargs) # type: ignore
---> 90 uxgrid = Grid(grid_ds,
91 gridspec=gridspec,
92 vertices=vertices,
93 islatlon=islatlon,
94 isconcave=isconcave,
95 source_grid=str(grid_filename_or_obj),
96 use_dual=use_dual)
98 return uxgrid

File ~/packages/uxarray/uxarray/core/grid.py:100, in Grid.init(self, input_obj, **kwargs)
97 # check if initializing from string
98 # TODO: re-add gridspec initialization when implemented
99 elif isinstance(input_obj, xr.Dataset):
--> 100 self.mesh_type = parse_grid_type(input_obj)
101 self.from_ds(dataset=input_obj)
102 else:

File ~/packages/uxarray/uxarray/utils/helpers.py:48, in parse_grid_type(dataset)
46 mesh_type = "mpas"
47 else:
---> 48 raise RuntimeError(f"Could not recognize dataset format.")
49 return mesh_type
51 # check mesh topology and dimension

RuntimeError: Could not recognize dataset format.

Expected behavior
Not sure yet since I haven't used uxarray successfully, but presumably read in the model output file without error and then be able to do some stuff with it.

OS:
Mac

Environment

# packages in environment at /Users/kthyng/miniconda3/envs/uxarray: # # Name Version Build Channel anyio 3.7.1 pyhd8ed1ab_0 conda-forge appnope 0.1.3 pyhd8ed1ab_0 conda-forge argon2-cffi 21.3.0 pyhd8ed1ab_0 conda-forge argon2-cffi-bindings 21.2.0 py311h5547dcb_3 conda-forge asttokens 2.2.1 pyhd8ed1ab_0 conda-forge async-lru 2.0.3 pyhd8ed1ab_0 conda-forge attrs 23.1.0 pyh71513ae_1 conda-forge babel 2.12.1 pyhd8ed1ab_1 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 pyhd8ed1ab_3 conda-forge backports.functools_lru_cache 1.6.5 pyhd8ed1ab_0 conda-forge beautifulsoup4 4.12.2 pyha770c72_0 conda-forge bleach 6.0.0 pyhd8ed1ab_0 conda-forge blosc 1.21.4 heccf04b_0 conda-forge brotli-python 1.0.9 py311h814d153_9 conda-forge bzip2 1.0.8 h0d85af4_4 conda-forge c-ares 1.19.1 h0dc2134_0 conda-forge ca-certificates 2023.5.7 h8857fd0_0 conda-forge certifi 2023.5.7 pyhd8ed1ab_0 conda-forge cffi 1.15.1 py311ha86e640_3 conda-forge cftime 1.6.2 py311hd5badaa_1 conda-forge charset-normalizer 3.2.0 pyhd8ed1ab_0 conda-forge comm 0.1.3 pyhd8ed1ab_0 conda-forge debugpy 1.6.7 py311h814d153_0 conda-forge decorator 5.1.1 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge entrypoints 0.4 pyhd8ed1ab_0 conda-forge exceptiongroup 1.1.2 pyhd8ed1ab_0 conda-forge executing 1.2.0 pyhd8ed1ab_0 conda-forge flit-core 3.9.0 pyhd8ed1ab_0 conda-forge hdf4 4.2.15 h9804679_6 conda-forge hdf5 1.14.1 nompi_hedada53_100 conda-forge icu 72.1 h7336db1_0 conda-forge idna 3.4 pyhd8ed1ab_0 conda-forge importlib-metadata 6.8.0 pyha770c72_0 conda-forge importlib_metadata 6.8.0 hd8ed1ab_0 conda-forge importlib_resources 6.0.0 pyhd8ed1ab_1 conda-forge ipykernel 6.24.0 pyh5fb750a_0 conda-forge ipython 8.14.0 pyhd1c38e8_0 conda-forge jedi 0.18.2 pyhd8ed1ab_0 conda-forge jinja2 3.1.2 pyhd8ed1ab_1 conda-forge json5 0.9.14 pyhd8ed1ab_0 conda-forge jsonschema 4.18.3 pyhd8ed1ab_0 conda-forge jsonschema-specifications 2023.6.1 pyhd8ed1ab_0 conda-forge jupyter-lsp 2.2.0 pyhd8ed1ab_0 conda-forge jupyter_client 8.3.0 pyhd8ed1ab_0 conda-forge jupyter_core 5.3.1 py311h6eed73b_0 conda-forge jupyter_events 0.6.3 pyhd8ed1ab_0 conda-forge jupyter_server 2.7.0 pyhd8ed1ab_0 conda-forge jupyter_server_terminals 0.4.4 pyhd8ed1ab_1 conda-forge jupyterlab 4.0.3 pyhd8ed1ab_0 conda-forge jupyterlab_pygments 0.2.2 pyhd8ed1ab_0 conda-forge jupyterlab_server 2.23.0 pyhd8ed1ab_0 conda-forge krb5 1.21.1 hb884880_0 conda-forge libaec 1.0.6 hf0c8a7f_1 conda-forge libblas 3.9.0 17_osx64_openblas conda-forge libcblas 3.9.0 17_osx64_openblas conda-forge libcurl 8.1.2 h5f667d7_1 conda-forge libcxx 16.0.6 hd57cbcb_0 conda-forge libedit 3.1.20191231 h0678c8f_2 conda-forge libev 4.33 haf1e3a3_1 conda-forge libexpat 2.5.0 hf0c8a7f_1 conda-forge libffi 3.4.2 h0d85af4_5 conda-forge libgfortran 5.0.0 11_3_0_h97931a8_31 conda-forge libgfortran5 12.2.0 he409387_31 conda-forge libiconv 1.17 hac89ed1_0 conda-forge libjpeg-turbo 2.1.5.1 hb7f2c08_0 conda-forge liblapack 3.9.0 17_osx64_openblas conda-forge libllvm14 14.0.6 hc8e404f_3 conda-forge libnetcdf 4.9.2 nompi_h6ba5aa8_107 conda-forge libnghttp2 1.52.0 he2ab024_0 conda-forge libopenblas 0.3.23 openmp_h429af6e_0 conda-forge libsodium 1.0.18 hbcb3906_1 conda-forge libsqlite 3.42.0 h58db7d2_0 conda-forge libssh2 1.11.0 hd019ec5_0 conda-forge libxml2 2.11.4 hd95e348_0 conda-forge libzip 1.9.2 h6db710c_1 conda-forge libzlib 1.2.13 h8a1eda9_5 conda-forge llvm-openmp 16.0.6 hff08bdf_0 conda-forge llvmlite 0.40.1 py311hcbb5c6d_0 conda-forge lz4-c 1.9.4 hf0c8a7f_0 conda-forge markupsafe 2.1.3 py311h2725bcf_0 conda-forge matplotlib-inline 0.1.6 pyhd8ed1ab_0 conda-forge mistune 3.0.0 pyhd8ed1ab_0 conda-forge nbclient 0.8.0 pyhd8ed1ab_0 conda-forge nbconvert-core 7.6.0 pyhd8ed1ab_0 conda-forge nbformat 5.9.1 pyhd8ed1ab_0 conda-forge ncurses 6.4 hf0c8a7f_0 conda-forge nest-asyncio 1.5.6 pyhd8ed1ab_0 conda-forge netcdf4 1.6.4 nompi_py311ha27cd03_101 conda-forge notebook-shim 0.2.3 pyhd8ed1ab_0 conda-forge numba 0.57.1 py311h5a8220d_0 conda-forge numpy 1.24.4 py311hc44ba51_0 conda-forge openssl 3.1.1 h8a1eda9_1 conda-forge overrides 7.3.1 pyhd8ed1ab_0 conda-forge packaging 23.1 pyhd8ed1ab_0 conda-forge pandas 2.0.3 py311hab14417_1 conda-forge pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge parso 0.8.3 pyhd8ed1ab_0 conda-forge pexpect 4.8.0 pyh1a96a4e_2 conda-forge pickleshare 0.7.5 py_1003 conda-forge pip 23.1.2 pyhd8ed1ab_0 conda-forge pkgutil-resolve-name 1.3.10 pyhd8ed1ab_0 conda-forge platformdirs 3.8.1 pyhd8ed1ab_0 conda-forge prometheus_client 0.17.1 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.39 pyha770c72_0 conda-forge prompt_toolkit 3.0.39 hd8ed1ab_0 conda-forge psutil 5.9.5 py311h5547dcb_0 conda-forge ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pure_eval 0.2.2 pyhd8ed1ab_0 conda-forge pycparser 2.21 pyhd8ed1ab_0 conda-forge pygments 2.15.1 pyhd8ed1ab_0 conda-forge pyobjc-core 9.2 py311hf110eff_0 conda-forge pyobjc-framework-cocoa 9.2 py311hf110eff_0 conda-forge pysocks 1.7.1 pyha2e5f31_6 conda-forge python 3.11.4 h30d4d87_0_cpython conda-forge python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge python-fastjsonschema 2.17.1 pyhd8ed1ab_0 conda-forge python-json-logger 2.0.7 pyhd8ed1ab_0 conda-forge python-tzdata 2023.3 pyhd8ed1ab_0 conda-forge python_abi 3.11 3_cp311 conda-forge pytz 2023.3 pyhd8ed1ab_0 conda-forge pyyaml 6.0 py311h5547dcb_5 conda-forge pyzmq 25.1.0 py311h5dacc12_0 conda-forge readline 8.2 h9e318b2_1 conda-forge referencing 0.29.1 pyhd8ed1ab_0 conda-forge requests 2.31.0 pyhd8ed1ab_0 conda-forge rfc3339-validator 0.1.4 pyhd8ed1ab_0 conda-forge rfc3986-validator 0.1.1 pyh9f0ad1d_0 conda-forge rpds-py 0.8.10 py311h299eb51_0 conda-forge send2trash 1.8.2 pyhd1c38e8_0 conda-forge setuptools 68.0.0 pyhd8ed1ab_0 conda-forge six 1.16.0 pyh6c4a22f_0 conda-forge snappy 1.1.10 h225ccf5_0 conda-forge sniffio 1.3.0 pyhd8ed1ab_0 conda-forge soupsieve 2.3.2.post1 pyhd8ed1ab_0 conda-forge stack_data 0.6.2 pyhd8ed1ab_0 conda-forge terminado 0.17.1 pyhd1c38e8_0 conda-forge tinycss2 1.2.1 pyhd8ed1ab_0 conda-forge tk 8.6.12 h5dbffcc_0 conda-forge tomli 2.0.1 pyhd8ed1ab_0 conda-forge tornado 6.3.2 py311h2725bcf_0 conda-forge traitlets 5.9.0 pyhd8ed1ab_0 conda-forge typing-extensions 4.7.1 hd8ed1ab_0 conda-forge typing_extensions 4.7.1 pyha770c72_0 conda-forge typing_utils 0.1.0 pyhd8ed1ab_0 conda-forge tzdata 2023c h71feb2d_0 conda-forge urllib3 2.0.3 pyhd8ed1ab_1 conda-forge wcwidth 0.2.6 pyhd8ed1ab_0 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 1.6.1 pyhd8ed1ab_0 conda-forge wheel 0.40.0 pyhd8ed1ab_0 conda-forge xarray 2023.6.0 pyhd8ed1ab_0 conda-forge xz 5.2.6 h775f41a_0 conda-forge yaml 0.2.5 h0d85af4_2 conda-forge zeromq 4.3.4 he49afe7_1 conda-forge zipp 3.16.0 pyhd8ed1ab_1 conda-forge zstd 1.5.2 h829000d_7 conda-forge
@kthyng kthyng added the bug Something isn't working label Jul 18, 2023
@kthyng
Copy link
Contributor Author

kthyng commented Jul 18, 2023

@rajeeja
Copy link
Contributor

rajeeja commented Jul 18, 2023

Thanks for the issue and PR.

How do I download and open this file? Can xarray (no uxarray) open it?

wget https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2023/07/14/nos.sfbofs.fields.f000.20230714.t03z.nc

--2023-07-18 10:55:10--  https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2023/07/14/nos.sfbofs.fields.f000.20230714.t03z.nc
Resolving opendap.co-ops.nos.noaa.gov (opendap.co-ops.nos.noaa.gov)... 137.75.111.152, 137.75.111.172
Connecting to opendap.co-ops.nos.noaa.gov (opendap.co-ops.nos.noaa.gov)|137.75.111.152|:443... connected.
HTTP request sent, awaiting response... 400
2023-07-18 10:55:10 ERROR 400: (no description).

@kthyng
Copy link
Contributor Author

kthyng commented Jul 18, 2023

@kthyng
Copy link
Contributor Author

kthyng commented Jul 18, 2023

You might need to drop another variable also, I just read this in with xarray, and note that the server isn't very reliable so sometimes I have to run it a few times.

import xarray as xr
url = 'https://opendap.co-ops.nos.noaa.gov/thredds/dodsC/NOAA/SFBOFS/MODELS/2023/07/18/nos.sfbofs.fields.f000.20230718.t03z.nc'
xr.open_dataset(url, drop_variables=["siglay","siglev","Itime2"])

@kthyng
Copy link
Contributor Author

kthyng commented Jul 18, 2023

cc @jonmjoyce also

@ChrisBarker-NOAA
Copy link

It would be good to provide a good test file, rather than links to a OPenDAP server.

I sent a 2D one that xarray will open via email for a start, but ideally, we'd make a test file that's a LOT smaller. Maybe I'll get to that in a couple days, but I"ve got some other pressing issues to dal with now.

@kthyng: maybe you could whip one out with extract_model?

@kthyng
Copy link
Contributor Author

kthyng commented Jul 18, 2023

It is pretty big but it is also what is used so it has that advantage. I would guess that for now it will work ok, esp since they can download the file if they want to work locally.

@philipc2 philipc2 moved this from 📝 To Do to 🏗 In progress in UXarray Development Jul 18, 2023
@philipc2 philipc2 linked a pull request Jul 19, 2023 that will close this issue
8 tasks
@philipc2
Copy link
Member

@kthyng @kthyng

You might need to drop another variable also, I just read this in with xarray, and note that the server isn't very reliable so sometimes I have to run it a few times.

This dataset works for now, but a smaller one would be great so we could add it to our test files!

I'll work on getting a reader function for the model output. Would either of you be able to provide me a reference document to the mesh definition/specifications, possibly similar to this one from MPAS

@ChrisBarker-NOAA
Copy link

You might need to drop another variable also

Latest dev version of xarray should now work out of the box :-)

pydata/xarray#7989

@rajeeja
Copy link
Contributor

rajeeja commented Aug 16, 2023

@kthyng any updates on a small test file? please test and report with if the above changes work for you. Thanks!

@philipc2 philipc2 moved this from 🏗 In progress to 📝 To Do in UXarray Development Aug 21, 2023
@dcherian
Copy link
Contributor

Just released. See if it works for you!

@kthyng
Copy link
Contributor Author

kthyng commented Aug 21, 2023

@ChrisBarker-NOAA Do you have a test FVCOM file handy? It would take me a bit to get up and running to get one made.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📚 Backlog
Development

Successfully merging a pull request may close this issue.

6 participants