Skip to content

Feature: NetCDF output precision configurable #117

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
fmalatino merged 8 commits into from
Mar 5, 2025
Merged

Feature: NetCDF output precision configurable #117

Changes from 6 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
992f404
Removed hard-code of np.float32 from NetCDFMonitor transfer_type, rep…
fmalatino Feb 21, 2025
784bcc3
Added multiple options for NetCDF precision
fmalatino Feb 25, 2025
65b605f
Added checking for use of 32 precision and float64 output
fmalatino Feb 25, 2025
8bf7f13
Merge branch 'develop' into feature/netcdfout
fmalatino Feb 27, 2025
bdbc81f
Using NumPy type instead of string in NetCDFMonitor precision variable
fmalatino Mar 4, 2025
8d68884
Merge branch 'develop' into feature/netcdfout
fmalatino Mar 4, 2025
d78a527
Added warning to netcdf_monitor.py for mismatch in precision settings
fmalatino Mar 5, 2025
4993d0e
Forgot f-string in warn message of netcdf_monitor
fmalatino Mar 5, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 23 additions & 2 deletions ndsl/monitor/netcdf_monitor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
import numpy as np

from ndsl.comm.communicator import Communicator
from ndsl.dsl.typing import Float
from ndsl.filesystem import get_fs
from ndsl.logging import ndsl_log
from ndsl.monitor.convert import to_numpy
Expand Down Expand Up @@ -114,6 +115,7 @@ def __init__(
path: str,
communicator: Communicator,
time_chunk_size: int = 1,
precision=Float,
):
"""Create a NetCDFMonitor.

Expand All @@ -130,6 +132,21 @@ def __init__(
self._time_chunk_size = time_chunk_size
self.__writer: Optional[_ChunkedNetCDFWriter] = None
self._expected_vars: Optional[Set[str]] = None
if precision == Float:

@FlorianDeconinck FlorianDeconinck Mar 5, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would simplify and warn of cast here to be more flexible. Something like

self._transfer_type=precision
if self._transfer_type == np.float32 and get_precision() > 32:
   warn("NetCDF save: requested 32-bit float but precision of NDSL is {get_precision()}, cast will occur with possible loss of precision")

@FlorianDeconinck FlorianDeconinck Mar 5, 2025

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The idea, if I follow your code, would be to just write to the given user precision, but warn when the default of NDSL is higher

@fmalatino fmalatino Mar 5, 2025

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes this is the intention. I will make these changes thanks!

self._transfer_type = Float
elif precision == np.float32:
self._transfer_type = np.float32
elif precision == np.float64:
if np.float32 == Float:
raise ValueError(
f"Cannot output float64 with PACE_FLOAT_PRECISION set to {Float}"
)
self._transfer_type = np.float64
else:
raise ValueError(
"precision must be set to one of 'Float', 'float32', or 'float64"
f"got {precision}"
)

@property
def _writer(self):
Expand Down Expand Up @@ -164,12 +181,16 @@ def store(self, state: dict) -> None:
set(state.keys()), self._expected_vars
)
)
state = self._communicator.tile.gather_state(state, transfer_type=np.float32)
state = self._communicator.tile.gather_state(
state, transfer_type=self._transfer_type
)
if state is not None: # we are on root rank
self._writer.append(state)

def store_constant(self, state: Dict[str, Quantity]) -> None:
state = self._communicator.gather_state(state, transfer_type=np.float32)
state = self._communicator.gather_state(
state, transfer_type=self._transfer_type
)
if state is not None: # we are on root rank
constants_filename = str(
Path(self._path) / NetCDFMonitor._CONSTANT_FILENAME
Expand Down