ENH: Infer suitable non optional valid_range

[cpelley]

Writing NetCDF files requires the the valid_range to be present. The not explicitly defined, NetCDF specification defines how this should be derived:

"If neither valid_min, valid_max nor valid_range is defined then generic applications should define a valid range as follows. If the data type is byte and _FillValue is not explicitly defined, then the valid range should include all possible values. Otherwise, the valid range should exclude the _FillValue (whether defined explicitly or by default) as follows. If the _FillValue is positive then it defines a valid maximum, otherwise it defines a valid minimum. For integer types, there should be a difference of 1 between the _FillValue and this valid minimum or maximum. For floating point types, the difference should be twice the minimum possible (1 in the least significant bit) to allow for rounding error.

If the variable is packed using scale_factor and add_offset attributes (see below), the _FillValue, missing_value, valid_range, valid_min, or valid_max attributes should have the data type of the packed data."

Reference:
http://www.unidata.ucar.edu/software/netcdf/docs/attribute_conventions.html

Relates to #2098

[cpelley]

In practice I think the specification means the following:

        # Ensure expected valid_range populated when not present               
        valid_range = cube.attributes.get('valid_range')                       
        if valid_range is None:                                                
            fill_value = None                                                  
            if isinstance(cube.data, ma.core.MaskedArray):                     
                fill_value = cube.data.fill_value                              
            valid_range = np.array([cube.data.min(), cube.data.max()],         
                                   dtype=data_dtype)                           
            if data_dtype.itemsize is 1 and fill_value is None:                
                    cube.attributes['valid_range'] = valid_range               
            else:                                                              
                fill_value = cube.lazy_data().fill_value                       
                if fill_value >= 0:                                            
                    valid_range[1] = fill_value                                
                    if np.issubclass_(valid_range.dtype.type, np.integer):     
                        valid_range[1] += 1                                    
                    else:                                                      
                        valid_range[1] += (fill_value +                        
                            (2 * np.finfo(valid_range.dtype.type).min()))      
                elif fill_value < 0:                                           
                    valid_range[0] = fill_value                                
                    if np.issubclass_(valid_range.dtype.type, np.integer):     
                        valid_range[0] -= 1                                    
                    else:                                                      
                        valid_range[0] -= (fill_value +                        
                            (2 * np.finfo(valid_range.dtype.type).min()))      
            cube.attributes['valid_range'] = valid_range

[marqh]

Writing NetCDF files requires the the valid_range to be present. The not explicitly defined, NetCDF specification defines how this should be derived:

I am not at all sure that this statement is correct. This is a purely optional facility.
A vast majority of netCDF files do not define a valid_range, in my experience

The language leaves something to be desired, but I do not think it supports this approach

[cpelley]

This is a purely optional facility.

Gosh, I think your right. I think the statement can mean two completely widely different things!
Thanks @marqh I had took the other interpretation, yes common usage suggests your interpretation to be the correct one.

This leaves the following cases which are driven by a real use-cases:

1. Supporting logical datasets in NetCDF (valid_range=[0, 1]).
2. Supporting unsigned integers for NetCDF-classic (valid_range=[0, fill_value]).
   • i.e. coercing uint8 to int16 and deriving valid_range accordingly for example.

Are you content with deriving valid_range in these limited cases?

Thanks @marqh

[marqh]

Are you content with deriving valid_range in these limited cases?

I am unsure what I am being asked to be content about, please add specific details.

Are you intending for there to be code and, if so, what will this code actually do?

[cpelley]

Are you intending for there to be code and, if so, what will this code actually do?

The two use cases are about deriving a valid_range to support datatypes in NetCDF:

Here is some very rough illustrative code for these two cases:

NetCDF - any (logical data)

if arr.dtype == np.bool:
    if valid_range not in cube.attributes and valid_min not in cube.attributes and valid_max in cube.attributes:
        cube.attributes['valid_range'] = [0, 1]
        cube.data = cube.data.astype('int8')

NetCDF3 or NetCDF-classic (unsigned integer data)

if arr.dtype.kind == 'u':
    if valid_range not in cube.attributes and valid_min not in cube.attributes and valid_max in cube.attributes:
        iinfo = np.iinfo(arr.dtype)
        cube.attributes['valid_range'] = [iinfo.min(), iinfo.max()]
        cube.data = cube.data.astype(....)

This is about providing support for boolean arrays (NetCDF) and unsigned integers (NetCDF-classic) without unnecessarily requiring human intervention. NetCDF3 documents this as the means to represent such datatypes so I don't see why we can't get iris to to this. Also, astype can be lazy.

I would prefer not to have remember to set cube.attributes['valid_range'] = [0, 1] every time I work with a boolean array when iris knows already by it being a boolean array...

Cheers

[cpelley]

Does this sound acceptable behaviour? We and our users currently populate this information on each case individually at this time when saving bool/int8 data. It would be good to pull this logic as above into iris to remove the need to hardcode this locally.

[marqh]

Hi @cpelley

I can see the interest in this approach. There are times when these metadata elements are required and inferring them from the data is useful.

I think there are also times when they need to be set directly, and this is supported by setting cube.attributes

I wonder whether there is a case for them not being set, the current default position.

If so, the questions arise:

• how to avoid this happening?
• is the default to have this on or off?

the saving API could be getting quite complex, so alternative options are being investigated, e.g.:
#2467

Do you think this, or a similar approach, would provide a framework to enable the features you desire at run time, by configuration file, or in code, in a simple fashion?

[cpelley]

I can't see from the ticket description/comments exactly what is proposed so I'll take a look at the source when I have time and get back to you.

Thanks for the ping

We now wrap iris NetCDF and supersede the iris NetCDF saver in order to provide data type coercion (where safe to do so), attribute inferring for logical/unsigned int and setting NETCDF4_CLASSIC as default in order to ensure CF compliance for users of our project. Perhaps we should meet to discuss this ticket in the wider context?

Thanks @marqh

Cheers

[cpelley]

Looks like netcdf-python interprets valid_min/max now for determining mask:
"6/10/2017: Version 1.2.9 released. Fixes for auto-scaling and masking when _Unsigned and/or valid_min, valid_max attributes present."

[pelson]

@cpelley - I've read this thread a few times now, and am wondering whether there is a proposal that we can develop, or whether there is still an open question? Has the recent work in netcdf4-python marked part of this issue obsolete?

[cpelley]

Thanks for ping @pelson
I haven't checked, but I didn't think that netcdf-python derived valid min and valid max when not present.
My understanding was that netcdf-python uses them when present to determines the mask (which is itself a problem). valid_min and valid_max have more than one purpose as highlighted by this issue.

I'll close this for now until such a time that it comes back into context with what I'm doing.

Cheers