Convert ISO date string to Python datetime object

[CoryMartin-NOAA]

Description

This PR adds to ioda_conv_engines the capability to check:

• Does the variable MetaData/dateTime exist?
• If so, is it a string?
• If so, convert it to datetime objects

Existing datetime objects are unaffected, and if the variable is defined as a integer with attributes describing the reference time, no changes occur.

Issue(s) addressed

Link the issues to be closed with this PR

• fixes Make ioda_conv_engines library handle datetime strings #827

Acceptance Criteria (Definition of Done)

Approved and tested by folks, note it won't work properly until changes are made to IODA

Dependencies

Needs changes to be made to IODA Python API to handle Python datetime objects before it will write to file properly.

Impact

None

[gthompsnJCSDA]

Excuse me for asking a couple dumb questions...

This just permits the use of something like 1999-12-31T09:28:42Z which we previously had as a datetime value in an arbitrary length-string to become an actual date-time python object, right?
2.
Is there any protection against out-of-range values for some component, like in the above example but setting it to 99 seconds? In other words, should there be a try-except clause to capture dumb edge cases? Or would date-time objects in python take the 99 seconds and add to the minutes (and upward) while giving a final answer of 39 seconds?

It appears there is no loop over the entire dtvar as the IF-test on line 163 checks only the first element of the vector. Is this safe for entire vector?

Final comment is that I was under an impression that this would enforce anyone using older datetime variable into using the newer type with seconds (or minutes or hours ...) from an Epoch time. I guess this kinda acts as number of seconds since Epoch is zero and the Epoch time itself is the dateTime value given.

[CoryMartin-NOAA]

@gthompsnJCSDA

1. now this allows for the authors of converters to do 1 of 3 things (int64+units attribute, ISO strings, datetime objects) with the latter 2 converted to the first by the IODA API.
2. When you define the object, python datetime is smart enough: t = dt.datetime(2022,3,11,30) will fail because hours (the 4th position) must be 0-23.
3. This is a good point. I don't know if it is 100% necessary to check the entire vector, as that might slow things down. But I suspect the writer will then error if it is trying to convert something that can't be converted, or if it attempts to write the wrong type.

This is so that people don't have to change the time in their converters from ISO strings and it will force the new epoch convention with the default 1970 time.

[gthompsnJCSDA]

@CoryMartin-NOAA Regarding your final comment about "forcing" to epoch of 1970. Is this actually needed to enforce? I just made some changes to Bo's modis-aod python script in which the incoming netCDF file is already using a specific reference time of seconds since 1993-01-01T00:00:00Z and has an array of seconds in double-precision floating point. The earlier converter was making those doubles transformed with datetime into ISO-8601 strings. I decided let's not waste any time at all and just allow the units of dateTime to be that 1993 date and then only needed to convert the double-precision times into int64 type. That should be saving a chunk of time. So then, in your coding is everything then forced into Epoch of 1970? Wouldn't that be wasted time for an incoming dataset already well structured to have its dateTime already fully standards compliant?

[CoryMartin-NOAA]

@gthompsnJCSDA if you (the converter developer) uses int64 and specifies the proper units, this is fine. But, another developer might just pass datetime objects or ISO strings, if those two are used, it doesn't actually know what your epoch is, so it uses the default. Does that make sense? I agree your approach is the best one, but not all datatypes will already have this convention and might just have ISO strings, or something else entirely.

[gthompsnJCSDA]

SNIP... another developer might just pass datetime objects or ISO strings, if those two are used, it doesn't actually know what your epoch is...

I realize what you're saying and agree. Having opened a few python converters during this sprint, I see in the case of modis-aod and probably viirs-aod, that the original data are already nearly what we want and have been converted to strings for the older datetime variable. So that is now wasteful and take minimal effort to correct during this sprint (if others notice what is being done). I made this change myself to modis-aod that Bo is working on. Perhaps I need to bring up this idea during our meeting to see if people can just fix this while they edit variable names and such.

[srherbener]

Thanks for adding this in @CoryMartin-NOAA! The ioda feature/sprint-ioda-converters branch has been sync'd with the ioda develop branch after I merged in jcsda-internal/ioda/pull/636. Merging this PR in should complete the library changes needed to use python datetime objects in the python converters. Correct?

[CoryMartin-NOAA]

@srherbener I think so yes, we can discuss dateTime with everyone at the top of the code sprint meeting.