FIX: handle NaT values in dt-accessor

[kmuehlbauer]

• Closes .dt accessor returns int instead of float, resulting in misrepresentation of NaT values #7928
• Tests added
• User visible changes (including notable bug fixes) are documented in whats-new.rst

[keewis]

if I understand correctly, pandas will return float arrays / series if there's NaT, so checking the output dtypes might be faster?

[kmuehlbauer]

if I understand correctly, pandas will return float arrays / series if there's NaT, so checking the output dtypes might be faster?

Thanks! Yes, indeed, so we only need to special case isocalendar.

[kmuehlbauer]

@keewis Do you have a recommendation how to retrieve resulting dtype for the dask-part?

[keewis]

right, with dask this could be tricky, since we cannot know if the data contains missing values or not... I really would like to have nullable integer dtypes in situations like these.

[kmuehlbauer]

with dask this could be tricky, since we cannot know if the data contains missing values or not...

Even testing a small subset does not reveal the dtype in any case, since the NaT might be in any chunk we do not test. So we would have to know a priori to be on the safe side also for further processing in the dask world.

[kmuehlbauer]

One idea for the dask-part is to just keep current behaviour. On compute, this RuntimeError is raised:

RuntimeWarning: invalid value encountered in cast

The user could then just use .astype("float64") before computation. No proper solution, but maybe sufficient? I'll add the needed changes in a day or two.

[kmuehlbauer]

The current solution works for most of the .dt.accessor out of the box (it just keeps float64 if provided by pandas). For isocalendar it is more of a workaround until a proper fix is added upstream (proposed pandas-dev/pandas#54657). The workaround checks for presence of <NA> in the pandas output and handles appropriately. This might introduce a performance penalty.

Note: This does not help for the dask part, which is left unchanged. Here we would somehow need a priory knowledge if NaT is involved or not.

Question: Should we add tests in this PR for handling in presence of NaT? This is currently not tested at all.

[kmuehlbauer]

This is good to go from my side.

Unfortunately there has not been any comment on the upstream-issue so far: pandas-dev/pandas#54657. We can revisit and change when this is resolved over at pandas.

[kmuehlbauer]

@keewis I did not touch the dask part. I'd appreciate if you or others could have a final look for approval. Thanks.

[kmuehlbauer]

Will merge after tests are passing.