API: Add Calendar Day Frequency ("CD")

[mroeschke]

This is a proposal to replace #20633.

offsets.Day ('D') is documented to represent calendar day; however, Day arithmetic and usage of the 'D' offset alias (e.g. df.resample('D'), timedelta_range(..., freq='D'), etc.) currently respects absolute time (i.e. Day acts like Timedelta(days=1)). This usage is ingrained pretty deeply in a lot of methods and operations and is fairly difficult to walk back to respect the notion of calendar day as stated in the docs.

Instead, I propose to keep Day as is (a timedelta-like frequency like Hour ('H')) and add a new frequency CalendarDay, 'CD'. It would act very similarly to DateOffset(day=1) but act like a frequency. Thoughts?

[chris-b1]

Only partially related, but I've also thought it would be nice to have frequencies for other DateOffset cases, e.g. a calendar year like the below.

In [7]: pd.date_range('2014-01-15', periods=10, freq=pd.DateOffset(years=1))
Out[7]: 
DatetimeIndex(['2014-01-15', '2015-01-15', '2016-01-15', '2017-01-15',
               '2018-01-15', '2019-01-15', '2020-01-15', '2021-01-15',
               '2022-01-15', '2023-01-15'],
              dtype='datetime64[ns]', freq='<DateOffset: kwds={'years': 1}>')

So, maybe a whole set of 'CD', 'CM', 'CA'?

[jorisvandenbossche]

e.g. df.resample('D') ... currently respects absolute time

@mroeschke can you give an example of that?

For example the below has a DST transition, but it keeps calendar days:

In [17]: pd.__version__
Out[17]: '0.23.4'

In [18]: idx = pd.date_range("2016-10-30", freq='H', periods=4*24, tz='Europe/Helsinki')

In [20]: pd.Series(index=idx).resample('D').count()
Out[20]: 
2016-10-30 00:00:00+03:00    0
2016-10-31 00:00:00+02:00    0
2016-11-01 00:00:00+02:00    0
2016-11-02 00:00:00+02:00    0
Freq: D, dtype: int64

[mroeschke]

Ah yes, you're correct @jorisvandenbossche. I had misspoke; df.resample('D') did respect calendar day.

Basically the crux of the issue is that Day ('D') is a subclass of Tick. Ticks act like Timedeltas and their arithmetic respects absolute time, but it's special cased in some operations like date_range and resample to respect calendar time:

In [1]: ts = pd.Timestamp('2016-10-30 00:00:00', tz='Europe/Helsinki')

In [2]: pd.date_range(start=ts, freq='D', periods=3)
Out[2]:
DatetimeIndex(['2016-10-30 00:00:00+03:00', '2016-10-31 00:00:00+02:00',
               '2016-11-01 00:00:00+02:00'],
              dtype='datetime64[ns, Europe/Helsinki]', freq='D')

In [3]: ts + pd.tseries.frequencies.to_offset('D')
Out[3]: Timestamp('2016-10-30 23:00:00+0200', tz='Europe/Helsinki')

So I think this is confusing, inconsistent to users, and the codebase has special caseslike if isinstance(offset, Day): #break Tick arithmetic.

It's pretty difficult to convert 'D' to just respect calendar day across the board because 'D' it's fully ingrained with timedeltas (absolute time) as well. Therefore, operations like timedelta_range(..., freq='D') would break if 'D' was converted to a calendar day i.e adding a calendar day to and absolute day doesn't make much sense.

Overall I think it's easier and cleaner to make 'D' respect absolute time across the board and create a new CalendarDay, 'CD', offset

[jorisvandenbossche]

So I think this is confusing, inconsistent to users

I don't debate the inconsistency, and that it would be nice to clean this up.

But there might be other ways to solve that.
In which cases does 'D' actually behave as 24H tick? Only when explicitly using an offset object like in your example? (or are there other cases? it would be good to better document this in this issue)
I think the cases where it is used as calendar day (resample, date_range) are more common, so we could also think about changing it the other way around?

timedelta_range(..., freq='D') would break if 'D' was converted to a calendar day

Yep, but we could say that 'D' is here interpreted as 24H, as timedelta's cannot hold a non-fixed frequency anyway

---

The current PR breaks (all in case of tz-aware data of course):

• the result of date_range
• the result of resample('D') (both the index and the values): the result are no longer daily values
• plotting of a result of any of the above (since the frequency is not regular any more, it uses a different code path)

IMO that impact is too big to be justifiable, but that's of course open for discussion. But I think we can certainly not do such a change without proper deprecations.

[mroeschke]

Cases where 'D' behave as 24H tick:

• Timestamp/DatetimeIndex Arithmetic

In [2]: ts = pd.Timestamp('2016-10-30 00:00:00', tz='Europe/Helsinki')

In [6]: ts + pd.tseries.frequencies.to_offset('D')
Out[6]: Timestamp('2016-10-30 23:00:00+0200', tz='Europe/Helsinki')

In [7]: pd.DatetimeIndex([ts]) + pd.tseries.frequencies.to_offset('D')
Out[7]: DatetimeIndex(['2016-10-30 23:00:00+02:00'], dtype='datetime64[ns, Europe/Helsinki]', freq=None)

• Shift

In [3]: dti = pd.date_range(start=ts, freq='D', periods=3)

In [4]: dti
Out[4]:
DatetimeIndex(['2016-10-30 00:00:00+03:00', '2016-10-31 00:00:00+02:00',
               '2016-11-01 00:00:00+02:00'],
              dtype='datetime64[ns, Europe/Helsinki]', freq='D')

In [5]: dti.shift(1, 'D')
Out[5]:
DatetimeIndex(['2016-10-30 23:00:00+02:00', '2016-10-31 23:00:00+02:00',
               '2016-11-01 23:00:00+02:00'],
              dtype='datetime64[ns, Europe/Helsinki]', freq='D')

And the only reason 'D' behaves like 1 calendar day in date_range (and therefore resample since date_range is used to generate bins) is because of this special casing which gives me the impression that Day is being forced to operate in a way it wasn't intended:

pandas/pandas/core/indexes/datetimes.py

Line 530 in 0409521

if hasattr(freq, 'delta') and freq != offsets.Day():

Also if 'D' were to follow a calendar day, I am not a big fan of it meaning 23-25 hours for datetimes with timezones and 24 hours for timedeltas. I would prefer if D was consistent across the board.

To ease the transition for date_range, the default frequency could be 'CD' instead of 'D' to still default to calendar day frequency. Sure, there's no easy transition for resample('D') for datetimetz data.

[jorisvandenbossche]

because of this special casing which gives me the impression that Day is being forced to operate in a way it wasn't intended:

Or you could also say: the behavior as calendar day for date_range/resample was very much intentional because we explicitly special cased for it .. :)

---

The shift is a pitty .. but still, we could also opt to make that consistent with date_range/resample, instead of the other way around.

I personally do never use pd.tseries.frequencies.to_offset explicitly, or hardly ever shift, but I do use date_range and resample a lot. I would think many users have a similar pattern, but of course different use cases give different usage patterns, so hard to say.
But given my usage pattern, for me every time I use the string 'D', I would need to change it to 'CD', and I actually would almost never want 'D'. So then it seems a strange choice to opt for changing the common use cases of 'D', instead of the rare ones. But again, I don't know how typical my usage pattern is.

[mroeschke]

the behavior as calendar day for date_range/resample was very much intentional because we explicitly special cased for it .. :)

Sure it was intentional, but it ultimately conflated the original meaning of 'D'. And I don't think a special case should force 'D' to change its original meaning.

My biggest -1 is the potential for 'D' to have calendar day semantics with timestamps and absolute day semantics with timedeltas. ('CD' cannot be currently added to timedeltas). Even if this would be documented, I am not a fan of the mixed semantics and feels fundamentally wrong.

For all intents and purposes, 'CD' will be equivalent to 'D' except for a timezone aware timeseries with a DST transition. I am not sure what subset of users this represents (I'd be very curious to know as well). If we release a v0.24rc, we can gauge if the community feels strongly about this API change.

I am open for date_range to default to 'CD' for continuity from prior behavior.

Any thoughts @jbrockmendel?