#31793 Add support for subtracting datetime from Timestamp

[AnjoMan]

• closes Difference between two Timestamp with different timezone object type fails #31793
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Based on the discussion in the attached issue, i've made the following changes:

1. add support for subtracting when the two items have different timezones
2. raise an error if one (but not both) timezone is naive

The root cause of 31793 was that the tzinfo objects were from different libraries -- Timestamp makes a pytz.UTC whereas a datetime object might just have datetime.timezone.utc. These objects are not necessarily equal, even if they represent the same timezone.

In addition, this change adds support for time-deltas where the timezone of each date is not the same. This follows the behaviour of datetime more closely.

I reference the python datetime module documentation which defines when a datetime is naive: when its tzinfo is None or a call to item.tzinfo.utcoffset(item) returns None.

[AnjoMan]

If this is a welcome change, please add the hacktoberfest-accepted to this issue so it can go towards my event contribution

[AnjoMan]

Ok, looking at the failed tests, it l;ooks like i should be relocating the tests i added so they would appear under tests/arithmetic/test_timedetlta64.py and tests/scalar/timestamp/test_arithmetic.py.

[arw2019]

Thanks @AnjoMan for the PR!

Some comments

[arw2019]

Also heads up about linting - needs to be fixed for the CI checks to pass

[jbrockmendel]

When changing this for Timestamp, we'll need to make the same change for DatetimeArray

[AnjoMan]

When changing this for Timestamp, we'll need to make the same change for DatetimeArray

@jbrockmendel thanks for the heads-up, I'll take a look and try to replicate this functionality there as well.

Just so we're clear, you are expecting to find the same bug when doing something like

date_array_1 = DatetimeArray(...)
date_array_2 = DatetimeArray(...)

date_array_1 - date_array_2

and one of the arrays has datetime objects instead of Timestamp objects?

[jreback]

I -1 on adding different timezone subtraction operations. We do not allow this anywhere else. I think being explicit is a really good thing here.

[pep8speaks]

Hello @AnjoMan! Thanks for updating this PR. We checked the lines you've touched for PEP 8 issues, and found:

There are currently no PEP 8 issues detected in this Pull Request. Cheers! 🍻

Comment last updated at 2021-12-28 15:51:50 UTC

[jbrockmendel]

Just so we're clear, you are expecting to find the same bug when doing something like

date_array_1 = DatetimeArray(...)
date_array_2 = DatetimeArray(...)
date_array_1 - date_array_2

and one of the arrays has datetime objects instead of Timestamp objects?

datetime objects vs Timestamp objects isn't really applicable here. Think of DatetimeArray as being a vectorized Timestamp. Just like with Timestamp, when subtracting dta - dta2 we need to have not ((dta.tzinfo is None) ^ (dta2.tzinfo is None)). ATM we also require that the tzinfos match, which this PR would cease to require.

[jorisvandenbossche]

[@jreback] I am -1 on adding different timezone subtraction operations. We do not allow this anywhere else. I think being explicit is a really good thing here.

Apart from the fact that the stdlib also supports this as @jbrockmendel mentioned, I also want to point out we actually do support differing timezones in several operations, eg comparison ops:

In [5]: t1 = pd.Timestamp("2012-01-01").tz_localize("UTC").tz_convert("Europe/Brussels")

In [6]: t2 = pd.Timestamp("2012-01-01").tz_localize("UTC").tz_convert("US/Eastern")

In [7]: t1 == t2
Out[7]: True

In [8]: t1 >= t2
Out[8]: True

[github-actions]

This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.

[arw2019]

In light of this discussions in #37605 does this PR go ahead? AFAICT yes but maybe I misunderstood @jbrockmendel @jorisvandenbossche @jreback @mroeschke

If yes @AnjoMan you will need to merge master and add a whatsnew note. Not sure if this will get in in time for 1.2 - if not we'll have to wait until we tag the next major release (1.3 or 2.0)

[jbrockmendel]

In light of this discussions in #37605 does this PR go ahead? AFAICT yes but maybe I misunderstood

@jorisvandenbossche and I are +1 on this, @jreback was -1 though there's some evidence of miscommunication, and @mroeschke was -1 though I'm hoping he'll reconsider when looking at this context.

@arw2019 @AnjoMan I encourage you both to weigh in on #37605.

[mroeschke]

I'm more -0 just in principle, but since we already support some mixed timezone aware operations, it's not a hill I will die on.

[arw2019]

In light of this discussions in #37605 does this PR go ahead? AFAICT yes but maybe I misunderstood

@jorisvandenbossche and I are +1 on this, @jreback was -1 though there's some evidence of miscommunication, and @mroeschke was -1 though I'm hoping he'll reconsider when looking at this context.

@arw2019 @AnjoMan I encourage you both to weigh in on #37605.

I'm +1 on adding this

[jreback]

@AnjoMan if you can rebase we can take another look here

[AnjoMan]

Ok, i've rebased on the current dev branch and added a "whatsnew" entry.

I noticed one of the CI checks fails with a bad import -- it doesn't seem too related to what I've changed

[arw2019]

Ok, i've rebased on the current dev branch and added a "whatsnew" entry.

I noticed one of the CI checks fails with a bad import -- it doesn't seem too related to what I've changed

will be fixed by #39744

[AnjoMan]

Ok, i've updated the PR to reflect new feedback from @jbrockmendel and moved the 'whatsnew' entry to 1.4.0

[jbrockmendel]

LGTM cc @jreback

[jbrockmendel]

@AnjoMan can you merge master one more time

[AnjoMan]

@jbrockmendel i've updated the branch against pandas master

[jbrockmendel]

gentle ping @jreback

[jreback]

ok on board here. @AnjoMan one more rebase if you would and @mroeschke if any comments.

[AnjoMan]

i've rebased on top of c3d3357

[mroeschke]

Thanks for hanging in there @AnjoMan. Great work!

[jbrockmendel]

nice!

[AnjoMan]

thanks @jbrockmendel @jreback @mroeschke for all the feedback and help getting this together!