Skip to content

BUG: Make DTI/TDI/PI argsort match their underlying arrays #37965

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jreback merged 5 commits into from
Dec 17, 2020
Merged

BUG: Make DTI/TDI/PI argsort match their underlying arrays #37965

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
d8c504b
BUG: Make DTI/TDI/PI argsort match their underlying arrays
jbrockmendel Nov 19, 2020
474c18c
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Nov 21, 2020
5ba963c
non-ea compat
jbrockmendel Nov 21, 2020
f08ad69
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 2, 2020
2a47536
Merge branch 'master' of https://github.com/pandas-dev/pandas into bu…
jbrockmendel Dec 2, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion pandas/core/groupby/grouper.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -374,7 +374,8 @@ def _set_grouper(self, obj: FrameOrSeries, sort: bool = False):
# possibly sort
if (self.sort or sort) and not ax.is_monotonic:
# use stable sort to support first, last, nth
indexer = self.indexer = ax.argsort(kind="mergesort")
# TODO: why does putting na_position="first" fix datetimelike cases?
Copy link
Member

@mroeschke mroeschke Nov 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not really familiar with the history of NaTs with respect to resample. I would have assumed NaT would/should have been dropped by this point since we just exclude them from the result #13164.

I'm okay with this solution as a stop gap solution.

indexer = self.indexer = ax.argsort(kind="mergesort", na_position="first")
Copy link
Contributor

@jreback jreback Nov 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more. 

   def argsort(
        self,
        ascending: bool = True,
        kind: str = "quicksort",
        na_position: str = "last",

in arrays/base.py so this is strange

Copy link
Member Author

@jbrockmendel jbrockmendel Nov 20, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it must involve the fact that on older numpys, dt64.argsort used to put NaT at the beginning instead of the end

ax = ax.take(indexer)
obj = obj.take(indexer, axis=self.axis)

Expand Down
4 changes: 0 additions & 4 deletions pandas/core/indexes/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4743,10 +4743,6 @@ def argsort(self, *args, **kwargs) -> np.ndarray:
>>> idx[order]
Index(['a', 'b', 'c', 'd'], dtype='object')
"""
if needs_i8_conversion(self.dtype):
# TODO: these do not match the underlying EA argsort methods GH#37863
return self.asi8.argsort(*args, **kwargs)

# This works for either ndarray or EA, is overriden
# by RangeIndex, MultIIndex
return self._data.argsort(*args, **kwargs)
Expand Down
8 changes: 8 additions & 0 deletions pandas/tests/indexes/datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,14 @@


class DatetimeLike(Base):
def test_argsort_matches_array(self):
rng = self.create_index()
rng = rng.insert(1, pd.NaT)

result = rng.argsort()
expected = rng._data.argsort()
tm.assert_numpy_array_equal(result, expected)

def test_argmax_axis_invalid(self):
# GH#23081
msg = r"`axis` must be fewer than the number of dimensions \(1\)"
Expand Down