API/ENH: overhaul/unify/improve .unique

[h-vetinari]

The state of the various flavours of .unique as of v0.23:

• [pd/Series/Index].unique does not have keep-kwarg
• Series.unique returns array, Series.drop_duplicates returns Series. Returning a plain np.ndarray is quite unusual for a Series method, and furthermore the differences between these closely-related methods are confusing from a user perspective, IMO
• same point for Index
• DataFrame.unique does not exist, but is a much more natural candidate (from the behaviour of numpy, resp. Series/Index) than .drop_duplicates
• pd.unique chokes on 2-dimensional data
• no return_inverse-kwarg for any of the .unique variants; see API: provide a better way of doing np.unique(return_inverses=True) #4087 (milestoned since 0.14), ENH: adding .unique() to DF (or return_inverse for duplicated) #21357

I originally wanted to add df.unique(..., return_inverse=True|False) for #21357, but got directed to add it to duplicated instead. After slow progress over 3 months in #21645 (PR essentially finished since 2), @jorisvandenbossche brought up the - justified (IMO) - feedback that:

I think my main worry is that we are adding a return_inverse keyword which actually does not return the inverse for that function (it does return the inverse for another function), and that it is in name similar to numpy's keyword, but in usage also different.

and

[...] it might make sense to add this to pd.unique / Series.unique as well? (not necessarily at the same time; or might actually be an easier starter)

This prompted me to have another look at the situation with .unique, and I found the list of the above inconsistencies. To resolve them, I suggest to:

• Change return type for [Series/Index].unique to be same as caller (deprecation cycle by introducing raw=None which at first defaults to True?)
• Add keep-kwarg to [Series/Index].unique (make .unique a wrapper around .drop_duplicates?)
• Add df.unique (as thin wrapper around .drop_duplicates?)
• Add keep-kwarg to pd.unique and dispatch to DataFrame/Series/Index as necessary
• Add return_inverse-kwarg to all of them (and add to EA interface); under the hood by exposing the same kwarg to duplicated and drop_duplicates as well
• (something for later) solve BUG: df.duplicated treats None as np.nan in object columns #21720 (treatment of np.nan/None in df.duplicated inconsistent vs. Series behaviour)

Each point is essentially self-contained and independent of the others, but of course they make more sense together.

[jreback]

there are multiple issues about changing .unique (for quite some time)

pls show the references

[jreback]

-1 in pd.unique being 2-d as this will make an already complicated function much more so

[h-vetinari]

@jreback

there are multiple issues about changing .unique (for quite some time)
pls show the references

I tried best I could. Searching open issues with "unique" yields 185 results which are at first glance completely unrelated - so I'll admit I didn't comb through all of those. (https://github.com/pandas-dev/pandas/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+unique)

Searching for "df.unique" or "dataframe.unique" yielded only #21357, and now also this issue.

[h-vetinari]

-1 in pd.unique being 2-d as this will make an already complicated function much more so

if df.unique wraps df.drop_duplicates as suggested, then it would be very easy to dispatch to the different types, and work with existing code at almost no extra cost.

Also, the implementation of pd.unique is not very complicated at all - just dispatching to hashtables (https://github.com/pandas-dev/pandas/blob/v0.23.4/pandas/core/algorithms.py#L358-L378).

[h-vetinari]

there are multiple issues about changing .unique (for quite some time)
pls show the references

Went over the list again and found one related #15442, and one partly related #13984 and two barely related: #18291 #19595

[jreback]

there was tremendous discussion on this - what should Series.unique return - rather than rehash much better to search for it - follow the links in the first issue

[h-vetinari]

Changing Index.unique() to return index: #4126 #13395 (where there's the discussion about what Series.unique() should return), finished in #13979.

rather than rehash much better to search for it

For someone who hasn't been through these discussions through the years, it's nigh impossible to find. Not going to go through thousands of closed issues to find some orthogonal discussions that may or not be hiding in there.

there was tremendous discussion on this - what should Series.unique return

This is but one aspect of this issue (in particular, the rest are independent of it). And still, most of the discussion in #13395 was what index Series.unique() would have - if we postulate unique = drop_duplicates, that question already has an answer.

[jorisvandenbossche]

@h-vetinari Thanks for all the discussion

I think the main previous discussions on the return value of unique have been about having an index vs array vs categorical as output (so all non-indexed objects). So that is a bit different than the issue raised here.

But thanks for looking through the older issues! That is indeed not always easy (even for a core dev that might have participated in them ..)

---

Some replies to the inconsistencies / solutions:

Change return type for [Series/Index].unique to be same as caller (deprecation cycle by introducing raw=None which at first defaults to True?)

Personally, I think that boat has sailed (for Series, Index already returns an Index). I don't really see the added value in having our users go through such a deprecation cycle for such a core function.

Also, returning a Series, i.e. returning an indexed object, gives you more complexity, which is what drop_duplicates is dealing with. Adding this to unique would make them essentially the same. This might be a good thing, but now at least they each serve a slightly different purpose.

[pd/Series/Index].unique does not have keep-kwarg

Which is not relevant as long as the the return value is non an indexed object?

DataFrame.unique does not exist, but is a much more natural candidate (from the behaviour of numpy, resp. Series/Index) than .drop_duplicates

In naming yes, but what is the "natural" behaviour? If you consider the difference between the current Series.drop_duplicates and Series.unique, then DataFrame.drop_duplicates is the consistent equivalent with Series.drop_duplicates I think?

no return_inverse-kwarg for any of the .unique variants; see #4087 (milestoned since 0.14), #21357

As I said in the other thread, I think there would be not much disagreement on adding return_inverse to the current unique methods (but that is only my opinion of course).

pd.unique chokes on 2-dimensional data

But what would you want it to do? Work on the flattened values, or on a certain axis? (for np.unique they recently added an axis argument)

Add keep-kwarg to pd.unique and dispatch to DataFrame/Series/Index as necessary

For keep the same comment as above. And currently it is the Series/Index version that dispatches to pd.unique (if the underlying values itself have no unique method), what would be the advantage of changing it around?

[jorisvandenbossche]

So maybe the main question for me is this (limiting it to Series methods here): Series.unique and Series.drop_duplicates are now two distinct functions, doing slightly different things. Is this good or not?

@h-vetinari's proposal is to unify them, but now they serve an overlapping but still different purpose, and it can also be fine to have two different methods for that (eg unique does not need to care about the keep argument)
Personally, I think it is fine to have both.

[h-vetinari]

@jorisvandenbossche
Thanks for the detailed response!

So maybe the main question for me is this (limiting it to Series methods here): Series.unique and Series.drop_duplicates are now two distinct functions, doing slightly different things. Is this good or not?

I would argue that they do exactly the same thing (up to and including keep; more on that below), and having them do it in slightly different ways is inconsistent, confusing and error-prone from a user-perspective.

[pd/Series/Index].unique does not have keep-kwarg

Which is not relevant as long as the the return value is non an indexed object?

No, it IS relevant even now. Pandas strongly advertises that [pd/Series/Index].unique does not sort, and then keep='first'|'last' makes a real difference. Whenever the order of a Series matters (especially, but not only, with a DatetimeIndex) there is a clear distinction between keeping the first and last occurrence, even if the output does not have an index itself.

pd.unique chokes on 2-dimensional data

But what would you want it to do? Work on the flattened values, or on a certain axis? (for np.unique they recently added an axis argument)

Add axis=0 to the signature and raise NotImplementedError if axis!=0. Then pd.unique can dispatch to the respective .unique method of pandas objects (and default to standard treatment for other objects). All methods would continue to take the row-uniques.

And currently it is the Series/Index version that dispatches to pd.unique (if the underlying values itself have no unique method), what would be the advantage of changing it around?

WRT that, I found another inconsistency, pd.unique(idx) != idx.unique():

>>> idx = pd.Index([1, 2, 3])
>>> idx.unique()
Int64Index([1, 2, 3], dtype='int64')
>>> pd.unique(idx)
array([1, 2, 3], dtype=int64)

Finally,

Personally, I think that boat has sailed (for Series, Index already returns an Index). I don't really see the added value in having our users go through such a deprecation cycle for such a core function.

If the change was ok for Index in 0.19 two years ago, and another such central function (groupby.apply) was changed in a similar way in 0.23 just months ago, then IMO users will be able to deal with it without issue, and the long-term benefits far outweigh that single transition hump.

[jorisvandenbossche]

BTW, @h-vetinari in case you would be interested, we are having a dev hangout tomorrow: https://mail.python.org/pipermail/pandas-dev/2018-September/000830.html

[h-vetinari]

@jorisvandenbossche
Thanks for the invitation! I've got a deadline on Friday, so I'm skeptical whether I'll make it. Hoping I could get a go-ahead on #22812 though. ;-)

[jorisvandenbossche]

@h-vetinari ah, that wasn't on the agenda yet, will add it to the list of potential topics to discuss, but can't promise we will get to it

[TomAugspurger]

Whenever the order of a Series matters (especially, but not only, with a DatetimeIndex) there is a clear distinction between keeping the first and last occurrence, even if the output does not have an index itself.

What do you mean by this? I don't see how the output would differ if just an array is being returned.