Fix all udf pytests

[galipremsagar]

Description

This PR fixes type assertion issues in udf tests.
pandas3:

== 766 failed, 77673 passed, 19475 skipped, 1551 xfailed in 493.01s (0:08:13) ==

This PR:

== 486 failed, 77953 passed, 19475 skipped, 1551 xfailed in 486.86s (0:08:06) ==

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[brandon-b-miller]

@galipremsagar did the output type of UDFs change in pandas 3?

[galipremsagar]

@galipremsagar did the output type of UDFs change in pandas 3?

That is not the real issue.

tm.assert_series_equal(left, right, check_dtype=False)
# Fails now

The core problem is that left is object dtype holding pd.NA values, while right is UInt64 (nullable integer). When pandas does the comparison with check_dtype=False, it casts UInt64 to float64 (turning NA into NaN), but pd.NA != NaN. Hence all these type-casts are needed.

[vyasr]

Should we be putting this cast into our actual apply implementation? Otherwise we're going to have the same problems in the pandas test suite and we can't simply cast in the tests there, we need to actually match the output.

[brandon-b-miller]

So will a any UDF that returns a null now return object dtype in pandas? I would think we'd get the nullable dtype with a true null.

[galipremsagar]

So will a any UDF that returns a null now return object dtype in pandas? I would think we'd get the nullable dtype with a true null.

There is no change in dtypes returned by pandas or cudf udfs. The change happened in assert_series_equal API.

In [1]: import pandas as pd

In [2]: pd.__version__
Out[2]: '3.0.1'

In [3]: import numpy as np

In [4]: l = pd.Series([np.nan])

In [5]: r = pd.Series([pd.NA])

In [7]: pd.testing.assert_series_equal(l, r, check_dtype=False)
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[7], line 1
----> 1 pd.testing.assert_series_equal(l, r, check_dtype=False)

    [... skipping hidden 2 frame]

File pandas/_libs/testing.pyx:53, in pandas._libs.testing.assert_almost_equal()

File pandas/_libs/testing.pyx:171, in pandas._libs.testing.assert_almost_equal()

File /raid/pgali/envs/cudfdev/lib/python3.13/site-packages/pandas/_testing/asserters.py:619, in raise_assert_detail(obj, message, left, right, diff, first_diff, index_values)
    616 if first_diff is not None:
    617     msg += f"\n{first_diff}"
--> 619 raise AssertionError(msg)

AssertionError: Series are different

Series values are different (100.0 %)
[index]: [0]
[left]:  [nan]
[right]: [<NA>]
At positional index 0, first diff: nan != <NA>

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: pd.__version__
Out[3]: '2.3.3'

In [4]: l = pd.Series([np.nan])

In [5]: r = pd.Series([pd.NA])

In [6]: pd.testing.assert_series_equal(l, r, check_dtype=False)
<ipython-input-6-ea3eeb97a7f8>:1: FutureWarning: Mismatched null-like values nan and <NA> found. In a future version, pandas equality-testing functions (e.g. assert_frame_equal) will consider these not-matching and raise.
  pd.testing.assert_series_equal(l, r, check_dtype=False)

Should we be putting this cast into our actual apply implementation? Otherwise we're going to have the same problems in the pandas test suite and we can't simply cast in the tests there, we need to actually match the output.

I agree that we should fix our apply to match pandas types, but in most cases pandas just returns object types for udf which means we will just have to raise MixedTypeError all across our udf's. I'd like to visit this angle if it is even worth raising in cudf or rather asking pandas to have them typed correctly once we get to there pandas test suite failures.

[vyasr]

In the interest of the cudf test suite becoming usable again on the pandas3 branch I'm OK with changing the test for now, but we're just kicking the can a short way down the road since we'll have to deal with the pandas test suite very soon.

The style check is failing, make sure to fix that before merging.

[brandon-b-miller]

I see whats happening here now, cuDF and pandas return the same thing but the equality comparison was going through non-nullable dtypes. This change seems fine to me.