Skip to content

Fix masked dtype equality in cudf.pandas so real_dtype == proxy_dtype holds - #22842

Merged
rapids-bot[bot] merged 3 commits into
NVIDIA:mainfrom
galipremsagar:cl_fix
Jun 11, 2026
Merged

Fix masked dtype equality in cudf.pandas so real_dtype == proxy_dtype holds#22842
rapids-bot[bot] merged 3 commits into
NVIDIA:mainfrom
galipremsagar:cl_fix

Conversation

@galipremsagar

@galipremsagar galipremsagar commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

Description

Fixes the test_reductions_2d_axis0 failures for masked dtypes (Boolean/Int*/UInt*) in the cudf.pandas pandas test suite.

Root cause

The test asserts dtype == expected.dtype, where:

  • dtype is a real pandas dtype pulled from pandas' internal NUMPY_INT_TO_DTYPE table, and
  • expected.dtype is a cudf.pandas proxy dtype.

This evaluates real_dtype.__eq__(proxy_dtype). pandas' ExtensionDtype.__eq__ does isinstance(other, type(self)). Since the proxy only subclasses the abstract ExtensionDtype (not the concrete pd.Int64Dtype), the isinstance check fails and equality returns False — even though the two dtypes are semantically identical. Note the asymmetry: proxy == real already worked (it dispatches through the proxy's own __eq__); only real == proxy failed.

Fix

Expose the concrete slow type as __class__ on each masked dtype proxy. isinstance consults __class__, so isinstance(proxy, pd.Int64Dtype) becomes True, equality proceeds, and since these masked dtypes have empty _metadata (()) it reduces to exactly that type check. type(proxy) is unaffected (it bypasses __class__), so the proxy machinery that keys off the real type is undisturbed.

Why this approach

  • Fix equality on the proxy, not the data. The real == proxy asymmetry occurs anywhere pandas internals hold real dtype instances (NUMPY_INT_TO_DTYPE, cast.py, the dtype registry, ...) and compare them against proxy-derived dtypes. Fixing the proxy repairs every such site at once; patching individual call sites would be whack-a-mole.
  • Proxying NUMPY_INT_TO_DTYPE is not viable. It's a plain dict whose values are real dtype instances built during pandas' own module execution; the module accelerator wraps namespace attributes, not the interior of data containers, so it is (and stays) un-proxied.
  • Scoped to dtype proxies only — not a blanket __class__ spoof. These dtypes are immutable, have empty _metadata, and are never reconstructed via self.__class__(...), so spoofing __class__ is safe. Doing it globally for stateful proxies (Series/DataFrame/arrays) would be dangerous: reconstruction via self.__class__(...) would yield native pandas objects, and proxy detection / copy / pickle all rely on the true class.

With equality fixed, all 28 previously-xfailed test_reductions_2d_axis0 parametrizations now pass, so their entries are removed from NODEIDS_THAT_FAIL.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Jun 10, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added Python Affects Python cuDF API. cudf.pandas Issues specific to cudf.pandas labels Jun 10, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python Jun 10, 2026
@galipremsagar galipremsagar changed the title fix Fix masked dtype equality in cudf.pandas so real_dtype == proxy_dtype holds Jun 10, 2026
@galipremsagar
galipremsagar marked this pull request as ready for review June 10, 2026 19:39
@galipremsagar
galipremsagar requested a review from a team as a code owner June 10, 2026 19:39
@galipremsagar
galipremsagar requested review from bdice and vyasr June 10, 2026 19:39
@galipremsagar galipremsagar added bug Something isn't working 3 - Ready for Review Ready for review by team non-breaking Non-breaking change labels Jun 10, 2026
@coderabbitai

coderabbitai Bot commented Jun 10, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: df6aa889-7be3-4982-a7e5-8fc6c9935129

📥 Commits

Reviewing files that changed from the base of the PR and between afd5497 and 4db5c14.

📒 Files selected for processing (1)
  • python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
🚧 Files skipped from review as they are similar to previous changes (1)
  • python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py

📝 Walkthrough

Summary by CodeRabbit

  • Bug Fixes
    • Fixed dtype equality comparisons between cuDF and pandas for nullable/masked Boolean, Integer (signed/unsigned) and Float dtypes.
    • Addressed tracked test failures for multi-dimensional reductions on masked/nullable arrays, improving compatibility and test stability.

Walkthrough

Proxy pandas dtypes in cuDF now expose the matching pandas dtype class via additional_attributes["__class__"] for Boolean, integer/unsigned integer, and float proxies. Corresponding test expectation entries for masked-array reductions were removed from the pandas-testing-plugin.

Changes

Dtype Proxy Equality Compatibility

Layer / File(s) Summary
Dtype proxy __class__ attribute mapping
python/cudf/cudf/pandas/_wrappers/pandas.py
BooleanDtype, signed/unsigned integer proxies (Int8/16/32/64Dtype, UInt8/16/32/64Dtype), and float proxies (Float32Dtype, Float64Dtype) add additional_attributes["__class__"] pointing to the corresponding pandas dtype types (e.g., pd.Int8Dtype, pd.Float32Dtype). Inline docstrings explain pandas' ExtensionDtype.__eq__ using isinstance(other, type(self)) checks.
Remove now-passing test expectations
python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
Removed previously recorded failing expectation entries for tests/extension/test_masked.py::TestMaskedArrays::test_reductions_2d_axis0[...] that mapped to "AssertionError".

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Suggested reviewers

  • vyasr
  • bdice
  • rjzamora
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes the main fix: enabling equality checks between real pandas dtypes and their cudf.pandas proxy equivalents for masked dtypes.
Description check ✅ Passed The description thoroughly explains the root cause (isinstance check asymmetry), the fix (exposing concrete dtype via class), and rationale, all directly related to the changeset.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands and usage tips.

@galipremsagar

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 8f8be17 into NVIDIA:main Jun 11, 2026
124 checks passed
@github-project-automation github-project-automation Bot moved this from In Progress to Done in cuDF Python Jun 11, 2026
abigalekim pushed a commit to abigalekim/cudf that referenced this pull request Jun 12, 2026
… holds (NVIDIA#22842)

Fixes the `test_reductions_2d_axis0` failures for masked dtypes (`Boolean`/`Int*`/`UInt*`) in the `cudf.pandas` pandas test suite.

### Root cause

The test asserts `dtype == expected.dtype`, where:
- `dtype` is a **real** pandas dtype pulled from pandas' internal `NUMPY_INT_TO_DTYPE` table, and
- `expected.dtype` is a `cudf.pandas` **proxy** dtype.

This evaluates `real_dtype.__eq__(proxy_dtype)`. pandas' `ExtensionDtype.__eq__` does `isinstance(other, type(self))`. Since the proxy only subclasses the abstract `ExtensionDtype` (not the concrete `pd.Int64Dtype`), the `isinstance` check fails and equality returns `False` — even though the two dtypes are semantically identical. Note the asymmetry: `proxy == real` already worked (it dispatches through the proxy's own `__eq__`); only `real == proxy` failed.

### Fix

Expose the concrete slow type as `__class__` on each masked dtype proxy. `isinstance` consults `__class__`, so `isinstance(proxy, pd.Int64Dtype)` becomes `True`, equality proceeds, and since these masked dtypes have empty `_metadata` (`()`) it reduces to exactly that type check. `type(proxy)` is unaffected (it bypasses `__class__`), so the proxy machinery that keys off the real type is undisturbed.

### Why this approach

- **Fix equality on the proxy, not the data.** The `real == proxy` asymmetry occurs anywhere pandas internals hold real dtype instances (`NUMPY_INT_TO_DTYPE`, `cast.py`, the dtype registry, ...) and compare them against proxy-derived dtypes. Fixing the proxy repairs every such site at once; patching individual call sites would be whack-a-mole.
- **Proxying `NUMPY_INT_TO_DTYPE` is not viable.** It's a plain dict whose values are real dtype instances built during pandas' own module execution; the module accelerator wraps namespace attributes, not the interior of data containers, so it is (and stays) un-proxied.
- **Scoped to dtype proxies only — not a blanket `__class__` spoof.** These dtypes are immutable, have empty `_metadata`, and are never reconstructed via `self.__class__(...)`, so spoofing `__class__` is safe. Doing it globally for stateful proxies (Series/DataFrame/arrays) would be dangerous: reconstruction via `self.__class__(...)` would yield native pandas objects, and proxy detection / copy / pickle all rely on the true class.

With equality fixed, all 28 previously-xfailed `test_reductions_2d_axis0` parametrizations now pass, so their entries are removed from `NODEIDS_THAT_FAIL`.

Authors:
  - GALI PREM SAGAR (https://github.com/galipremsagar)

Approvers:
  - Matthew Roeschke (https://github.com/mroeschke)

URL: NVIDIA#22842
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

3 - Ready for Review Ready for review by team bug Something isn't working cudf.pandas Issues specific to cudf.pandas non-breaking Non-breaking change Python Affects Python cuDF API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants