Fix groupby and single-column to_numpy pandas-compatibility bugs

[galipremsagar]

Description

Fixes several cudf behaviors to match pandas, enabling the full tests/groupby/test_groupby_dropna.py pandas suite (and a few related tests) under cudf.pandas:

• DataFrame.to_numpy()/.to_cupy()/.values on a single nullable-integer column now return NaN (float64) instead of leaking the integer null sentinel, matching Series and the multi-column path.
• groupby.idxmin/idxmax return index labels (via gather) instead of positions.
• groupby.ngroup respects sort and dropna.
• groupby.apply uses the UDF-returned index for the inner MultiIndex level instead of the original row positions.
• GroupBy.indices returns numpy arrays in pandas-compatible mode.
• groupby null keys (pandas-compatible mode): NaN/null collapse into a single group, and an all-null object key yields a float64 NaN label.

Removes the corresponding xfail/skip entries from the pandas testing plugin.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[galipremsagar]

/okay to test 1b463e4

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Promotes single-column nullable integer arrays to float64 when dtype is unset and nulls exist; updates many GroupBy behaviors (iteration order, indices construction, cumcount, agg finalization, idxmin/idxmax label mapping, ngroup determinism, pct_change, key normalization) and adjusts tests and the pandas-testing-plugin registry.

Changes

GroupBy pandas-compatibility fixes and frame array conversion

Layer / File(s)	Summary
Array conversion dtype promotion for nullable integers python/cudf/cudf/core/frame.py	In Frame._to_array, when converting a single nullable integer column without explicit dtype, if the column contains nulls the integer to_dtype is promoted to float64 so nulls can be represented as NaN.
GroupBy iteration, indices, cumcount, pct_change python/cudf/cudf/core/groupby/groupby.py	GroupBy.__iter__ reorders groups for sort=False to match pandas' earliest-input-row ordering; GroupBy.indices refactors split construction; cumcount forwards dropna=self._dropna; pct_change threads sort and dropna into its intermediate groupby.
Aggregation finalization (scan vs reduction) python/cudf/cudf/core/groupby/groupby.py	agg treats scan aggregations specially (returns via _mimic_pandas_order) and applies reset_index() for non-scan reductions only when as_index=False.
GroupBy reduction ops: idxmin/idxmax label mapping and ngroup python/cudf/cudf/core/groupby/groupby.py	_wrap_idxmin_idxmax maps libcudf positional idxmin/idxmax results to source index labels with null-safety and skipna validation; GroupBy.ngroup deterministically computes group ids using unique key order, explicit null-group handling, and ascending/descending sequencing.
Apply post-processing MultiIndex reconstruction python/cudf/cudf/core/groupby/groupby.py	apply reconstructs inner MultiIndex level from result.index._column for concat-produced outputs.
Grouping key normalization python/cudf/cudf/core/groupby/groupby.py	_Grouping replaces all-null object-typed key columns with float64 NaN placeholders and applies nans_to_nulls() to all key columns before storing them.
Test updates: ngroup sort param and xfail removal python/cudf/cudf/tests/groupby/test_ngroup.py, python/cudf/cudf/tests/groupby/test_reductions.py, python/cudf/cudf/tests/groupby/test_attributes.py	test_groupby_ngroup gained a sort parameter and applies sort/ascending consistently; test_groupby_level_zero and test_groupby_series_level_zero removed the request fixture and associated xfail logic; grouping tests now parametrize sort.
Pandas-testing-plugin registry updates python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py	Replaces and retargets several groupby-related registry entries (dropna/multi-index/grouping expectations), retunes flaky test mappings for copy/view groupby tests, and adds GPU/numba-related registry entries.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• rapidsai/cudf#22621: Modifies Frame._to_array dtype/null-handling logic, related to single-column nullable-integer promotion behavior.

Suggested reviewers

• vyasr
• Matt711

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 29.41% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title directly and clearly describes the main changes: fixing groupby and single-column to_numpy pandas-compatibility bugs, which aligns with all substantial changes in the PR.
Description check	✅ Passed	The description is well-detailed and directly related to the changeset, covering the key fixes made across groupby operations, to_numpy behavior, and test updates.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[vyasr]

This change broke one of cudf's own tests.

[galipremsagar]

This change broke one of cudf's own tests.

Fixed the failures.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@python/cudf/cudf/core/groupby/groupby.py`:
- Around line 3898-3915: The new unconditional key normalization in the
_Grouping block alters GroupBy behavior; only perform the
object-key-to-float-NaN and nans_to_nulls normalization when pandas-compatible
mode is enabled (the same condition used in GroupBy.__init__ that gates
obj.nans_to_nulls()). Wrap the loop that mutates self._key_columns (the
normalized list creation and replacement) behind the pandas compatibility check
(e.g., the existing mode.pandas_compatible/GroupBy __init__ condition) so that
outside pandas_compatible mode keys are left untouched.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: c1355959-6fc0-4d27-b820-996dfb1fa457

📥 Commits

Reviewing files that changed from the base of the PR and between 21548dc and c7426fb.

📒 Files selected for processing (3)

• python/cudf/cudf/core/groupby/groupby.py
• python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
• python/cudf/cudf/tests/groupby/test_reductions.py

[galipremsagar]

/okay to test a63bdc4

[galipremsagar]

/okay to test d21dffc

[galipremsagar]

/okay to test 39c22d0

[galipremsagar]

/merge