Preserve categorical column index in DataFrame.describe and handle pure iterators in cudf.pandas fallback

[galipremsagar]

Description

This PR fixes two categorical-related issues in cudf.pandas that surfaced in the pandas groupby categorical test suite:

1. DataFrame.describe dropped the source column index. describe builds its result with to_frame/concat, which discard the original column index (including its dtype and name, e.g. a CategoricalIndex). Since describe produces one column per described column in the same order, we now reassign the source column index onto the result.

2. Pure iterators were passed to the slow path unwrapped in fast_slow_proxy. Generators, map, filter, zip, and enumerate are resource-free but consumable iterators. On the fast path we continue to force a fallback (they can't be re-consumed). On the slow path, instead of handing the bare iterator straight to the slow library — which would pass wrapped proxies and lose type information (e.g. MultiIndex.from_product of a map over categoricals dropping the category dtype) — we now materialize and transform the elements, the same way list/tuple args are handled.

Together these unblock the following previously-xfailed pandas tests, whose entries are removed from the pandas-testing plugin:

• test_category_order_reducer (all/any over range)
• test_describe_categorical_columns
• test_observed[False]
• test_observed_codes_remap[False]
• test_observed_two_columns[False]
• test_unstack_categorical

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0924b7cc-18c1-4414-b5a6-008fd3108e2d

📥 Commits

Reviewing files that changed from the base of the PR and between 984353a and b0de552.

📒 Files selected for processing (3)

• python/cudf/cudf/core/dataframe.py
• python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
• python/cudf/cudf_pandas_tests/test_fast_slow_proxy.py

💤 Files with no reviewable changes (3)

• python/cudf/cudf_pandas_tests/test_fast_slow_proxy.py
• python/cudf/cudf/core/dataframe.py
• python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py

---

📝 Walkthrough

Summary by CodeRabbit

• Bug Fixes

  • DataFrame.describe() now correctly preserves source column index metadata (e.g., CategoricalIndex dtype).
  • GroupBy operations with categorical columns now pass additional edge-case tests.
  • GroupBy.all()/any() with as_index=False properly handles categorical grouping keys.

• Tests

  • Expanded test coverage for iterator and proxy behavior.

Walkthrough

DataFrame.describe is changed to preserve CategoricalIndex dtype/name metadata by restoring result.columns from the source data index after concatenation. _transform_arg gains a special-case branch for resource-free iterators (GeneratorType, map, filter, zip, enumerate) that forces the slow path on fast conversion and transforms elements lazily on slow conversion. Groupby tests and the known-fail registry are updated to reflect newly fixed and still-failing categorical cases.

Changes

DataFrame.describe column index preservation

Layer / File(s)	Summary
DataFrame.describe column index restoration python/cudf/cudf/core/dataframe.py	Single-column path assigns to_frame() into result; multi-column path assigns cudf.concat output to result and then overwrites result.columns with data_to_describe._data.to_pandas_index to preserve CategoricalIndex dtype/name.
Groupby reduction test and known-fail registry update python/cudf/cudf/tests/groupby/test_reductions.py, python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py	Adds parametrized test_groupby_all_any_as_index_false verifying categorical key columns are not coerced to boolean; removes resolved categorical groupby known-fail entries and updates the test_describe_categorical_columns reason from a TODO to a specific stack()/dtype-drop description.

Pure iterator handling in fast/slow proxy

Layer / File(s)	Summary
Pure iterator branch in _transform_arg python/cudf/cudf/pandas/fast_slow_proxy.py	Adds a special-case branch for GeneratorType, map, filter, zip, and enumerate before the general Iterator branch: raises on _fsproxy_fast to force the slow path, and returns a lazy generator over transformed elements on _fsproxy_slow.
Pure iterator proxy tests python/cudf/cudf_pandas_tests/test_fast_slow_proxy.py	Introduces _PURE_ITERATOR_FACTORIES/_PURE_ITERATOR_IDS constants and two parametrized tests: one asserts fast-path raises while the slow callable receives a bare iterator; the other asserts the slow-path result stays lazy and yields unwrapped slow objects.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Possibly related PRs

• rapidsai/cudf#22813: Fixes the underlying GroupBy._bool_reduce behavior for all()/any() with as_index=False that the new test_groupby_all_any_as_index_false test in this PR validates.

Suggested reviewers

• vyasr
• mroeschke
• bdice
• Matt711

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 30.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately and clearly describes the two main changes: preserving categorical column index in DataFrame.describe and handling pure iterators in cudf.pandas fallback.
Description check	✅ Passed	The description provides detailed context for both issues being fixed, explaining the problems and solutions with specific examples, and lists the unblocked tests.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[galipremsagar]

/merge