Fix DataFrame.stack/unstack pandas incompatibilities - #23255
Closed
galipremsagar wants to merge 10 commits into
Closed
Fix DataFrame.stack/unstack pandas incompatibilities#23255galipremsagar wants to merge 10 commits into
galipremsagar wants to merge 10 commits into
Conversation
github-actions
Bot
added
Python
galipremsagar
added
bug
Contributor
Author
|
/okay to test e02681b |
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughUpdates cuDF column conversion, MultiIndex metadata preservation, level-aware sorting, stack/unstack ordering, pivot integer promotion, and pandas compatibility tests. ChangesReshape and metadata compatibility
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
📝 WalkthroughSummary by CodeRabbit
WalkthroughChangesReshape and metadata compatibility
Proxy descriptor isolation
Estimated code review effort: 4 (Complex) | ~60 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
python/cudf/cudf/core/column/numerical.py (1)
861-909: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winMove the equivalent-dtype short-circuit above the float→nullable-float branch
The current ordering makes thenans_to_nulls()fast path unreachable for plain NumPy float columns cast to pandas nullable floats; the earlier branch returns first. Reordering restores the intended zero-copy path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf/cudf/core/column/numerical.py` around lines 861 - 909, Move the dtype-equivalence short-circuit in the numerical column cast flow before the float-to-pandas-nullable-float branch. Ensure plain NumPy float columns targeting an equivalent nullable-float dtype reach the existing nans_to_nulls() handling and ColumnBase.create path, while preserving the other conversion checks and return behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/cudf/cudf/core/column_accessor.py`:
- Around line 368-394: Update the round-trip validation in the dtype restoration
loop around result.levels and self._level_dtypes to compare the cast-back index
using Index.equals. Preserve the existing cast exception handling and only
assign cast_lvl when the Index.equals comparison confirms the values, including
missing entries, are unchanged.
In `@python/cudf/cudf/core/indexed_frame.py`:
- Around line 2876-2899: Update the MultiIndex column sorting branch in the
level-based axis=1 path to preserve per-level ascending values instead of
reducing ascending to a single reverse boolean. Apply the same iterable-aware
ordering semantics used by the row-axis _get_sorted_inds path while retaining
scalar ascending behavior and the existing key_order/sort_remaining logic.
In `@python/cudf/cudf/core/reshape.py`:
- Around line 1213-1222: The pivot path in
python/cudf/cudf/core/reshape.py:1213-1222 must pass
promote_ints_on_missing=True when calling _pivot. In the pivot_table path at
python/cudf/cudf/core/reshape.py:1741-1758, pass
promote_ints_on_missing=(fill_value is None) to _unstack, preserving integer
types when missing combinations are filled by default while skipping promotion
when fill_value is provided, including 0.
---
Outside diff comments:
In `@python/cudf/cudf/core/column/numerical.py`:
- Around line 861-909: Move the dtype-equivalence short-circuit in the numerical
column cast flow before the float-to-pandas-nullable-float branch. Ensure plain
NumPy float columns targeting an equivalent nullable-float dtype reach the
existing nans_to_nulls() handling and ColumnBase.create path, while preserving
the other conversion checks and return behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: ac36fa3c-ec81-46c6-bd20-89780580e270
📒 Files selected for processing (11)
python/cudf/cudf/core/column/column.pypython/cudf/cudf/core/column/numerical.pypython/cudf/cudf/core/column_accessor.pypython/cudf/cudf/core/dataframe.pypython/cudf/cudf/core/groupby/groupby.pypython/cudf/cudf/core/indexed_frame.pypython/cudf/cudf/core/multiindex.pypython/cudf/cudf/core/reshape.pypython/cudf/cudf/pandas/fast_slow_proxy.pypython/cudf/cudf/pandas/scripts/pandas-testing-plugin.pypython/cudf/cudf/tests/reshape/test_unstack.py
There was a problem hiding this comment.
Actionable comments posted: 7
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
python/cudf/cudf/core/groupby/groupby.py (1)
1214-1240: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick winPreserve
multiindexin the empty-columns path. Whenlen(data) == 0, this branch still forcesmultiindex=False, so a frame with MultiIndex columns and no value columns will rebuild as a flat emptyIndexinstead of an emptyMultiIndex. Mirror the adjacent branch and passself.obj._data.multiindexhere too.♻️ Proposed fix
if len(data) == 0 and not multilevel and self.obj.ndim == 2: data = ColumnAccessor( data, - multiindex=False, + multiindex=self.obj._data.multiindex, level_names=self.obj._data.level_names, rangeindex=self.obj._data.rangeindex, label_dtype=self.obj._data.label_dtype, level_dtypes=self.obj._data.level_dtypes, )🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@python/cudf/cudf/core/groupby/groupby.py` around lines 1214 - 1240, Update the empty-columns branch in the groupby result construction to pass self.obj._data.multiindex to ColumnAccessor instead of forcing multiindex=False. Preserve the existing source column metadata and behavior for non-empty data and the adjacent branch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/cudf/cudf/core/column_accessor.py`:
- Around line 368-394: Update the round-trip guard in the level restoration
logic around result.set_levels to use Index.equals() for the cast_lvl comparison
instead of elementwise ==. Preserve the existing cast exception handling and
only restore the level dtype when the missing-aware equality check confirms a
lossless round trip.
In `@python/cudf/cudf/core/indexed_frame.py`:
- Around line 2893-2897: Update the label-sorting logic in the indexed-frame
sorting method around the self._column_names branch to consume iterable
ascending values per key level and honor na_position for null labels. Replace
the single reverse=not ascending sort with a null-aware stable multi-key sort,
preserving correct precedence and direction for each level.
- Around line 2882-2887: Update the local _level_number helper used to build
key_order so negative integer levels are normalized only when they are within
the valid range; reject values below -nlevels and nonnegative values at or above
nlevels before indexing or sorting. Preserve valid negative and nonnegative
level behavior while raising the established invalid-level exception for
out-of-range inputs.
In `@python/cudf/cudf/core/reshape.py`:
- Around line 1011-1014: Update the column-level name construction in the
unstack reshape flow to use the public axis names from columns_labels, via its
names/axis-name API, instead of the internal _column_names keys. Preserve
col_accessor.level_names and ensure duplicate level names remain unchanged and
correctly represented for positional multi-level unstack.
- Around line 1433-1443: Update the MultiIndex restoration logic around
full_level, pdi, and new_codes so missing labels represented by -1 do not
prevent restoring unused categories from full_level. Preserve the existing valid
codes, map missing entries to the appropriate missing-code representation, and
rebuild result._data.to_pandas_index with the complete level metadata even when
the unstacked level contains NA.
- Line 1757: Update the _unstack call in the pivot-table path so
promote_ints_on_missing is enabled only when fill_value is not None, preserving
pandas-style float promotion for sparse integer results without a fill value.
Add a regression test covering a sparse integer sum/min/max pivot with
fill_value=None and verifying the resulting nullable behavior.
---
Outside diff comments:
In `@python/cudf/cudf/core/groupby/groupby.py`:
- Around line 1214-1240: Update the empty-columns branch in the groupby result
construction to pass self.obj._data.multiindex to ColumnAccessor instead of
forcing multiindex=False. Preserve the existing source column metadata and
behavior for non-empty data and the adjacent branch.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 48c25aa3-4729-4173-8aaa-24e63c11f708
📒 Files selected for processing (12)
python/cudf/cudf/core/column/column.pypython/cudf/cudf/core/column/numerical.pypython/cudf/cudf/core/column_accessor.pypython/cudf/cudf/core/dataframe.pypython/cudf/cudf/core/groupby/groupby.pypython/cudf/cudf/core/indexed_frame.pypython/cudf/cudf/core/multiindex.pypython/cudf/cudf/core/reshape.pypython/cudf/cudf/pandas/fast_slow_proxy.pypython/cudf/cudf/pandas/scripts/pandas-testing-plugin.pypython/cudf/cudf/tests/reshape/test_concat.pypython/cudf/cudf/tests/reshape/test_unstack.py
| else: | ||
| to_unstack.append(name) | ||
| table = agged.unstack(to_unstack) | ||
| table = _unstack(agged, to_unstack, promote_ints_on_missing=False) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
rg -n -C3 \
'pivot_table|fill_value|aggfunc.*(sum|min|max)' \
python/cudf/cudf/tests -g 'test_*.py'Repository: rapidsai/cudf
Length of output: 50369
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the relevant implementation around the cited lines.
sed -n '1710,1785p' python/cudf/cudf/core/reshape.py
echo
echo '--- pivot_table tests ---'
rg -n -C 3 'pivot_table|fill_value|sum|min|max' python/cudf/cudf/tests -g 'test_*.py'Repository: rapidsai/cudf
Length of output: 50370
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo '--- reshape implementation ---'
sed -n '1730,1775p' python/cudf/cudf/core/reshape.py
echo
echo '--- pivot table tests focused ---'
sed -n '1,220p' python/cudf/cudf/tests/reshape/test_pivot_table.py
echo
echo '--- find fill_value=None pivot tests ---'
rg -n -C 2 'fill_value\s*=\s*None|fill_value=None|pivot_table.*None' python/cudf/cudf/tests/reshape/test_pivot_table.pyRepository: rapidsai/cudf
Length of output: 4203
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 4 'def _unstack|promote_ints_on_missing' python/cudf/cudf/core/reshape.py python/cudf/cudf -g '*.py'Repository: rapidsai/cudf
Length of output: 10552
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n -C 6 'def _unstack|promote_ints_on_missing' python/cudf/cudf/core/reshape.pyRepository: rapidsai/cudf
Length of output: 4169
Gate integer promotion on whether missing cells are filled
_unstack(..., promote_ints_on_missing=False) is still used when fill_value=None, so sparse integer sum/min/max pivot tables can keep nullable integer dtypes instead of pandas-style floats. Make the flag depend on fill_value is not None, and add a sparse integer fill_value=None case.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/cudf/cudf/core/reshape.py` at line 1757, Update the _unstack call in
the pivot-table path so promote_ints_on_missing is enabled only when fill_value
is not None, preserving pandas-style float promotion for sparse integer results
without a fill value. Add a regression test covering a sparse integer
sum/min/max pivot with fill_value=None and verifying the resulting nullable
behavior.
Contributor
Author
There was a problem hiding this comment.
Fixed in b769725 — promote_ints_on_missing=fill_value is None, matching pandas (verified: pandas 3.0.3 gives float64 for a sparse int sum with fill_value=None and int64 with fill_value=0). Added test_pivot_table_sparse_int_fill_value covering sum/min/max × fill_value None/0.
Contributor
Author
|
Merged |
Contributor
Author
|
/okay to test b769725 |
|
Note We couldn't fetch the incremental changes for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@python/cudf/cudf/core/reshape.py`:
- Around line 1076-1078: Update the MultiIndex unstack flow around remaining,
encode_side, and _pivot to special-case a one-level MultiIndex. Validate the
requested level, flatten that sole index level, and route through the existing
single-index path so the result is the expected stacked Series rather than an
empty DataFrame; preserve current behavior for multi-level indexes.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 1073532d-d61f-49ef-91c9-53a9aac17b99
📒 Files selected for processing (15)
python/cudf/cudf/core/column/column.pypython/cudf/cudf/core/column/numerical.pypython/cudf/cudf/core/column_accessor.pypython/cudf/cudf/core/dataframe.pypython/cudf/cudf/core/groupby/groupby.pypython/cudf/cudf/core/indexed_frame.pypython/cudf/cudf/core/multiindex.pypython/cudf/cudf/core/reshape.pypython/cudf/cudf/pandas/fast_slow_proxy.pypython/cudf/cudf/pandas/scripts/pandas-testing-plugin.pypython/cudf/cudf/tests/dataframe/test_np_ufuncs.pypython/cudf/cudf/tests/reshape/test_concat.pypython/cudf/cudf/tests/reshape/test_pivot_table.pypython/cudf/cudf/tests/reshape/test_unstack.pypython/cudf/cudf/tests/series/test_np_ufuncs.py
🚧 Files skipped from review as they are similar to previous changes (8)
- python/cudf/cudf/tests/dataframe/test_np_ufuncs.py
- python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
- python/cudf/cudf/tests/series/test_np_ufuncs.py
- python/cudf/cudf/core/column_accessor.py
- python/cudf/cudf/pandas/fast_slow_proxy.py
- python/cudf/cudf/core/indexed_frame.py
- python/cudf/cudf/core/column/column.py
- python/cudf/cudf/tests/reshape/test_concat.py
Contributor
Author
|
CI failures investigated and fixed in 6dab03e. Three root causes:
Plugin reconciliation: this PR's reshape/metadata fixes turned 61 existing entries into strict XPASSes in the last CI run — 49 verified passing in isolation are pruned; the 12 that only pass/fail depending on suite ordering keep their entries, plus 5 stack/unstack entries re-added that regress to main's ordering behavior with the proxy revert. |
Contributor
Author
|
/okay to test 44184b0 |
Contributor
Author
|
Fixed the pandas-tests CI failures (33 = 28 + 5, both plugin bookkeeping — no code changes needed):
Net plugin state vs main: 79 entries removed (58 stack/unstack + 21 neighbors), 0 added; the 3 remaining stack/unstack entries are the documented inherent ones. Local re-validation: classic reshape suite green (4652 tests); pandas-tests |
Contributor
Author
|
/okay to test 69e8bb9 |
Contributor
Author
|
/okay to test 963202f |
Fixes 60 pandas unit-test failures in tests/frame/test_stack_unstack.py under cudf.pandas (64 -> 4; the remaining 4 assert BlockManager internals or monkeypatch pandas' private _Unstacker). DataFrame.stack: * Resolve levels positionally: integer column-level names no longer collide with level positions in get_level_values lookups. * Validate out-of-bounds integer levels (IndexError) and duplicated level names (ValueError) like pandas. * Build the stacked level keys from the column MultiIndex's own levels/codes so level dtypes survive (int64 levels with missing entries no longer upcast to float64; categorical levels stay categorical through the tile step). * Emit stacked keys in appearance order like pandas, replacing the argsort-based reordering that misaligned data for non-involution column permutations and NaN keys. * Attach pandas-faithful levels/codes to the result index eagerly (reusing the original MultiIndex's levels) so a later unstack restores the original row/column order; legacy dropna keeps them. unstack/_pivot: * Order result rows/columns by the removed level's codes (level order, missing keys first) instead of sorted values, matching pandas. * Propagate the source frame's column-axis level names instead of hardcoding None; fixes the 'Length of names must match number of levels' failure for frames with MultiIndex columns. * Promote integer columns to float64 when unstack introduces missing cells (pandas block semantics); pivot_table/crosstab opt out since they fill missing cells afterwards. * Preserve unused categories of the removed level in the result's column levels (pandas GH 17845). * Validate flat-index level (KeyError) and duplicated index names (ValueError) like pandas. Supporting fixes: * ColumnAccessor: NaN-containing labels now match under pandas' all-NaNs-equal semantics; to_pandas_index restores recorded per-level dtypes when the cast round-trips losslessly; the primed/cached pandas columns index survives accessor copies so explicit unsorted level layouts are not lost on fast-to-slow conversion. * MultiIndex: lazy codes/levels materialization sorts levels (pandas-canonical for per-row-value construction). * sort_index(axis=1) now honors level= and sort_remaining=. * GroupBy.agg keeps MultiIndex columns for MultiIndex-column sources. * NumericalColumn.as_numerical_column no longer mutates the column dtype in place on equal-pylibcudf-type casts. * Bool columns with nulls convert to pandas with np.nan (not None) in pandas-compatible mode. * cudf.pandas: do not bake one instance's transfer-blocking state into the class-level cached _MethodProxy (order-dependent test poisoning). Removes the 60 fixed xfail entries from the pandas-testing plugin and un-xfails now-passing cudf unstack tests with categorical indexes.
Bool was the only dtype whose default-mode to_pandas emitted None for missing values; strings, categoricals and ints already produce nan and datetimes produce NaT, and pandas itself never places None in an upcast-to-object bool column. Dropping the mode.pandas_compatible gate makes the conversion consistent across dtypes and with pandas. This also makes concat of bool and float frames match pandas' float coercion; the corresponding strict xfail in test_concat.py now passes and is removed.
The ufunc tests masked expected bool results with None to match the old to_pandas conversion, with a comment asking whether it should be np.nan instead; it is now.
* GroupBy.agg: preserve MultiIndex columns in the empty-columns branch. * ColumnAccessor.to_pandas_index: use missing-aware Index.equals for the lossless-cast round-trip guard. * sort_index(axis=1, level=...): validate out-of-range integer levels (matching pandas' IndexError), honor per-level ascending lists, and place missing labels per na_position via a null-aware stable multi-key sort. * MultiIndex._level_index_from_level: reject still-negative levels after normalization instead of silently indexing from the end; align the error message with pandas. * unstack: use the public level names for the result's column levels and keep restoring unused categories when the removed level contains NA (-1 codes for NA labels are pandas' canonical representation). * pivot_table: promote integer values to float64 like pandas when missing cells are left unfilled (fill_value=None); keep the integer dtype when fill_value is provided. Add a regression test.
Contributor
Author
|
/okay to test 17e81ef |
…ts-stack-unstack # Conflicts: # python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
Contributor
Author
|
/okay to test d50efcc |
Contributor
|
This is a pretty large PR. Any chance you can split it up? |
…ts-stack-unstack # Conflicts: # python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py
This was referenced Jul 21, 2026
Contributor
Author
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
…sts (#23364) Split out of #23255 (1/6). `NumericalColumn.as_numerical_column` short-circuits casts between equivalent dtypes (same pylibcudf type, e.g. `float64` → `Float64`), but implemented the shortcut by assigning the target dtype onto `self._dtype` in place. The column object is shared with the caller's Series/DataFrame, so the *source* object silently changed dtype as a side effect of the cast. This returns a fresh column over the same pylibcudf data instead (`nans_to_nulls` first for float → masked casts), and adds a classic regression test. Fixes 5 pandas-tests (`test_stack_nullable_dtype[*]`, `test_loc_set_nan_in_categorical_series[Float64]`, `test_assert_series_equal_extension_dtype_mismatch`, `test_assert_frame_equal_extension_dtype_mismatch`); their xfail entries are removed. Attribution verified by running the node ids against an isolated build containing only this fix (they pass) and a clean build (they fail). Independent of the other #23255 split PRs; can merge in any order. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #23364
4 tasks
davidwendt
pushed a commit
to wjxiz1992/cudf
that referenced
this pull request
Jul 21, 2026
…sts (NVIDIA#23364) Split out of NVIDIA#23255 (1/6). `NumericalColumn.as_numerical_column` short-circuits casts between equivalent dtypes (same pylibcudf type, e.g. `float64` → `Float64`), but implemented the shortcut by assigning the target dtype onto `self._dtype` in place. The column object is shared with the caller's Series/DataFrame, so the *source* object silently changed dtype as a side effect of the cast. This returns a fresh column over the same pylibcudf data instead (`nans_to_nulls` first for float → masked casts), and adds a classic regression test. Fixes 5 pandas-tests (`test_stack_nullable_dtype[*]`, `test_loc_set_nan_in_categorical_series[Float64]`, `test_assert_series_equal_extension_dtype_mismatch`, `test_assert_frame_equal_extension_dtype_mismatch`); their xfail entries are removed. Attribution verified by running the node ids against an isolated build containing only this fix (they pass) and a clean build (they fail). Independent of the other NVIDIA#23255 split PRs; can merge in any order. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: NVIDIA#23364
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
Split out of #23255 (6/6). `sort_index(axis=1)` silently ignored `level=` and `sort_remaining=` and always sorted by the full column labels. Sort by the requested levels (stable multi-key, least significant first), append the remaining levels when `sort_remaining=True`, resolve integer and named levels with pandas' bounds validation, and place missing labels per `na_position` independently of the per-key sort direction. Fixes 1 pandas-test (`test_stack_mixed_dtype[True]`, which sorts the stacked frame's columns by level); its xfail entry is removed. Attribution verified by running the node id against an isolated build containing only this change (passes) and a clean build (fails). Independent of the other #23255 split PRs; can merge in any order. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #23367
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 22, 2026
…23366) Split out of #23255 (5/6). `GroupBy.agg` flattened a MultiIndex-column source's aggregation result to flat tuple labels instead of keeping hierarchical columns like pandas. Preserve the MultiIndex (and its per-level metadata) when the aggregation keeps the source's tuple labels; relabeling aggregations (`agg(new=(col, func))`) emit new flat labels, so the source's multi-level metadata is not attached to those. Fixes 3 pandas-tests (`test_groupby_with_hier_columns`, `test_wrap_aggregated_output_multindex`, `test_multiindex_custom_func[<lambda>0]`); their xfail entries are removed. Attribution verified by running the node ids against an isolated build containing only this change (pass) and a clean build (fail). Independent of the other #23255 split PRs; the unstack PR (4/6) depends on this one for two entangled tests. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #23366
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 22, 2026
#23365) Split out of #23255 (2/6). Rebuilding a frame's pandas columns `MultiIndex` from tuples re-sorts the levels and re-infers their dtypes, losing the exact source layout: unsorted explicit level orders (which change the behavior of pandas operations that work on level codes, e.g. legacy `stack(sort=True)` after a fast-to-slow conversion under `cudf.pandas`), categorical/object/int64 level dtypes (int64 levels with missing entries upcast to float64), and NaN column labels (fresh `float('nan')` objects hash unequal, so lookups miss). - Prime the cached `to_pandas_index` with the exact source `pd.MultiIndex` at `DataFrame` construction and propagate it through accessor copies. - Restore recorded per-level dtypes in `to_pandas_index` when the cast round-trips losslessly. - Match NaN-containing column labels under pandas' all-NaNs-equal semantics. - Read level dtypes off `MultiIndex.levels` (`get_level_values` materializes missing entries as NaN and upcasts), also for `cudf.MultiIndex` columns. - Keep hierarchical columns through DataFrame binops when only level dtypes differ (restored `Int8` vs `int64` fails `Index.equals`). Fixes 13 pandas-tests (constructor dict-NaN-key, concat keys with specific levels, groupby ordered multi-func aggregate, MultiIndex loc, and several `test_stack_unstack.py` cases); their xfail entries are removed. Attribution verified by running each node id against an isolated build containing only this change (pass) and a clean build (fail). Independent of the other #23255 split PRs, but the stack (3/6) and unstack (4/6) PRs depend on this one. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Vyas Ramasubramani (https://github.com/vyasr) URL: #23365
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 23, 2026
#23370) Split out of #23255 (3/6), superseding it. **Depends on #23365 (MultiIndex column fidelity)** — 14 of the 28 un-xfailed pandas-tests need both fixes, so this PR's pandas-tests job goes green once #23365 merges. - Resolve `level` positionally: integer column-level *names* no longer collide with level *positions* (pandas' `Index.get_level_values` resolves integers by name first, so frames with integer level names returned data from the wrong level). - Validate out-of-bounds integer levels (`IndexError`) and duplicated level names (`ValueError`) with pandas' messages; negative out-of-bounds levels previously wrapped around silently. - Build the stacked level keys from the column MultiIndex's own levels/codes so per-level dtypes survive: int64 levels with missing entries no longer upcast to float64, and categorical levels stay categorical through the pylibcudf `tile` step (which only sees codes). - Emit stacked keys in appearance order, matching pandas. This replaces the argsort-based reordering, which misaligned column data for non-involution column permutations (e.g. a 3-cycle) and NaN keys; pandas legacy stack sorts multi-level keys by level *codes*, not values. - Attach pandas-faithful levels/codes to the result index eagerly (the original index contributes its own levels/codes; flat indexes and the tiled level get appearance-order factorization) so a later `unstack` restores the original row/column order; the legacy `dropna` path preserves them by masking codes instead of gathering the index. Fixes 28 pandas-tests; their xfail entries are removed. Three classic categorical unstack params are un-xfailed (fixed by this change together with #23365). Attribution verified per node id against isolated builds: 14 pass with only this change, 14 need this plus #23365. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Murray (https://github.com/Matt711) URL: #23370
rapids-bot Bot
pushed a commit
that referenced
this pull request
Jul 23, 2026
Split out of #23255 (4/6). **Depends on #23365 (fidelity), the stack PR (#23370), and #23366 (groupby agg)** — 8 of the 29 un-xfailed pandas-tests need those fixes too, so this PR's pandas-tests job goes green once they merge. - Order result rows/columns by the removed level's codes (level order preserved, missing keys first) instead of sorted values with nulls last, by encoding the integer code columns instead of the level values. - Propagate the source frame's column-axis level names into the result instead of hardcoding `None`; also fixes the `ValueError: Length of names must match number of levels` crash when unstacking MultiIndex-column frames. - Promote integer source columns to float64 when the reshape introduces missing cells (pandas' block semantics), gated on `mode.pandas_compatible`; `pivot_table`/`crosstab` opt out via a module-private `_unstack` parameter when `fill_value` fills the cells afterwards. - Preserve unused categories of the removed level in the result's column levels (pandas GH 17845); also fixes a libcudf `Column sizes don't match` crash for indexes with unused categorical categories. - Validate the level on flat-index frames (`KeyError`) and duplicated index names (`ValueError`) like pandas; `pivot` with `values=` drops the original columns-axis names. Fixes 29 pandas-tests; their xfail entries are removed, three remaining `test_stack_unstack.py` entries get real failure reasons, and two classic categorical unstack params are un-xfailed. Attribution verified per node id against isolated builds: 21 pass with only this change, 4 need the stack PR, 2 need stack+fidelity, 2 need the groupby-agg PR. Authors: - GALI PREM SAGAR (https://github.com/galipremsagar) Approvers: - Matthew Roeschke (https://github.com/mroeschke) URL: #23368
This was referenced Jul 23, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Running pandas' own test suite under
cudf.pandas,tests/frame/test_stack_unstack.pyhad 64 failing tests. This PR fixes 60 of them in cuDF classic (the remaining 4 assert pandasBlockManagerblock layouts or monkeypatch pandas' private_Unstacker, which cuDF cannot meaningfully satisfy) and removes the corresponding 60 xfail entries from the pandas-testing plugin.DataFrame.stacklevelpositionally: integer column-level names no longer collide with level positions (pandas'Index.get_level_valuesresolves integers by name first, so frames with integer level names returned data from the wrong level).IndexError) and duplicated level names (ValueError) with pandas' messages; previously negative out-of-bounds levels silently wrapped around.tilestep (which only sees codes).unstackrestore the original row/column order. The legacydropnapath preserves them by masking the codes instead of gathering the index.unstack/_pivotNone. This also fixes theValueError: Length of names must match number of levelscrash when unstacking frames with MultiIndex columns.mode.pandas_compatible.pivot_table/crosstabopt out via a module-private_unstackparameter since they fill missing cells afterwards and keep the integer dtype.Column sizes don't matchcrash for indexes with unused categorical categories.KeyError) and duplicated index names (ValueError) like pandas.Supporting fixes
ColumnAccessor: NaN-containing labels now match under pandas' all-NaNs-equal semantics;to_pandas_indexrestores recorded per-level dtypes when the cast round-trips losslessly; the cached pandas columns index is primed with the exact sourcepd.MultiIndexat construction and survives accessor copies. Rebuilding from tuples re-sorts the levels, which changes the behavior of pandas operations that work on codes (e.g. legacystack(sort=True)) after a fast-to-slow conversion.MultiIndex: lazy codes/levels materialization now factorizes withsort=True, matching the sorted levels pandas produces for per-row-value construction (set_index/from_arrays).sort_index(axis=1)now honorslevel=andsort_remaining=(previously silently ignored).GroupBy.aggkeeps MultiIndex columns for MultiIndex-column sources instead of flattening to tuple labels.NumericalColumn.as_numerical_columnno longer mutates the column's dtype in place on equal-pylibcudf-type casts; the column object may be shared with the caller's frame.np.nan(notNone) in pandas-compatible mode, matching pandas' upcast-to-object representation.cudf.pandas: an instance's transfer-blocking state is no longer baked into the class-level cached_MethodProxy, where it leaked into every later call on any instance of the class (order-dependent test failures).Two previously-xfailed
test_unstack_multiindexcategorical params in cuDF's own test suite now pass and are un-xfailed.Verified against the full cuDF test suites (reshape, dataframe, series, indexes, groupby, window, private-objects,
cudf_pandas_tests) and neighboring pandas-tests files (pivot/melt/crosstab/sort_index/indexes-multi) with no regressions.Checklist