diff --git a/.agents/skills/debug-cudf-pandas/SKILL.md b/.agents/skills/debug-cudf-pandas/SKILL.md index df1897076be9..6d60e189038e 100644 --- a/.agents/skills/debug-cudf-pandas/SKILL.md +++ b/.agents/skills/debug-cudf-pandas/SKILL.md @@ -15,6 +15,7 @@ When the pandas test suite is run with `-p cudf.pandas`, test failures indicate - A **missing proxy registration** — a pandas type or return value has no registered cudf equivalent - A **to/from_pandas conversion bug** — data is corrupted or lost when converting between cudf and pandas objects - A **test setup bug** — the testing scripts or conftest-patch introduce an issue +- A **dependency/environment gap** — the test requires a package (e.g. xlsxwriter) that pandas CI has but our test environment lacks, causing a different code path to execute - A **pandas bug** — rarely, the expected behavior in the pandas test itself is wrong Your job is to find the root cause and implement the fix. @@ -28,6 +29,7 @@ The following patterns are prohibited regardless of whether they make a test pas - **Private pandas APIs**: Do not import or call any symbol from `pandas.core`, `pandas.compat`, or any underscored pandas module (e.g. `pandas._libs.tslibs.parsing`). These are explicitly unstable per the pandas API policy. Use public pandas APIs or write equivalent local logic instead. - **PyArrow as a CPU execution backend**: Do not route GPU operations through `pyarrow.compute` on CPU as a substitute for cudf/libcudf semantics. Arrow is an interchange format; it is not an acceptable execution backend for cudf operations. - **Returning pandas objects from cudf APIs**: cudf public methods (`Series`, `Index`, `DataFrame` operations and accessors) must return cudf-native objects, not `pd.Series`, `pd.Index`, or `pd.DataFrame`. Use `_return_or_inplace` and the existing cudf container reconstruction helpers. +- **Diverging from pandas to pass a test**: The goal is to match pandas behavior exactly. Do not implement proxy overrides that suppress exceptions or alter behavior that vanilla pandas exhibits. If pandas raises an error in a given scenario, cudf.pandas should raise the same error. A fix that makes cudf.pandas behave *differently* from pandas — even if it makes a test pass — is wrong. --- @@ -157,6 +159,7 @@ Results: - **cudf result differs from pandas** → cudf implementation bug → go to Step 4a - **cudf raises an exception** → missing feature or bug → evaluate scope; may need user input if the feature is large. Note: this may be OK if the test is verifying that an exception *should* be raised. - **cudf result matches pandas** → proxy/dispatch bug → go to Step 4b +- **cudf result matches pandas AND the test still fails** → check if vanilla pandas (without cudf.pandas) also fails → go to Step 3d **Classify the root cause before writing any fix.** Ask yourself: Is this a specific method/keyword handling bug? A broad dtype casting mismatch affecting many operations? A proxy/wrapping issue? A missing cudf capability? For broad issues, the fix should be applied at the shared/base layer, not patched per individual method. If the only apparent fix is test-shaped (i.e. it looks like it exists to make exactly these node IDs pass), step back and re-examine the general API contract. @@ -192,6 +195,25 @@ python -m cudf.pandas test_debug.py This gives you full control to narrow down exactly where the divergence begins. +### 3d. Verify vanilla pandas behavior (critical sanity check) + +Before implementing any proxy-layer fix, check whether the test passes under vanilla pandas in your environment: + +```bash +python -m pytest pandas-testing/pandas-tests/tests/:: -xvs +``` + +(Without `-p cudf.pandas` — just run it directly.) + +If the test **also fails under vanilla pandas**, the issue is NOT a cudf bug. Common causes: +- **Missing dependency**: pandas CI has a package installed (e.g. `xlsxwriter`, `lxml`, `odfpy`) that changes code path selection. Check pandas' `ci/deps/` YAML files to see what they install. +- **Version mismatch**: the installed version of a third-party library differs from what pandas CI uses. +- **Pandas test bug**: the test itself is broken (e.g. relies on side effects of other packages being present). + +Resolution for dependency gaps: add the missing package to `dependencies.yaml` under the `test_cudf_pandas_pandas_tests` section (for conda environments that don't use pip extras), then run `rapids-dependency-file-generator` to propagate. See Step 4c. + +Resolution for pandas bugs: xfail the test with an explanation string that describes why it's a pandas/upstream issue, and optionally write up a bug report for upstream. + --- ## Step 4a — Fix a cudf Implementation Bug @@ -233,6 +255,46 @@ Only reach this step after Step 3a has confirmed that cudf itself is correct. So **`fast_slow_proxy.py` and `module_accelerator.py`** are core infrastructure files. Fix them only if you believe the bug is in one of them. +### Important constraint for proxy fixes + +**Never make cudf.pandas diverge from pandas to pass a test.** If your proposed proxy fix would cause cudf.pandas to behave *differently* from vanilla pandas (e.g. suppressing an exception that pandas raises, or returning a different value), that fix is wrong — even if it makes the test pass. The test may be broken, or the issue may be an environment/dependency gap rather than a proxy bug. Always verify vanilla pandas behavior first (Step 3d). + +--- + +## Step 4c — Fix a Dependency or Environment Gap + +Only reach this step if Step 3d confirmed the test also fails under vanilla pandas due to a missing package or version mismatch. + +1. **Identify the missing dependency.** Check what pandas CI installs by examining their CI config files (available in `pandas-testing/pandas/ci/deps/`). Common culprits: `xlsxwriter`, `lxml`, `odfpy`, `python-calamine`, `pyxlsb`. + +2. **Add to `dependencies.yaml`** under the `test_cudf_pandas_pandas_tests` section. This group provides packages that pandas CI has installed (via pip extras like `pandas[excel]`) but conda environments need listed explicitly: + +```yaml + # Additional dependencies for running the pandas test suite under cudf.pandas. + # Unlike test_python_pandas_cudf (which uses pip extras like pandas[excel]), + # conda environments need these listed explicitly. + test_cudf_pandas_pandas_tests: + common: + - output_types: [conda] + packages: + - +``` + +3. **Regenerate dependency files:** + +```bash +rapids-dependency-file-generator +``` + +This propagates the change to `python/cudf/pyproject.toml` and any other generated files. + +4. **If the test still fails even with the dependency present** (e.g. the test has a genuine pandas/upstream bug that happens regardless), xfail it with an explanation: + +```python +"tests/io/excel/test_openpyxl.py::test_name": "", +``` + +The explanation string in the xfail dict should describe the *root cause* (e.g. "openpyxl limitation", "pandas test bug: assumes xlsxwriter present"), not just the error message. --- ## Step 5 — Verify the Fix @@ -319,3 +381,6 @@ For intentional divergence: stop and ask the user. In most cases, the goal is to - Never fix the testing APIs (like `assert_frame_equal`, `assert_series_equal`) — fix the actual APIs that produce wrong results. - First see if the problem is in cudf classic and fix it there; if not, then move over to cudf.pandas. - Tests run with `xfail_strict = true` — a test listed in `NODEIDS_THAT_FAIL` that unexpectedly passes is reported as `XPASS` (also a failure). Remove from the list before testing. +- When a fix requires adding a test dependency, update `dependencies.yaml` (under `test_cudf_pandas_pandas_tests` for conda environments) and run `rapids-dependency-file-generator` to propagate. Never manually edit the generated `pyproject.toml` entries marked as auto-generated. +- Always verify vanilla pandas behavior before implementing proxy-layer fixes. If the test also fails without cudf.pandas, the problem is upstream or environmental, not a cudf bug. +- xfail explanation strings should describe the root cause ("openpyxl limitation", "pandas test assumes xlsxwriter is installed"), not just the error type ("AssertionError", "IndexError"). diff --git a/conda/environments/all_cuda-129_arch-aarch64.yaml b/conda/environments/all_cuda-129_arch-aarch64.yaml index 3af8a424405a..77c6655b322a 100644 --- a/conda/environments/all_cuda-129_arch-aarch64.yaml +++ b/conda/environments/all_cuda-129_arch-aarch64.yaml @@ -103,6 +103,7 @@ dependencies: - structlog - sysroot_linux-aarch64==2.28 - typing_extensions>=4.0.0 +- xlsxwriter - zlib>=1.2.13 - zstandard name: all_cuda-129_arch-aarch64 diff --git a/conda/environments/all_cuda-129_arch-x86_64.yaml b/conda/environments/all_cuda-129_arch-x86_64.yaml index df21cbb2f4f5..1f606aafab49 100644 --- a/conda/environments/all_cuda-129_arch-x86_64.yaml +++ b/conda/environments/all_cuda-129_arch-x86_64.yaml @@ -103,6 +103,7 @@ dependencies: - structlog - sysroot_linux-64==2.28 - typing_extensions>=4.0.0 +- xlsxwriter - zlib>=1.2.13 - zstandard name: all_cuda-129_arch-x86_64 diff --git a/conda/environments/all_cuda-132_arch-aarch64.yaml b/conda/environments/all_cuda-132_arch-aarch64.yaml index 30555396b3e8..e4a9071c2601 100644 --- a/conda/environments/all_cuda-132_arch-aarch64.yaml +++ b/conda/environments/all_cuda-132_arch-aarch64.yaml @@ -103,6 +103,7 @@ dependencies: - structlog - sysroot_linux-aarch64==2.28 - typing_extensions>=4.0.0 +- xlsxwriter - zlib>=1.2.13 - zstandard name: all_cuda-132_arch-aarch64 diff --git a/conda/environments/all_cuda-132_arch-x86_64.yaml b/conda/environments/all_cuda-132_arch-x86_64.yaml index 1525670ec4ac..a2c20b88a7ab 100644 --- a/conda/environments/all_cuda-132_arch-x86_64.yaml +++ b/conda/environments/all_cuda-132_arch-x86_64.yaml @@ -103,6 +103,7 @@ dependencies: - structlog - sysroot_linux-64==2.28 - typing_extensions>=4.0.0 +- xlsxwriter - zlib>=1.2.13 - zstandard name: all_cuda-132_arch-x86_64 diff --git a/dependencies.yaml b/dependencies.yaml index 7a795232f16f..d47fac2e3181 100644 --- a/dependencies.yaml +++ b/dependencies.yaml @@ -1317,13 +1317,14 @@ dependencies: # https://github.com/pytest-dev/pytest-rerunfailures/issues/302 - pytest-rerunfailures!=16.0.0 # Additional dependencies for running the pandas test suite under cudf.pandas. - # Unlike test_python_pandas_cudf (which uses pip extras like pandas[performance]), + # Unlike test_python_pandas_cudf (which uses pip extras like pandas[excel]), # conda environments need these listed explicitly. test_cudf_pandas_pandas_tests: common: - output_types: [conda] packages: - numexpr + - xlsxwriter depends_on_dask_cuda: common: - output_types: conda diff --git a/python/cudf/cudf/pandas/_wrappers/pandas.py b/python/cudf/cudf/pandas/_wrappers/pandas.py index d295cf647dbe..e7d1cf3ab546 100644 --- a/python/cudf/cudf/pandas/_wrappers/pandas.py +++ b/python/cudf/cudf/pandas/_wrappers/pandas.py @@ -20,6 +20,8 @@ from pandas._libs.tslibs import offsets as liboffsets from pandas._testing import at, getitem, iat, iloc, loc, setitem from pandas.compat._optional import import_optional_dependency +from pandas.io.excel._openpyxl import OpenpyxlWriter as pd_OpenpyxlWriter +from pandas.io.excel._xlsxwriter import XlsxWriter as pd_XlsxWriter from pandas.tseries.holiday import ( AbstractHolidayCalendar as pd_AbstractHolidayCalendar, EasterMonday as pd_EasterMonday, @@ -281,6 +283,26 @@ def _to_xarray(self): return xr.Dataset.from_dataframe(self) +# pandas.ExcelWriter uses __new__ to dispatch to the engine-specific subclass +# (OpenpyxlWriter, XlsxWriter, etc.) based on the `engine` kwarg. The proxy +# must replicate this: construct the real writer with the accelerator disabled +# (so we get the actual pandas writer, not a recursive proxy) then wrap the +# result. __init__ is a no-op because construction is fully handled in __new__. +def _ExcelWriter__new__(cls, *args, **kwargs): + if cls is not ExcelWriter: + return object.__new__(cls) + + from ..module_accelerator import disable_module_accelerator + + with disable_module_accelerator(): + writer = pd.ExcelWriter(*args, **kwargs) + return _maybe_wrap_result(writer, pd.ExcelWriter, *args, **kwargs) + + +def _ExcelWriter__init__(self, *args, **kwargs): + pass + + DataFrame = make_final_proxy_type( "DataFrame", cudf.DataFrame, @@ -1339,11 +1361,37 @@ def Index__setattr__(self, name, value): additional_attributes={ "__hash__": _FastSlowAttribute("__hash__"), "__fspath__": _FastSlowAttribute("__fspath__"), + "__init__": _ExcelWriter__init__, + "__new__": _ExcelWriter__new__, }, bases=(os.PathLike,), metaclasses=(abc.ABCMeta,), ) +OpenpyxlWriter = make_final_proxy_type( + "OpenpyxlWriter", + _Unusable, + pd_OpenpyxlWriter, + fast_to_slow=_Unusable(), + slow_to_fast=_Unusable(), + additional_attributes={ + "__fspath__": _FastSlowAttribute("__fspath__"), + }, + bases=(ExcelWriter,), +) + +XlsxWriter = make_final_proxy_type( + "XlsxWriter", + _Unusable, + pd_XlsxWriter, + fast_to_slow=_Unusable(), + slow_to_fast=_Unusable(), + additional_attributes={ + "__fspath__": _FastSlowAttribute("__fspath__"), + }, + bases=(ExcelWriter,), +) + try: from pandas.io.formats.style import Styler as pd_Styler # isort: skip from pandas.io.formats.style import StylerRenderer as pd_StylerRenderer diff --git a/python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py b/python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py index 1f518298965b..3cd56ac6a485 100644 --- a/python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py +++ b/python/cudf/cudf/pandas/scripts/pandas-testing-plugin.py @@ -4097,9 +4097,11 @@ def pytest_unconfigure(config): "tests/io/excel/test_readers.py::TestReaders::test_read_excel_blank_with_header[(None, '.xlsm')]": "AssertionError: Attributes of DataFrame.iloc[:, 0] (column name='col_1') are different", "tests/io/excel/test_readers.py::TestReaders::test_read_excel_blank_with_header[(None, '.xlsx')]": "AssertionError: Attributes of DataFrame.iloc[:, 0] (column name='col_1') are different", "tests/io/excel/test_readers.py::TestReaders::test_read_excel_ods_nested_xml[('odf', '.ods')-gh-36122-expected1]": "AssertionError: Attributes of DataFrame.iloc[:, 0] (column name='got 2nd sa') are different", + "tests/io/excel/test_openpyxl.py::test_engine_kwargs_append_data_only": "openpyxl data_only=True reads cached formula results; freshly-written files have no cache, which is an openpyxl/Excel limitation rather than a cudf bug", "tests/io/excel/test_style.py::test_format_hierarchical_rows_periodindex[False]": "AttributeError: _compute. Did you mean: 'compare'?", "tests/io/excel/test_style.py::test_format_hierarchical_rows_periodindex[True]": "AttributeError: _compute. Did you mean: 'compare'?", "tests/io/excel/test_style.py::test_format_hierarchical_rows_periodindex[columns]": "AttributeError: _compute. Did you mean: 'compare'?", + "tests/io/excel/test_style.py::test_styler_custom_converter": "openpyxl raises IndexError on workbook with no visible sheets; cudf.pandas fallback triggers this openpyxl limitation", "tests/io/excel/test_writers.py::TestExcelWriter::test_excel_date_datetime_format[odf-.ods]": "TODO: Add a reason for failure", "tests/io/excel/test_writers.py::TestExcelWriter::test_excel_date_datetime_format[openpyxl-.xlsm]": "TODO: Add a reason for failure", "tests/io/excel/test_writers.py::TestExcelWriter::test_excel_date_datetime_format[openpyxl-.xlsx]": "TODO: Add a reason for failure",