Replace --executor with extended --frontend choices in cudf-polars benchmarks

[madsbk]

Description

Unify the engine/executor arguments into --frontend.

python python/cudf_polars/cudf_polars/experimental/benchmarks/pdsh.py \
    --frontend in-memory \
    --path /datasets/datasets/tpch-rs/scale-10 \
    --output pdsh-output.json \
    --iterations 2 \
    1

Swap --frontend in-memory for polars-cpu, duckdb, dask, ray, or spmd to switch backends; everything else stays the same.

Breaking Changes

CLI / argparse

• --engine removed from pdsh.py / pdsds.py
• --executor removed
• --frontend values are now:
  • duckdb
  • polars-cpu
  • spmd
  • ray
  • dask
  • in-memory
• --baseline value renames (and restricted choices)
  • --baseline cpu → --baseline polars-cpu
  • Default changed from "duckdb" to "duckdb" (same semantics, new spelling)
• --collect-traces
  • --collect-traces now raises ValueError when combined with:
    • --frontend polars-cpu
    • --frontend duckdb
  • Previously, these combinations were silently accepted as a no-op

Output JSON schema

• executor
  • Removed from the serialized run config as a consequence of removing --executor
• --collect-traces --frontend in-memory
  • Runs now populate records[*].traces
  • Previously, traces were empty or missing
  • This is a bug fix, but downstream consumers that assumed "in-memory" runs never produced traces will now observe new data

[wence-]

I would have expected also just migrating the contexts of utils_new_frontends.py into utils.py?

[madsbk]

I would have expected also just migrating the contexts of utils_new_frontends.py into utils.py?

Done now.

Keeping utils_new_frontends.py to avoid merge conflict for up coming PRs. I will do the move when they have been merged.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: caed82ed-7367-4388-8ac2-538b4a82c124

📥 Commits

Reviewing files that changed from the base of the PR and between 1d4901b and 783a125.

📒 Files selected for processing (1)

• python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py

---

📝 Walkthrough

Summary by CodeRabbit

• Refactor
  • Benchmark CLI interface updated: --frontend parameter replaces --executor for engine selection
  • DuckDB execution path removed from benchmark scripts
  • Query plan printing behavior updated based on frontend type
  • Debug mode now restricted to specific frontends; incompatible combinations raise errors
  • Validation baseline computation rewritten to use polars-cpu or duckdb sources

Walkthrough

Switches benchmark selection from an executor field to a typed RunConfig.frontend; CLI/frontends determine engine, plan printing, debug behavior, and validation baseline. Adds CPU and in-memory runners, widens engine-typed signatures to accept pl.GPUEngine | None, and conditions statistics/tracing on streaming-capable engines.

Changes

Frontend-based execution model

Layer / File(s)	Summary
Remove DuckDB branching in benchmark scripts python/cudf_polars/cudf_polars/experimental/benchmarks/pdsds.py, python/cudf_polars/cudf_polars/experimental/benchmarks/pdsh.py	run_duckdb import removed; __main__ entrypoints unconditionally call run_polars(...), eliminating prior engine-selection CLI logic.
Configuration contract and CLI argument updates python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py	Removes assert_never import; introduces _STREAMING_FRONTENDS and _CPU_ENGINES; replaces RunConfig.executor with RunConfig.frontend literal (dask, duckdb, in-memory, polars-cpu, ray, spmd); from_args derives engine_name and baseline expected_source; serialize()/summarize() drop executor output; parser/help text and --frontend/--baseline choices updated.
Query planning, execution branching, and baseline selection python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py	print_query_plan gates logical/physical explain based on run_config.frontend; CPU branch uses Polars collect(); debug execution split between in-memory (evaluate IR -> Polars) and streaming-capable frontends (evaluate_streaming), with unsupported combos raising ValueError; baseline selection uses match args.baseline for polars-cpu vs duckdb.
Engine signatures, statistics, tracing, and CPU/in-memory runners python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py	Widened engine parameter types to `pl.GPUEngine
run_polars CLI validation and dispatcher routing python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py	run_polars validates --connect/--num-gpus and rejects --collect-traces/--validate/--debug for CPU frontends; dispatcher routes frontends to run_polars_dask, run_duckdb, run_polars_in_memory, run_polars_cpu, run_polars_spmd, and errors on unknown frontends.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~75 minutes

Possibly related PRs

• rapidsai/cudf#22365: Overlapping changes in utils_new_frontends.py around serialization of engine-derived configuration and rapidsmpf options.

Suggested reviewers

• TomAugspurger
• pentschev
• bdice

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately reflects the main change: replacing --executor with --frontend options in cudf-polars benchmarks.
Description check	✅ Passed	The description comprehensively documents the purpose, breaking changes, and output schema updates related to the --frontend unification.
Docstring Coverage	✅ Passed	Docstring coverage is 80.95% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py`:
- Around line 1419-1424: Assertion arises because _consolidate_logs assumes a
1:1 positional match between grouped trace batches and run_records; instead
group traces by record.iteration and allow missing groups. Modify
_consolidate_logs to parse all trace Scope entries into groups keyed by
record.iteration (use the iteration field on records and any iteration metadata
on Scope logs), then when iterating run_records (including FailedRecord), look
up the corresponding trace group by iteration and tolerate a missing entry
rather than asserting counts; update any logic that relied on positional zipping
to use record.iteration-based lookups so that failed iterations with no traces
do not cause AssertionError.
- Around line 914-917: The polars-cpu validation uses the wrong execution path:
inside the match on args.baseline (the branch handling "polars-cpu") replace the
plain q.collect() call with the streaming invocation
q.collect(engine="streaming") so the baseline uses the same streaming engine as
the polars-cpu frontend; update the case handling for "polars-cpu" (the branch
referencing q.collect()) to call q.collect(engine="streaming") and keep the
existing assignment to expected.
- Around line 2022-2035: Add a validation that rejects --debug for CPU
frontends: if run_config.debug is true and run_config.frontend is in
_CPU_ENGINES, raise a ValueError with a clear message like "--debug is not
supported with --frontend {run_config.frontend}; debug mode only applies to GPU
frontends (in-memory, dask, ray, spmd)". Place this check alongside the existing
validations that reference run_config.collect_traces and
run_config.validation_method so execute_query() and run_duckdb() no longer
silently ignore the flag.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1ac9f079-fd9c-463f-866d-8b8e22d31052

📥 Commits

Reviewing files that changed from the base of the PR and between 721084f and 6032420.

📒 Files selected for processing (1)

• python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py

[madsbk]

Thanks @wence-, I merged with main not release/26.06

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py`:
- Around line 1074-1076: The current validation-summary gating uses
args.validate so validation-run metadata (like validation_failures) happens but
the human-readable summary is suppressed for --validate-directory; change the
condition on the validation-summary branch to check run_config.validation_method
is not None instead of args.validate (keep the existing frontend exclusion check
run_config.frontend not in _CPU_ENGINES), i.e., make the if that currently reads
"if args.validate and run_config.frontend not in _CPU_ENGINES:" use
"run_config.validation_method is not None" to ensure summaries are printed for
directory-based validation while preserving CPU-engine gating and referencing
args.summarize and run_config.summarize() as before.
- Around line 819-823: Add the runtime import for StreamingEngine to the
existing optional cudf-polars import try/except block (the block that currently
handles the optional cudf-polars/rapidsmpf frontend imports) so StreamingEngine
is available at module level, then remove the local function-level imports of
StreamingEngine (the ones inside the functions that check isinstance(engine,
StreamingEngine)); keep the isinstance(engine, StreamingEngine) check and the
call to engine.global_statistics(clear=True).to_dict() as-is but rely on the
module-level import to avoid PLC0415 violations.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 32bb0f48-9951-4831-baad-39bc85b171b2

📥 Commits

Reviewing files that changed from the base of the PR and between 80d719e and 1d4901b.

📒 Files selected for processing (1)

• python/cudf_polars/cudf_polars/experimental/benchmarks/utils_new_frontends.py

[madsbk]

/merge