feat(rust,python): Add config to specify GPU polars as the default engine

[Matt711]

Apart of #19797. We should also follow up the PR to allow other engines to be the default.

[ion-elgreco]

Wouldn't it be more practical if you can set any engine then as default?

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 79.70%. Comparing base (73cb2a2) to head (c4f03e9).

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #20717      +/-   ##
==========================================
- Coverage   79.70%   79.70%   -0.01%     
==========================================
  Files        1561     1561              
  Lines      221930   221947      +17     
  Branches     2530     2530              
==========================================
+ Hits       176892   176905      +13     
- Misses      44456    44460       +4     
  Partials      582      582              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[orlp]

I think this isn't as simple as it seems, because we do also call Polars methods in the Python code internally which are only supported on, for example, the eager engine.

[wence-]

I think this isn't as simple as it seems, because we do also call Polars methods in the Python code internally which are only supported on, for example, the eager engine.

Can you expand on what you mean here? Effectively what this is trying to do is add a global config option for the default value in LazyFrame.collect(..., engine="cpu") (to make that literal "cpu" string optionally "gpu" instead). The Python-side logic that then perhaps nonetheless turns off the gpu engine still exists.

Why would hooking this up via a config option not work?

[orlp]

@wence- I have no idea what the full scope of such examples are throughout the codebase, but one example is pl.align_frames dispatching to other calls. Such a function might suddenly stop working if you change the default engine, because not all engines have the exact same supported feature set.

[wence-]

OK, thanks, I think I see. I think there's some amount of tension here between making this "easy" to configure globally and I guess some things breaking at a distance from where the user might expect.

[eitsupi]

Just a question: why even change polars-core when changing the engine is a Python-specific feature?