Skip to content

[FEA] Enable Polars GPU execution via global configuration (default options) #16723

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
beckernick opened this issue Sep 3, 2024 · 4 comments
Closed
Assignees
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request Python Affects Python cuDF API.

Comments

@beckernick
Copy link
Member

Polars provides a Config object that can be used globally or as a context manager.

Recent discussions have clarified that some users (and downstream libraries) would benefit from being able to globally configure Polars to use the GPU engine rather than configuring it on a per-collect call basis.

The Polars Config system is environment variable based, so there's some complexity challenges in supporting a full set of configuration options via strings. But, as @wence- has noted, it would likely be more straightforward to enable the default configuration via an environment variable.

We should implement a Config option for the default GPU engine configuration, as it would be useful to many end-users and library developers building on top of Polars.

@beckernick beckernick added the feature request New feature or request label Sep 3, 2024
@wence- wence- added the cudf.polars Issues specific to cudf.polars label Sep 3, 2024
@wence- wence- removed this from cuDF Python Sep 3, 2024
@vyasr
Copy link
Contributor

vyasr commented Oct 4, 2024

Note: Ibis recently implemented the functionality to forward the engine down to Polars, which in theory means that we no longer need a global Polars config for that case to work (there are of course other valid use cases).

@bdice
Copy link
Contributor

bdice commented Oct 16, 2024

It would be good for this global configuration to support changing the RMM memory resource. It would be sufficient to have some coarse control like what we have for libcudf benchmarks via a CLI flag --rmm_mode and cudf.pandas via an environment variable CUDF_PANDAS_RMM_MODE.

@wence-
Copy link
Contributor

wence- commented Nov 14, 2024

c.f. also pola-rs/polars#19797

@vyasr vyasr added the Python Affects Python cuDF API. label Nov 14, 2024
@Matt711 Matt711 self-assigned this Jan 14, 2025
@Matt711 Matt711 moved this from Todo to In Progress in cuDF Python Jan 15, 2025
@Matt711
Copy link
Contributor

Matt711 commented Mar 20, 2025

Closed by #18132

@Matt711 Matt711 closed this as completed Mar 20, 2025
@github-project-automation github-project-automation bot moved this from In Progress to Done in cuDF Python Mar 20, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cudf.polars Issues specific to cudf.polars feature request New feature or request Python Affects Python cuDF API.
Projects
Status: Done
Development

No branches or pull requests

5 participants