Skip to content

JNI Set RMM_LOG_LEVEL and RMM_LOG_ACTIVE_LEVEL to allow setting log level at compile time - #19435

Merged
rapids-bot[bot] merged 2 commits into
NVIDIA:branch-25.08from
abellina:set_rmm_log_level
Jul 24, 2025
Merged

JNI Set RMM_LOG_LEVEL and RMM_LOG_ACTIVE_LEVEL to allow setting log level at compile time#19435
rapids-bot[bot] merged 2 commits into
NVIDIA:branch-25.08from
abellina:set_rmm_log_level

Conversation

@abellina

@abellina abellina commented Jul 18, 2025

Copy link
Copy Markdown
Contributor

We have confusion from the customer side when they look at logs and see ERRORs from our pinned memory pool, since it is using a pooled_memory_resource that when exhausted we handle gracefully (here's a report NVIDIA/cudf-spark#13131):

[2602819][08:08:45:815536][error ] [A][Stream 0x1][Upstream 1024B][FAILURE maximum pool size exceeded: Not enough room to grow, current/max/try size = 0.000000 B, 0.000000 B, 1024.000000 B]

The problem stems from us setting RMM_LOGGING_LEVEL in our cmake builds, but this is not getting propagated to RMM itself, somehow. I have been able to make this propagate, and silence the error @revans2 posted, but I am not 100% sure this is the right thing to do. Asking folks here for some help (@bdice) to check the CMakeLists change.

Description

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

…vel at compile time

Signed-off-by: Alessandro Bellina <abellina@nvidia.com>
@abellina
abellina requested a review from a team as a code owner July 18, 2025 19:00
@copy-pr-bot

copy-pr-bot Bot commented Jul 18, 2025

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions github-actions Bot added CMake CMake build issue Java Affects Java cuDF API. labels Jul 18, 2025
@abellina
abellina marked this pull request as draft July 18, 2025 19:01
@abellina abellina added Spark Functionality that helps Spark RAPIDS non-breaking Non-breaking change improvement Improvement / enhancement to an existing function labels Jul 18, 2025
endif()

# Set RMM logging level
set(RMM_LOGGING_LEVEL

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@vyasr Is this the correct way to use rapids-logger? I discussed a bit with @abellina offline but I don't know the proper way to solve this problem.

@bdice bdice Jul 18, 2025

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's a related conversation here about what should be precompiled in RMM here: rapidsai/rmm#1983 (comment)

It feels like these two topics might be solvable in similar ways.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming rmm is still being from source as part of the build, this will work, yes. It won't have any effect if you are using precompiled binaries, but since Spark doesn't do that this seems fine. The actual logs observed at runtime may still require setting the runtime log level if the logging functions are used.

@vyasr

vyasr commented Jul 21, 2025

Copy link
Copy Markdown
Contributor

This PR would probably be a good time to also resolve #15416 and #15417.

@GregoryKimball GregoryKimball moved this to Burndown in libcudf Jul 22, 2025
@abellina
abellina marked this pull request as ready for review July 23, 2025 20:31
@abellina

Copy link
Copy Markdown
Contributor Author

@vyasr @bdice we are going to go ahead with this in cuDF JNI. I haven't found a better way.

@abellina

Copy link
Copy Markdown
Contributor Author

/build

@abellina

Copy link
Copy Markdown
Contributor Author

/ok to test

@copy-pr-bot

copy-pr-bot Bot commented Jul 23, 2025

Copy link
Copy Markdown

/ok to test

@abellina, there was an error processing your request: E1

See the following link for more information: https://docs.gha-runners.nvidia.com/cpr/e/1/

@abellina

Copy link
Copy Markdown
Contributor Author

/ok to test df05035

@abellina

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 04db8dc into NVIDIA:branch-25.08 Jul 24, 2025
@mhaseeb123 mhaseeb123 moved this from Burndown to Landed in libcudf Jul 24, 2025
@bdice

bdice commented Jul 25, 2025

Copy link
Copy Markdown
Contributor

This PR would probably be a good time to also resolve #15416 and #15417.

@abellina Were you able to look at the two linked issues? Are those still relevant or can they be fixed/closed?

@abellina

Copy link
Copy Markdown
Contributor Author

This PR would probably be a good time to also resolve #15416 and #15417.

@abellina Were you able to look at the two linked issues? Are those still relevant or can they be fixed/closed?

Thanks for reminding me @bdice. Closed the other issues.

@abellina
abellina deleted the set_rmm_log_level branch July 25, 2025 21:04
@GregoryKimball GregoryKimball removed this from libcudf Sep 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CMake CMake build issue improvement Improvement / enhancement to an existing function Java Affects Java cuDF API. non-breaking Non-breaking change Spark Functionality that helps Spark RAPIDS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants