Skip to content

Conversation

@HyukjinKwon
Copy link
Member

What changes were proposed in this pull request?

This PR proposes to:

  • Suppress error logs when the number of timers is set to 0.
  • Add documentation for describing its behaviour clearly:
    • 0: disable the feature
    • -1: no limit
    • Otherwise, explicit limit.
  • Throw an exception if the configuration is set to the number lower than -1.
    This is an internal configuration, numbers lower than -1 don't make much sense, and this configuration sort of less known. So I think it's pretty safe to do this.

Why are the changes needed?

To avoid noisy error logs, and document the feature properly.

Does this PR introduce any user-facing change?

Yes. When spark.scheduler.listenerbus.metrics.maxListenerClassesTimed is set to 0, it does not show a warning such as:

LiveListenerBusMetrics: Not measuring processing time for listener class org.apache.spark.sql.util.ExecutionListenerBus because a maximum of 0 listener classes are already timed.

How was this patch tested?

Unittest is added.

@github-actions github-actions bot added the CORE label Aug 8, 2022
Copy link
Member

@dongjoon-hyun dongjoon-hyun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1, LGTM. It makes sense and looks safe.

@HyukjinKwon
Copy link
Member Author

Merged to master.

@HyukjinKwon
Copy link
Member Author

thank you guys!

@HyukjinKwon HyukjinKwon deleted the SPARK-39973 branch January 15, 2024 00:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants