[SPARK-52458][CORE] Support spark.eventLog.excludedPatterns

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to support spark.eventLog.excludedPatterns to exclude specific SparkEvents. This has two goals.

1. Save the cost of event log processing.
2. Provide a full control to save or not on top of the existing logEvent.

spark/common/utils/src/main/scala/org/apache/spark/scheduler/SparkListenerEvent.scala

Lines 25 to 28 in 68136fd

	trait SparkListenerEvent {
	/* Whether output this event to the event log */
	protected[spark] def logEvent: Boolean = true
	}

Why are the changes needed?

Historically, Apache Spark provides multiple ways to manage the event logs to save a storage cost.

• spark.history.fs.cleaner.maxAge: Delete old Spark jobs by age
• spark.history.fs.cleaner.maxNum: Delete old Spark jobs by the total number of jobs
• spark.history.fs.eventLog.rolling.maxFilesToRetain: Decompress + Compact + Compress back

For example, after compaction, Spark event logs only have the following.

// The extracted event names from a `compacted` event log file
"SparkListenerLogStart"
"SparkListenerResourceProfileAdded"
"org.apache.spark.sql.connect.service.SparkListenerConnectServiceStarted"
"SparkListenerBlockManagerAdded"
"SparkListenerEnvironmentUpdate"
"SparkListenerApplicationStart"
"SparkListenerExecutorAdded"
"SparkListenerBlockManagerAdded"
"SparkListenerExecutorAdded"
"SparkListenerBlockManagerAdded"
"SparkListenerJobStart"
"SparkListenerStageSubmitted"
"SparkListenerTaskStart"
"SparkListenerTaskEnd"
"SparkListenerStageCompleted"

This PR aims to provide a simple alternative to allow the users to skip specific Spark events completely.

org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart
org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionEnd
org.apache.spark.sql.execution.ui.SparkListenerDriverAccumUpdates

Does this PR introduce any user-facing change?

No. This is a new feature.

How was this patch tested?

Pass the CIs with the newly added test case.

Was this patch authored or co-authored using generative AI tooling?

No.

[dongjoon-hyun]

I addressed your comments.

$ build/sbt "core/testOnly *EventLogFile*Writer*Suite -- -z SPARK-52458"
...
[info] RollingEventLogFilesWriterSuite:
[info] - SPARK-52458: Support spark.eventLog.excludedPatterns (117 milliseconds)
[info] SingleEventLogFileWriterSuite:
[info] - SPARK-52458: Support spark.eventLog.excludedPatterns (17 milliseconds)
[info] Run completed in 1 second, 129 milliseconds.
[info] Total number of tests run: 2
[info] Suites: completed 2, aborted 0
[info] Tests: succeeded 2, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
[success] Total time: 10 s, completed Jun 11, 2025, 8:05:36 PM

[yaooqinn]

QQ: Is it possible that logs become unable to be rendered if some events are missing?

[dongjoon-hyun]

QQ: Is it possible that logs become unable to be rendered if some events are missing?

Of course, yes, the users need to provide a meaningful configuration.

For example, the event names like the following should not be used. As you see in the PR description, those survived after compactions also.

"SparkListenerLogStart"
"SparkListenerResourceProfileAdded"
"org.apache.spark.sql.connect.service.SparkListenerConnectServiceStarted"
"SparkListenerBlockManagerAdded"
"SparkListenerEnvironmentUpdate"
"SparkListenerApplicationStart"
"SparkListenerExecutorAdded"
"SparkListenerBlockManagerAdded"
"SparkListenerExecutorAdded"
"SparkListenerBlockManagerAdded"
"SparkListenerJobStart"
"SparkListenerStageSubmitted"
"SparkListenerTaskStart"
"SparkListenerTaskEnd"
"SparkListenerStageCompleted"

The meaningful and intuitive set of configurations are the Spark UI ones in the PR descriptions like the following. Also, they should be excluded together. For example, SparkListenerSQLExecutionEnd and SparkListenerSQLExecutionStart are related.

org.apache.spark.sql.execution.ui.SparkListenerSQLAdaptiveExecutionUpdate
org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionStart
org.apache.spark.sql.execution.ui.SparkListenerSQLExecutionEnd
org.apache.spark.sql.execution.ui.SparkListenerDriverAccumUpdates

In addition, this can be used to prevent the user-defined SparkEvent classes from being exposed to the persistent event logs.

[yaooqinn]

+1, LGTM

[dongjoon-hyun]

Thank you, @yaooqinn !

[dongjoon-hyun]

Merged to master for Apache Spark 4.1.0.