[SPARK-47276][PYTHON][CONNECT] Introduce spark.profile.clear for SparkSession-based profiling

[xinrong-meng]

What changes were proposed in this pull request?

Introduce spark.profile.clear for SparkSession-based profiling.

Why are the changes needed?

A straightforward and unified interface for managing and resetting profiling results for SparkSession-based profilers.

Does this PR introduce any user-facing change?

Yes. spark.profile.clear is supported as shown below.

Preparation:

>>> from pyspark.sql.functions import pandas_udf
>>> df = spark.range(3)
>>> @pandas_udf("long")
... def add1(x):
...   return x + 1
... 
>>> added = df.select(add1("id"))
>>> spark.conf.set("spark.sql.pyspark.udf.profiler", "perf")
>>> added.show()
+--------+                                                                      
|add1(id)|
+--------+
...
+--------+
>>> spark.profile.show()
============================================================
Profile of UDF<id=2>
============================================================
         1410 function calls (1374 primitive calls) in 0.004 seconds
...

Example usage:

>>> spark.profile.profiler_collector._profile_results
{2: (<pstats.Stats object at 0x7ff6484d22e0>, None)}

>>> spark.profile.clear(1)  # id mismatch
>>> spark.profile.profiler_collector._profile_results
{2: (<pstats.Stats object at 0x7ff6484d22e0>, None)}

>>> spark.profile.clear(type="memory")  # type mismatch
>>> spark.profile.profiler_collector._profile_results
{2: (<pstats.Stats object at 0x7ff6484d22e0>, None)}

>>> spark.profile.clear()  # clear all
>>> spark.profile.profiler_collector._profile_results
{}
>>> spark.profile.show()
>>> 

How was this patch tested?

Unit tests.

Was this patch authored or co-authored using generative AI tooling?

No.

[xinrong-meng]

Failed tests are irrelevant to changes proposed in this PR. Rerun failed tests https://github.com/xinrong-meng/spark/actions/runs/8162084262.

[HyukjinKwon]

Is this a user-facing API? If not, we don't need this version directive

[xinrong-meng]

It is a user-facing API, along with profile.show and profile.dump. We will also add it to API doc.

[ueshin]

Actually this is not. The clear in Profile should be a user-facing API.

[HyukjinKwon]

Seems fine but cc @ueshin

[ueshin]

Otherwise, LGTM, pending tests.

[zhengruifeng]

nit, it seems we don't have a parity test for test_session. does it make sense to move SparkSessionProfileTests out of test_session and add parity test for it?

[xinrong-meng]

Good idea!

For now, all logic tested by SparkSessionProfileTests is directly imported in Spark Connect with no modification. But I do agree separating it later will improve readability and ensure future parity. I'll refactor later. Thanks!

[xinrong-meng]

Merged to master, thank you all!