[SPARK-29654][CORE] Add configuration to allow disabling registration of static sources to the metrics system

[LucaCanali]

What changes were proposed in this pull request?

The Spark metrics system produces many different metrics and not all of them are used at the same time. This proposes to introduce a configuration parameter to allow disabling the registration of metrics in the "static sources" category.

Why are the changes needed?

This allows to reduce the load and clutter on the sink, in the cases when the metrics in question are not needed. The metrics registerd as "static sources" are under the namespaces CodeGenerator and HiveExternalCatalog and can produce a significant amount of data, as they are registered for the driver and executors.

Does this PR introduce any user-facing change?

It introduces a new configuration parameter spark.metrics.register.static.sources.enabled

How was this patch tested?

Manually tested.

$ cat conf/metrics.properties
*.sink.prometheusServlet.class=org.apache.spark.metrics.sink.PrometheusServlet
*.sink.prometheusServlet.path=/metrics/prometheus
master.sink.prometheusServlet.path=/metrics/master/prometheus
applications.sink.prometheusServlet.path=/metrics/applications/prometheus

$ bin/spark-shell

$ curl -s http://localhost:4040/metrics/prometheus/ | grep Hive
metrics_local_1573330115306_driver_HiveExternalCatalog_fileCacheHits_Count 0
metrics_local_1573330115306_driver_HiveExternalCatalog_filesDiscovered_Count 0
metrics_local_1573330115306_driver_HiveExternalCatalog_hiveClientCalls_Count 0
metrics_local_1573330115306_driver_HiveExternalCatalog_parallelListingJobCount_Count 0
metrics_local_1573330115306_driver_HiveExternalCatalog_partitionsFetched_Count 0

$ bin/spark-shell --conf spark.metrics.static.sources.enabled=false
$ curl -s http://localhost:4040/metrics/prometheus/ | grep Hive

[dongjoon-hyun]

ok to test

[SparkQA]

Test build #113241 has finished for PR 26320 at commit 57c713f.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[LucaCanali]

Thank you @dongjoon-hyun for reviewing this.

[SparkQA]

Test build #113251 has finished for PR 26320 at commit 7e9e8bb.

• This patch fails PySpark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

Retest this please.

[SparkQA]

Test build #113403 has finished for PR 26320 at commit 7e9e8bb.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #113405 has finished for PR 26320 at commit 5554489.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #113464 has finished for PR 26320 at commit 1a03124.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
• class SourceConfigSuite extends SparkFunSuite with LocalSparkContext

[dongjoon-hyun]

this one too.

[dongjoon-hyun]

should be -> should not be.

[dongjoon-hyun]

Thank you for adding UTs, @LucaCanali .

[SparkQA]

Test build #113473 has finished for PR 26320 at commit 564ec07.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[dongjoon-hyun]

+1, LGTM. Merged to master.

I tested this with http://localhost:4040/metrics/prometheus/ manually like the PR description, too.

[gatorsmile]

Could you explain why they are static and the others are not static?

[dongjoon-hyun]

Apache Spark defines like the following.

private[spark] object StaticSources {
  /**
   * The set of all static sources. These sources may be reported to from any class, including
   * static classes, without requiring reference to a SparkEnv.
   */
  val allSources = Seq(CodegenMetrics, HiveCatalogMetrics)
}

[gatorsmile]

spark.metrics.staticSources.enabled might look better. Using static as the name space looks weird. cc @cloud-fan @Ngone51

[cloud-fan]

staticSources sounds better.

[LucaCanali]

I agree that staticsSources sounds better.
BTW there is probably room for a naming convention for the "switch parameters" for enabling/disabling metrics? Currently we have:
spark.metrics.static.sources.enabled
spark.app.status.metrics.enabled
spark.sql.streaming.metricsEnabled

[LucaCanali]

See also #26692

[green2k]

Is there any chance to have this backported to a minor version of Spark 2.x.x (2.4.7) ?

[dongjoon-hyun]

Since SPARK-29654 is an improvement JIRA, we have no plan for backporting, @green2k . Sorry about that.