Skip to content

[SPARK-30689][CORE][FOLLOW-UP] Rename config name of discovery plugin #27689

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Ngone51 wants to merge 1 commit into from
Closed

[SPARK-30689][CORE][FOLLOW-UP] Rename config name of discovery plugin #27689

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion core/src/main/scala/org/apache/spark/internal/config/package.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,7 @@ package object config {
.createOptional

private[spark] val RESOURCES_DISCOVERY_PLUGIN =
ConfigBuilder("spark.resources.discovery.plugin")
ConfigBuilder("spark.resources.discoveryPlugin")
.doc("Comma-separated list of class names implementing" +
"org.apache.spark.api.resource.ResourceDiscoveryPlugin to load into the application." +
"This is for advanced users to replace the resource discovery class with a " +
Expand Down
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/resource/ResourceDiscoveryScriptPlugin.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,7 @@ import java.io.File
import java.util.Optional

import org.apache.spark.{SparkConf, SparkException}
import org.apache.spark.annotation.DeveloperApi
import org.apache.spark.api.resource.ResourceDiscoveryPlugin
import org.apache.spark.internal.Logging
import org.apache.spark.util.Utils.executeAndGetOutput
Expand All @@ -32,6 +33,7 @@ import org.apache.spark.util.Utils.executeAndGetOutput
* If the user specifies custom plugins, this is the last one to be executed and
* throws if the resource isn't discovered.
*/
@DeveloperApi
class ResourceDiscoveryScriptPlugin extends ResourceDiscoveryPlugin with Logging {
Copy link
Member

@HyukjinKwon HyukjinKwon May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just happened to double check these modules. Shall we add @since for APIs ResourceDiscoveryScriptPlugin, ResourceInformation, etc. ?

I remember it was targeted to be APIs for Spark 3.0, and we kind of postponed to Spark 3.1 (?) cc @tgravescs - am I correct? Let's add @since for exposed APIs for Spark 3.0 at least.

Copy link
Contributor

@tgravescs tgravescs May 18, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is 3.0 and is a separate feature from the stage level scheduling that got postponed to 3.1. We can add the @SInCE and its 3.0

override def discoverResource(
request: ResourceRequest,
Expand Down
2 changes: 1 addition & 1 deletion docs/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,7 @@ of the most common options to set are:
</td>
</tr>
<tr>
<td><code>spark.resources.discovery.plugin</code></td>
<td><code>spark.resources.discoveryPlugin</code></td>
<td>org.apache.spark.resource.ResourceDiscoveryScriptPlugin</td>
<td>
Comma-separated list of class names implementing
Expand Down