Skip to content

[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly #28880

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
rajatahujaatinmobi wants to merge 9 commits into from
Closed

[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.ui.port) property if mentioned explicitly #28880

Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ce4b6ea
[SPARK-32039][YARN][WEBUI][Master] Adding Check to not to set UI port…
Jun 21, 2020
10b9f7e
[SPARK-32039][YARN][WEBUI][Master] Adding Check to not to set UI port…
Jun 21, 2020
b42c3ed
[SPARK-32021][SQL] Increase precision of seconds and fractions of `ma…
Jun 22, 2020
2703edb
[SPARK-32039][YARN][WEBUI][Master] Adding Check to not to set UI port…
Jun 22, 2020
8fe0aa9
[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.…
Jun 27, 2020
bee8b21
[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.…
Jun 27, 2020
24ecfcf
Rebasing with Upstream
Jun 30, 2020
2b190c4
[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.…
Jun 30, 2020
1896905
[SPARK-29465][YARN][WEBUI] Adding Check to not to set UI port (spark.…
Jun 30, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,9 +211,13 @@ private[spark] class ApplicationMaster(
final def run(): Int = {
try {
val attemptID = if (isClusterMode) {
// Set the web ui port to be ephemeral for yarn so we don't conflict with
// other spark processes running on the same box
System.setProperty(UI_PORT.key, "0")
// Set the web ui port to be ephemeral for yarn if not set explicitly
// so we don't conflict with other spark processes running on the same box
Copy link
Contributor

@tgravescs tgravescs Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets update the comment to say if explicitly set, it acts as a port range based on spark.port.maxRetries so make sure that is sufficiently large for your setup to avoid port conflicts.

Copy link
Author

@rajatahujaatinmobi rajatahujaatinmobi Jun 27, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tgravescs Updated my comment. If that looks okay, can we merge this request ? And I would like to backport this change until spark 2.4

// If set explicitly, Web UI will attempt to run on UI_PORT and try
// incrementally until UI_PORT + `spark.port.maxRetries`
if (System.getProperty(UI_PORT.key) == null) {
System.setProperty(UI_PORT.key, "0")
}
Copy link
Member

@dongjoon-hyun dongjoon-hyun Jun 21, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi, @rajatahujaatinmobi . While reviewing this PR, I noticed that SPARK-29465 is a previous one for this issue.

Copy link
Contributor

@tgravescs tgravescs Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, please see the discussions there, I think a range of ports would be much better. A single port set on a multi-tenant yarn cluster could easily end up with conflicts. What requirements do you have from Knox, a certain range or ports?
Normally on yarn I would expect you to use the resource manager web ui proxy on a secure cluster so you wouldn't be accessing the UI directly.

Copy link
Author

@rajatahujaatinmobi rajatahujaatinmobi Jun 22, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correct me if I am wrong

As per the discussions that UI port is always get assigned in the range based on spark.port.maxRetries. So we can always mention the starting port and afterward, it will try to start service based in between (starting port, starting port +spark.port.maxRetries). So we are including the range implicitly.

Copying the comment
other port types were spawned in the range (port_mentioned to port_mentioned+spark.port.maxRetries) except the UI port which started on random port.

We have a Knox proxy that accesses UI via RM link. And Knox proxy has a certain range of ports that it can access on the cluster which is (18000-18159). Unless we start web UI on that range, we can not access the UI. and we can achieve above the range if we set the following properties
spark.ui.port=18000
spark.port.maxRetries=159

Hence setting up a spark.ui.port as starting port is enough instead of setting up a random port number in the range.

Copy link
Member

@Ngone51 Ngone51 Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yea, I think yet Spark could utilize the combination of spark.ui.port and spark.port.maxRetries to perform "range port".

What else the "range port" do you refer to? @tgravescs

Copy link
Contributor

@tgravescs tgravescs Jun 23, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes you are correct you can get range that way. the ports originally wouldn't do the retries or at least we didn't allow configuring maxRetries.

Copy link
Author

@rajatahujaatinmobi rajatahujaatinmobi Jun 24, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I blocked ports from 18000 to 18050 so now spark.ui.port can not get port b/w that range. But in Spark properties, I mentioned spark.ui.port as 18018 and spark.port.maxRetries=60 so now spark service will start trying from 18018 port until 18018 + 60 in an increment way but ports are blocked until 18050 so spark.ui.port will pick up 18051 port. @Ngone51

Copy link
Member

@Ngone51 Ngone51 Jun 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see. This is what we expected, right?

Copy link
Author

@rajatahujaatinmobi rajatahujaatinmobi Jun 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's expected. So I think we are good to go unless someone else has any other doubts.
@Ngone51 @tgravescs

Copy link
Contributor

@tgravescs tgravescs Jun 24, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for testing that, please update the comment like my below comment requests.

Copy link
Author

@rajatahujaatinmobi rajatahujaatinmobi Jun 29, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tgravescs I have done that. Can we proceed further?


// Set the master and deploy mode property to match the requested mode.
System.setProperty("spark.master", "yarn")
Expand Down