[SPARK-41365][UI] Stages UI page fails to load for proxy in specific yarn environment #38882

yabola · 2022-12-02T10:10:38Z

What changes were proposed in this pull request?

Stages UI page fails to load for proxy in some specific yarn environment.

Why are the changes needed?

My environment CDH 5.8 , click to enter the spark UI from the yarn Resource Manager page
when visit the stage URI, it fails to load, URI is
http://:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0

The issue is similar to, the final phenomenon of the issue is the same, because the parameter encode twice
SPARK-32467
SPARK-33611

The two issues solve two scenarios to avoid encode twice:

https redirect proxy
set reverse proxy enabled (spark.ui.reverseProxy) in Nginx

But if encode twice due to other reasons, such as this issue (yarn proxy), it will also fail when visit stage page.
It is better to decode parameter twice here.
Just like fix here SPARK-12708 codes

Does this PR introduce any user-facing change?

No

How was this patch tested?

new added UT

…onment

yabola

The change in the second place is necessary, at least the page can be opened normally and code logic has not changed.

yabola · 2022-12-02T10:20:21Z

core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala

+      // Decode URI twice here to avoid percent-encoding twice on the query string
+      val decodeURI = URLDecoder.decode(uriInfo.getRequestUri.getRawQuery, UTF_8.name())
+      val uriQueryParameters = new ImmutableMultivaluedMap[String, String](
+        UriComponent.decodeQuery(decodeURI, true))


Not sure if decoding twice is appropriate here.
If do not change here, the stage page can be opened, but the task sort column button cannot be clicked (locality level etc.)
Given the change before, I think it is reasonable.
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/UIUtils.scala#L626

yabola · 2022-12-02T10:22:29Z

core/src/main/scala/org/apache/spark/status/api/v1/StagesResource.scala

    }
    withUI(_.store.taskList(stageId, stageAttemptId, pageStartIndex, pageLength,
-      indexName(columnNameToSort), isAscendingStr.equalsIgnoreCase("asc")))
+      indexName(columnNameToSort), "asc".equalsIgnoreCase(isAscendingStr)))


isAscendingStr may be NPE if URI was encoded twice.
After change here, at least the stage page can be opened now

yabola · 2022-12-02T10:52:05Z

Stages UI page failed

AmplabJenkins · 2022-12-03T01:29:32Z

Can one of the admins verify this patch?

mridulm · 2022-12-05T05:09:42Z

+CC @gengliangwang

gengliangwang · 2022-12-06T00:30:46Z

@yabola could you add reproduce steps or tests for the issue?

yabola · 2022-12-06T03:39:59Z

@gengliangwang
My hadoop environment is CDH 5.8 ( yarn 2.6.0+cdh5.8.0+1589). It can reproduce easily in any application.
Run the latest spark code using yarn-client mode (I haven't tried other modes yet).
Click ApplicationMaster URL from ResourceManager Web UI (http://:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0)

yabola · 2022-12-06T10:51:16Z

@gengliangwang
I see comments (the last one) from other people in this issue , I think he had the same problem, it is after this issues fixed time

gengliangwang · 2022-12-06T20:46:03Z

@yabola From the screenshot you provided, I don't see any double-quoted URL like %255B....

yabola · 2022-12-07T03:14:46Z

@gengliangwang Yes, but the URI will be processed by yarn proxy (encoded twice). I collect URIInfo in the interface (org.apache.spark.status.api.v1.StagesResource#taskTable).

WEB UI request(screenshot): http://10.1.3.16:8088/proxy/application_1669877165233_0040/api/v1/applications/application_1669877165233_0040/stages/0/0/taskTable?draw=2&order%5B0%5D%5Bcolumn%5D=4&order%5B0%5D%5Bdir%5D=asc&start=0&length=20&search%5Bvalue%5D=&search%5Bregex%5D=false&numTasks=21&columnIndexToSort=4&columnNameToSort=Locality%20Level&_=1670382363732

uriInfo.getRequestUri : http://10.1.3.16:4040/api/v1/applications/application_1669877165233_0040/stages/0/0/taskTable?draw=2&order%255B0%255D%255Bcolumn%255D=4&order%255B0%255D%255Bdir%255D=asc&start=0&length=20&search%255Bvalue%255D=&search%255Bregex%255D=false&numTasks=21&columnIndexToSort=4&columnNameToSort=Locality%2520Level&_=1670382363732

gengliangwang · 2022-12-07T03:43:59Z

@yabola I am not quite sure about the "yarn proxy" you mentioned.
Can we fix the issue in a narrow waist method? IIUC there are also ajax requests in the executor page.

yabola · 2022-12-07T11:47:56Z

@yabola I am not quite sure about the "yarn proxy" you mentioned. Can we fix the issue in a narrow waist method? IIUC there are also ajax requests in the executor page.

@gengliangwang I check Executor page requests don't have query params (not sure about all) , so it work well.
Do you mean find a general method to solve all query parameters ( web Filter etc.)? I tried to find, but I don't have a good idea right now.

gengliangwang · 2022-12-08T07:41:26Z

@yabola ok, can you try adding a new test case for it?

…onment

yabola · 2022-12-09T04:30:39Z

@gengliangwang I add new UT and change codes to carefully decode each parameter, I think it aligns with the previous behavior and is more accurate ( I reuse origin decodeURLParameter codes , It supports multiple decode). How do you feel it?

spark/core/src/main/scala/org/apache/spark/ui/UIUtils.scala

Lines 624 to 637 in b5a9e1f

    
             /** 
        
              * Decode URLParameter if URL is encoded by YARN-WebAppProxyServlet. 
        
              * Due to YARN-2844: WebAppProxyServlet cannot handle urls which contain encoded characters 
        
              * Therefore we need to decode it until we get the real URLParameter. 
        
              */ 
        
             def decodeURLParameter(urlParam: String): String = { 
        
               var param = urlParam 
        
               var decodedParam = URLDecoder.decode(param, UTF_8.name()) 
        
               while (param != decodedParam) { 
        
                 param = decodedParam 
        
                 decodedParam = URLDecoder.decode(param, UTF_8.name()) 
        
               } 
        
               param 
        
             }

yabola · 2022-12-15T11:20:30Z

@gengliangwang If you have time, please have a look, thanks. And do I need to change codes to decode parameters only twice here?

gengliangwang · 2022-12-16T02:38:24Z

@yabola Thanks for the fix. The changes LGTM.
Merging to master.

gengliangwang · 2022-12-16T02:39:50Z

@yabola There is merge conflict on branch-3.3. Could you create a backport PR against branch-3.3?

yabola · 2022-12-16T06:55:54Z

@gengliangwang @mridulm Thank you all~ I had created back port PR #39087

…ific yarn environment backport #38882 ### What changes were proposed in this pull request? Stages UI page fails to load for proxy in some specific yarn environment. ### Why are the changes needed? My environment CDH 5.8 , click to enter the spark UI from the yarn Resource Manager page when visit the stage URI, it fails to load, URI is http://<yarn-url>:8088/proxy/application_1669877165233_0021/stages/stage/?id=0&attempt=0 The issue is similar to, the final phenomenon of the issue is the same, because the parameter encode twice [SPARK-32467](https://issues.apache.org/jira/browse/SPARK-32467) [SPARK-33611](https://issues.apache.org/jira/browse/SPARK-33611) The two issues solve two scenarios to avoid encode twice: 1. https redirect proxy 2. set reverse proxy enabled (spark.ui.reverseProxy) in Nginx But if encode twice due to other reasons, such as this issue (yarn proxy), it will also fail when visit stage page. It is better to decode parameter twice here. Just like fix here [SPARK-12708](https://issues.apache.org/jira/browse/SPARK-12708) [codes](https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/UIUtils.scala#L626) ### Does this PR introduce any user-facing change? No ### How was this patch tested? new added UT Closes #39087 from yabola/fixui-backport. Authored-by: chenliang.lu <marssss2929@gmail.com> Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

SPARK-41365 Stages UI page fails to load for proxy in some yarn envir…

f4b9fee

…onment

github-actions bot added the CORE label Dec 2, 2022

yabola commented Dec 2, 2022

View reviewed changes

yabola changed the title ~~[SPARK-41365][CORE-UI] Stages UI page fails to load for proxy in some yarn environment~~ [SPARK-41365][CORE-UI] Stages UI page fails to load for proxy in specific yarn environment Dec 2, 2022

yabola changed the title ~~[SPARK-41365][CORE-UI] Stages UI page fails to load for proxy in specific yarn environment~~ [SPARK-41365][UI] Stages UI page fails to load for proxy in specific yarn environment Dec 2, 2022

SPARK-41365 Stages UI page fails to load for proxy in some yarn envir…

3fa4aff

…onment

github-actions bot added the WEB UI label Dec 9, 2022

ut fix

5dcceaf

code style

af99841

gengliangwang approved these changes Dec 16, 2022

View reviewed changes

gengliangwang closed this in 9831a09 Dec 16, 2022

yabola mentioned this pull request Dec 16, 2022

[SPARK-41365][UI][3.3] Stages UI page fails to load for proxy in specific yarn environment #39087

Closed

[SPARK-41365][UI] Stages UI page fails to load for proxy in specific yarn environment #38882

[SPARK-41365][UI] Stages UI page fails to load for proxy in specific yarn environment #38882

Uh oh!

Conversation

yabola commented Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

yabola left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yabola Dec 2, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yabola Dec 2, 2022

Choose a reason for hiding this comment

Uh oh!

yabola commented Dec 2, 2022

Uh oh!

AmplabJenkins commented Dec 3, 2022

Uh oh!

mridulm commented Dec 5, 2022

Uh oh!

gengliangwang commented Dec 6, 2022

Uh oh!

yabola commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yabola commented Dec 6, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Dec 6, 2022

Uh oh!

yabola commented Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Dec 7, 2022

Uh oh!

yabola commented Dec 7, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Dec 8, 2022

Uh oh!

yabola commented Dec 9, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

yabola commented Dec 15, 2022

Uh oh!

gengliangwang commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gengliangwang commented Dec 16, 2022

Uh oh!

yabola commented Dec 16, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

yabola commented Dec 2, 2022 •

edited

Loading

yabola left a comment •

edited

Loading

yabola Dec 2, 2022 •

edited

Loading

yabola commented Dec 6, 2022 •

edited

Loading

yabola commented Dec 6, 2022 •

edited

Loading

yabola commented Dec 7, 2022 •

edited

Loading

yabola commented Dec 7, 2022 •

edited

Loading

yabola commented Dec 9, 2022 •

edited

Loading

gengliangwang commented Dec 16, 2022 •

edited

Loading

yabola commented Dec 16, 2022 •

edited

Loading