[SPARK-2687] [yarn]amClient should remove ContainerRequest#3245
[SPARK-2687] [yarn]amClient should remove ContainerRequest#3245lianhuiwang wants to merge 1 commit intoapache:masterfrom
Conversation
|
Test build #23311 has started for PR 3245 at commit
|
|
Test build #23311 has finished for PR 3245 at commit
|
|
Test PASSed. |
There was a problem hiding this comment.
so even though the preferred node locality stuff isn't working right now I think I would rather see us attempt to do the right thing here. If there weren't any preferred nodes specified then taking any of the requests works just fine, but if there were host or rack specific requests we should remove the ones that were actually allocated to us.
There was a problem hiding this comment.
i donot think so. because it is FIFO, we can remove head of request queue because head of queue is allocated by Yarn when we receive allocated Containers.
There was a problem hiding this comment.
When you request containers from yarn and specify hosts or racks then it is not guaranteed that you get them back in the order you requests them. Certain hosts may be busy, not available, etc. You might get a rack request fullfilled instead of a host request, etc. Since the preferred locality stuff is broken right now anyway (SPARK-2089), we can fix this up separately if you would prefer?
There was a problem hiding this comment.
yes,i know what you mean. preferred locality donot work in our application. i will take a look at it.thanks.
|
Hey @lianhuiwang by the way the YARN alpha support is now dropped. It would be good if you can rebase the changes here to master to resolve the merge conflicts manually. |
|
Just noticed this. My PR for SPARK-1714 #3765 also fixes this issue. |
|
ok, i will close this PR. |
6 participants
in https://issues.apache.org/jira/browse/YARN-1902, after receving allocated containers,if amClient donot remove ContainerRequest,RM will continually allocate container for spark AppMaster.@tgravescs