Skip to content

[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced #7038

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
thegiive wants to merge 4 commits into from
Closed

[SPARK-8656][WebUI] Fix the webUI and JSON API number is not synced #7038

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
8b3b8e8
[SPARK-8656] Fix the webUI and JSON API number is not synced
Jun 26, 2015
431d2b0
Worker List should contain DEAD node also
Jun 26, 2015
2c8ea89
Change some styling and add local variable
Jul 3, 2015
9e54bf0
Change variable name to camel case
Jul 3, 2015
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions core/src/main/scala/org/apache/spark/deploy/JsonProtocol.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -76,12 +76,13 @@ private[deploy] object JsonProtocol {
}

def writeMasterState(obj: MasterStateResponse): JObject = {
val alive_workers = obj.workers.filter(_.isAlive())
Copy link
Contributor

@andrewor14 andrewor14 Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in Spark we generally use camel case, so:

val aliveWorkers = obj.workers.filter(_.state == WorkerState.ALIVE)

Copy link
Author

@thegiive thegiive Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Andrew

WorkerState is org.apache.spark.deploy.master's private object. JsonProtocol cannot resolved WorkerState due to class visibility.This code cannot work unless we change WorkerState code.

I think best way to do that is add a one line method on WorkerInfo

How do you think?

Copy link
Author

@thegiive thegiive Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW, I will change the variable name to camel case without any problem

("url" -> obj.uri) ~
("workers" -> obj.workers.toList.map(writeWorkerInfo)) ~
("cores" -> obj.workers.map(_.cores).sum) ~
("coresused" -> obj.workers.map(_.coresUsed).sum) ~
("memory" -> obj.workers.map(_.memory).sum) ~
("memoryused" -> obj.workers.map(_.memoryUsed).sum) ~
("cores" -> alive_workers.map(_.cores).sum) ~
("coresused" -> alive_workers.map(_.coresUsed).sum) ~
("memory" -> alive_workers.map(_.memory).sum) ~
("memoryused" -> alive_workers.map(_.memoryUsed).sum) ~
("activeapps" -> obj.activeApps.toList.map(writeApplicationInfo)) ~
("completedapps" -> obj.completedApps.toList.map(writeApplicationInfo)) ~
("activedrivers" -> obj.activeDrivers.toList.map(writeDriverInfo)) ~
Expand Down
2 changes: 2 additions & 0 deletions core/src/main/scala/org/apache/spark/deploy/master/WorkerInfo.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -107,4 +107,6 @@ private[spark] class WorkerInfo(
def setState(state: WorkerState.Value): Unit = {
this.state = state
}

def isAlive(): Boolean = this.state == WorkerState.ALIVE
Copy link
Contributor

@andrewor14 andrewor14 Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we probably don't need this since we only use it in 1 place

Copy link
Author

@thegiive thegiive Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi Andrew

Base on same reason on previous. I think best way to do that is add a one line method on WorkerInfo

How do you think?

Copy link
Contributor

@andrewor14 andrewor14 Jul 3, 2015

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah ok, that sounds fine then

}