Skip to content

Commit 7fe0f3f

Browse files
sarutakpwendell
sarutak
authored and
pwendell
committed
[SPARK-3468] [WEBUI] Timeline-View feature
I sometimes trouble-shoot and analyse the cause of long time spending job. At the time, I find the stages which spends long time or fails, then I find the tasks which spends long time or fails, next I analyse the proportion of each phase in a task. Another case, I find executors which spends long time for running a task and analyse the details of a task. In such situation, I think it's helpful to visualize timeline for each application, job, task and the details of proportion of activity for each task. I added 3 timeline features to existing Web UI. [Application Timeline View] This view shows following things. * When each executor was added/removed and the reason why it's removed. * When each job was started/finished. * Status of each job. ![screenshot from 2015-04-01 16 49 25](https://cloud.githubusercontent.com/assets/4736016/6936886/e35fd582-d891-11e4-980d-8de13f50e442.png) [Stage Timeline View] Similar to Application Timeline View, this view shows following things. * When each executor was added/removed and the reason why it's removed. * When each job was started/finished. * Status of each stage. ![screenshot from 2015-04-01 16 50 59](https://cloud.githubusercontent.com/assets/4736016/6936900/0dca6526-d892-11e4-84a8-efd9037af444.png) [Task Assignment Timeline View] This view shows following things. * When each task started/finished * How long each task spent and the proportion. * Status of each task. * Where each task ran on. ![screenshot from 2015-04-01 16 51 54](https://cloud.githubusercontent.com/assets/4736016/6936910/20fd5acc-d892-11e4-9018-80e463881fc2.png) All the view above is zoomable by mouse wheel action and scrollable by drag action. Author: Kousuke Saruta <[email protected]> Closes #2342 from sarutak/timeline-viewer-feature and squashes the following commits: 11fe67d [Kousuke Saruta] Fixed conflict 79ac03d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature a91abd3 [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature ef34a5b [Kousuke Saruta] Implement tooltip using bootstrap b09d0c5 [Kousuke Saruta] Move `stroke` and `fill` attribute of rect elements to css d3c63c8 [Kousuke Saruta] Fixed a little bit bugs a36291b [Kousuke Saruta] Merge branch 'master' of https://github.com/apache/spark into timeline-viewer-feature 28714b6 [Kousuke Saruta] Fixed highlight issue 0dc4278 [Kousuke Saruta] Addressed most of Patrics's feedbacks 8110acf [Kousuke Saruta] Added scroll limit to Job timeline 974a64a [Kousuke Saruta] Removed unused function ee7a7f0 [Kousuke Saruta] Refactored 6a91872 [Kousuke Saruta] Temporary commit 6693f34 [Kousuke Saruta] Added link to job/stage box in the timeline in order to move to corresponding row when we click 8f88222 [Kousuke Saruta] Added job/stage description aeed4b1 [Kousuke Saruta] Removed stage timeline fc1696c [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature 999ccd4 [Kousuke Saruta] Improved scalability 0fc6a31 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 19815ae [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 68b7540 [Kousuke Saruta] Merge branch 'timeline-viewer-feature' of github.com:sarutak/spark into timeline-viewer-feature 52b5f0b [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature dec85db [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature fcdab7d [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature dab7cc1 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature 09cce97 [Kousuke Saruta] Cleanuped 16f82cf [Kousuke Saruta] Cleanuped 9fb522e [Kousuke Saruta] Cleanuped d05f2c2 [Kousuke Saruta] Merge branch 'master' of git://git.apache.org/spark into timeline-viewer-feature e85e9aa [Kousuke Saruta] Cleanup: Added TimelineViewUtils.scala a76e569 [Kousuke Saruta] Removed unused setting in timeline-view.css 5ce1b21 [Kousuke Saruta] Added vis.min.js, vis.min.css and vis.map to .rat-exclude 082f709 [Kousuke Saruta] Added Timeline-View feature for Applications, Jobs and Stages
1 parent c24aeb6 commit 7fe0f3f

File tree

15 files changed

+785
-32
lines changed

15 files changed

+785
-32
lines changed

‎.rat-excludes‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -30,6 +30,9 @@ log4j-defaults.properties
3030
bootstrap-tooltip.js
3131
jquery-1.11.1.min.js
3232
sorttable.js
33+
vis.min.js
34+
vis.min.css
35+
vis.map
3336
.*avsc
3437
.*txt
3538
.*json

‎core/src/main/resources/org/apache/spark/ui/static/timeline-view.css‎

Lines changed: 182 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
div#application-timeline, div#job-timeline {
19+
margin-bottom: 30px;
20+
}
21+
22+
#application-timeline div.legend-area {
23+
margin-top: 5px;
24+
}
25+
26+
.vis.timeline div.content {
27+
width: 100%;
28+
}
29+
30+
.vis.timeline .item.stage {
31+
cursor: pointer;
32+
}
33+
34+
.vis.timeline .item.stage.succeeded {
35+
background-color: #D5DDF6;
36+
}
37+
38+
.vis.timeline .item.stage.succeeded.selected {
39+
background-color: #D5DDF6;
40+
border-color: #97B0F8;
41+
z-index: auto;
42+
}
43+
44+
.legend-area rect.completed-stage-legend {
45+
fill: #D5DDF6;
46+
stroke: #97B0F8;
47+
}
48+
49+
.vis.timeline .item.stage.failed {
50+
background-color: #FF5475;
51+
}
52+
53+
.vis.timeline .item.stage.failed.selected {
54+
background-color: #FF5475;
55+
border-color: #97B0F8;
56+
z-index: auto;
57+
}
58+
59+
.legend-area rect.failed-stage-legend {
60+
fill: #FF5475;
61+
stroke: #97B0F8;
62+
}
63+
64+
.vis.timeline .item.stage.running {
65+
background-color: #FDFFCA;
66+
}
67+
68+
.vis.timeline .item.stage.running.selected {
69+
background-color: #FDFFCA;
70+
border-color: #97B0F8;
71+
z-index: auto;
72+
}
73+
74+
.legend-area rect.active-stage-legend {
75+
fill: #FDFFCA;
76+
stroke: #97B0F8;
77+
}
78+
79+
.vis.timeline .item.job {
80+
cursor: pointer;
81+
}
82+
83+
.vis.timeline .item.job.succeeded {
84+
background-color: #D5DDF6;
85+
}
86+
87+
.vis.timeline .item.job.succeeded.selected {
88+
background-color: #D5DDF6;
89+
border-color: #97B0F8;
90+
z-index: auto;
91+
}
92+
93+
.legend-area rect.succeeded-job-legend {
94+
fill: #D5DDF6;
95+
stroke: #97B0F8;
96+
}
97+
98+
.vis.timeline .item.job.failed {
99+
background-color: #FF5475;
100+
}
101+
102+
.vis.timeline .item.job.failed.selected {
103+
background-color: #FF5475;
104+
border-color: #97B0F8;
105+
z-index: auto;
106+
}
107+
108+
.legend-area rect.failed-job-legend {
109+
fill: #FF5475;
110+
stroke: #97B0F8;
111+
}
112+
113+
.vis.timeline .item.job.running {
114+
background-color: #FDFFCA;
115+
}
116+
117+
.vis.timeline .item.job.running.selected {
118+
background-color: #FDFFCA;
119+
border-color: #97B0F8;
120+
z-index: auto;
121+
}
122+
123+
.legend-area rect.running-job-legend {
124+
fill: #FDFFCA;
125+
stroke: #97B0F8;
126+
}
127+
128+
.vis.timeline .item.executor.added {
129+
background-color: #D5DDF6;
130+
}
131+
132+
.legend-area rect.executor-added-legend {
133+
fill: #D5DDF6;
134+
stroke: #97B0F8;
135+
}
136+
137+
.vis.timeline .item.executor.removed {
138+
background-color: #EBCA59;
139+
}
140+
141+
.legend-area rect.executor-removed-legend {
142+
fill: #EBCA59;
143+
stroke: #97B0F8;
144+
}
145+
146+
.vis.timeline .item.executor.selected {
147+
border-color: #FFC200;
148+
background-color: #FFF785;
149+
z-index: 2;
150+
}
151+
152+
tr.corresponding-item-hover>td, tr.corresponding-item-hover>th {
153+
background-color: #FFE1FA !important;
154+
}
155+
156+
#application-timeline.collapsed {
157+
display: none;
158+
}
159+
160+
#job-timeline.collapsed {
161+
display: none;
162+
}
163+
164+
.control-panel {
165+
margin-bottom: 5px;
166+
}
167+
168+
span.expand-application-timeline, span.expand-job-timeline {
169+
cursor: pointer;
170+
}
171+
172+
.control-panel input+span {
173+
cursor: pointer;
174+
}
175+
176+
.vis.timeline .item.range .content {
177+
position: unset;
178+
}
179+
180+
.vis.timeline .item .tooltip-inner {
181+
max-width: unset !important;
182+
}

‎core/src/main/resources/org/apache/spark/ui/static/timeline-view.js‎

Lines changed: 168 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,168 @@
1+
/*
2+
* Licensed to the Apache Software Foundation (ASF) under one or more
3+
* contributor license agreements. See the NOTICE file distributed with
4+
* this work for additional information regarding copyright ownership.
5+
* The ASF licenses this file to You under the Apache License, Version 2.0
6+
* (the "License"); you may not use this file except in compliance with
7+
* the License. You may obtain a copy of the License at
8+
*
9+
* http://www.apache.org/licenses/LICENSE-2.0
10+
*
11+
* Unless required by applicable law or agreed to in writing, software
12+
* distributed under the License is distributed on an "AS IS" BASIS,
13+
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
14+
* See the License for the specific language governing permissions and
15+
* limitations under the License.
16+
*/
17+
18+
function drawApplicationTimeline(groupArray, eventObjArray, startTime) {
19+
var groups = new vis.DataSet(groupArray);
20+
var items = new vis.DataSet(eventObjArray);
21+
var container = $("#application-timeline")[0];
22+
var options = {
23+
groupOrder: function(a, b) {
24+
return a.value - b.value
25+
},
26+
editable: false,
27+
showCurrentTime: false,
28+
min: startTime,
29+
zoomable: false
30+
};
31+
32+
var applicationTimeline = new vis.Timeline(container);
33+
applicationTimeline.setOptions(options);
34+
applicationTimeline.setGroups(groups);
35+
applicationTimeline.setItems(items);
36+
37+
setupZoomable("#application-timeline-zoom-lock", applicationTimeline);
38+
setupExecutorEventAction();
39+
40+
function setupJobEventAction() {
41+
$(".item.range.job.application-timeline-object").each(function() {
42+
var getJobId = function(baseElem) {
43+
var jobIdText = $($(baseElem).find(".application-timeline-content")[0]).text();
44+
var jobId = jobIdText.match("\\(Job (\\d+)\\)")[1];
45+
return jobId;
46+
};
47+
48+
$(this).click(function() {
49+
window.location.href = "job/?id=" + getJobId(this);
50+
});
51+
52+
$(this).hover(
53+
function() {
54+
$("#job-" + getJobId(this)).addClass("corresponding-item-hover");
55+
$($(this).find("div.application-timeline-content")[0]).tooltip("show");
56+
},
57+
function() {
58+
$("#job-" + getJobId(this)).removeClass("corresponding-item-hover");
59+
$($(this).find("div.application-timeline-content")[0]).tooltip("hide");
60+
}
61+
);
62+
});
63+
}
64+
65+
setupJobEventAction();
66+
67+
$("span.expand-application-timeline").click(function() {
68+
$("#application-timeline").toggleClass('collapsed');
69+
70+
// Switch the class of the arrow from open to closed.
71+
$(this).find('.expand-application-timeline-arrow').toggleClass('arrow-open');
72+
$(this).find('.expand-application-timeline-arrow').toggleClass('arrow-closed');
73+
});
74+
}
75+
76+
function drawJobTimeline(groupArray, eventObjArray, startTime) {
77+
var groups = new vis.DataSet(groupArray);
78+
var items = new vis.DataSet(eventObjArray);
79+
var container = $('#job-timeline')[0];
80+
var options = {
81+
groupOrder: function(a, b) {
82+
return a.value - b.value;
83+
},
84+
editable: false,
85+
showCurrentTime: false,
86+
min: startTime,
87+
zoomable: false,
88+
};
89+
90+
var jobTimeline = new vis.Timeline(container);
91+
jobTimeline.setOptions(options);
92+
jobTimeline.setGroups(groups);
93+
jobTimeline.setItems(items);
94+
95+
setupZoomable("#job-timeline-zoom-lock", jobTimeline);
96+
setupExecutorEventAction();
97+
98+
function setupStageEventAction() {
99+
$(".item.range.stage.job-timeline-object").each(function() {
100+
var getStageIdAndAttempt = function(baseElem) {
101+
var stageIdText = $($(baseElem).find(".job-timeline-content")[0]).text();
102+
var stageIdAndAttempt = stageIdText.match("\\(Stage (\\d+\\.\\d+)\\)")[1].split(".");
103+
return stageIdAndAttempt;
104+
};
105+
106+
$(this).click(function() {
107+
var idAndAttempt = getStageIdAndAttempt(this);
108+
var id = idAndAttempt[0];
109+
var attempt = idAndAttempt[1];
110+
window.location.href = "../../stages/stage/?id=" + id + "&attempt=" + attempt;
111+
});
112+
113+
$(this).hover(
114+
function() {
115+
var idAndAttempt = getStageIdAndAttempt(this);
116+
var id = idAndAttempt[0];
117+
var attempt = idAndAttempt[1];
118+
$("#stage-" + id + "-" + attempt).addClass("corresponding-item-hover");
119+
$($(this).find("div.job-timeline-content")[0]).tooltip("show");
120+
},
121+
function() {
122+
var idAndAttempt = getStageIdAndAttempt(this);
123+
var id = idAndAttempt[0];
124+
var attempt = idAndAttempt[1];
125+
$("#stage-" + id + "-" + attempt).removeClass("corresponding-item-hover");
126+
$($(this).find("div.job-timeline-content")[0]).tooltip("hide");
127+
}
128+
);
129+
});
130+
}
131+
132+
setupStageEventAction();
133+
134+
$("span.expand-job-timeline").click(function() {
135+
$("#job-timeline").toggleClass('collapsed');
136+
137+
// Switch the class of the arrow from open to closed.
138+
$(this).find('.expand-job-timeline-arrow').toggleClass('arrow-open');
139+
$(this).find('.expand-job-timeline-arrow').toggleClass('arrow-closed');
140+
});
141+
}
142+
143+
function setupExecutorEventAction() {
144+
$(".item.box.executor").each(function () {
145+
$(this).hover(
146+
function() {
147+
$($(this).find(".executor-event-content")[0]).tooltip("show");
148+
},
149+
function() {
150+
$($(this).find(".executor-event-content")[0]).tooltip("hide");
151+
}
152+
);
153+
});
154+
}
155+
156+
function setupZoomable(id, timeline) {
157+
$(id + '>input[type="checkbox"]').click(function() {
158+
if (this.checked) {
159+
timeline.setOptions({zoomable: false});
160+
} else {
161+
timeline.setOptions({zoomable: true});
162+
}
163+
});
164+
165+
$(id + ">span").click(function() {
166+
$(this).parent().find('input:checkbox').trigger('click');
167+
});
168+
}

‎core/src/main/resources/org/apache/spark/ui/static/vis.map‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

0 commit comments

Comments
 (0)