Skip to content

[SPARK-16986][WEB-UI] Converter Started, Completed and Last Updated to client time zone in history page #19640

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wangyum wants to merge 11 commits into from
Closed

[SPARK-16986][WEB-UI] Converter Started, Completed and Last Updated to client time zone in history page #19640

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
17bddef
Replace GMT with history server side TimeZone.
wangyum Nov 2, 2017
c7339c8
Support configure time zone.
wangyum Nov 3, 2017
f36023a
Move getTimeZone from SparkHadoopUtil to Util
wangyum Nov 6, 2017
5291d16
getTimeZone -> getHistoryServerTimeZone
wangyum Nov 7, 2017
d9994ea
Add "spark.history.timeZone" to internal/config.
wangyum Nov 8, 2017
7b04e08
Only convert the UI part to client time zone
wangyum Nov 11, 2017
9a40da0
padZeroes
wangyum Nov 13, 2017
cb1db11
Show local time zone
wangyum Nov 14, 2017
4a78965
Fix HistoryServerSuite test error
wangyum Nov 14, 2017
40ec67e
Intl.DateTimeFormat().resolvedOptions().timeZone
wangyum Nov 15, 2017
08f5e68
Intl.DateTimeFormat().resolvedOptions().timeZone
wangyum Nov 15, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,8 @@ private[history] class FsHistoryProvider(conf: SparkConf, clock: Clock)
} else {
Map()
}
Map("Event log directory" -> logDir.toString) ++ safeMode
Map("Event log directory" -> logDir.toString,
"Time zone" -> Utils.getTimeZone.getID) ++ safeMode
}

override def stop(): Unit = {
Expand Down
9 changes: 6 additions & 3 deletions core/src/main/scala/org/apache/spark/status/api/v1/JacksonMessageWriter.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,14 +21,16 @@ import java.lang.annotation.Annotation
import java.lang.reflect.Type
import java.nio.charset.StandardCharsets
import java.text.SimpleDateFormat
import java.util.{Calendar, Locale, SimpleTimeZone}
import java.util.{Calendar, Locale}
import javax.ws.rs.Produces
import javax.ws.rs.core.{MediaType, MultivaluedMap}
import javax.ws.rs.ext.{MessageBodyWriter, Provider}

import com.fasterxml.jackson.annotation.JsonInclude
import com.fasterxml.jackson.databind.{ObjectMapper, SerializationFeature}

import org.apache.spark.util.Utils

/**
* This class converts the POJO metric responses into json, using jackson.
*
Expand Down Expand Up @@ -86,8 +88,9 @@ private[v1] class JacksonMessageWriter extends MessageBodyWriter[Object]{

private[spark] object JacksonMessageWriter {
def makeISODateFormat: SimpleDateFormat = {
val iso8601 = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'GMT'", Locale.US)
val cal = Calendar.getInstance(new SimpleTimeZone(0, "GMT"))
val timeZone = Utils.getTimeZone
val iso8601 = new SimpleDateFormat(s"yyyy-MM-dd'T'HH:mm:ss.SSS'${timeZone.getID}'", Locale.US)
val cal = Calendar.getInstance(timeZone)
iso8601.setCalendar(cal)
iso8601
}
Expand Down
6 changes: 4 additions & 2 deletions core/src/main/scala/org/apache/spark/status/api/v1/SimpleDateParam.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,11 +17,13 @@
package org.apache.spark.status.api.v1

import java.text.{ParseException, SimpleDateFormat}
import java.util.{Locale, TimeZone}
import java.util.Locale
import javax.ws.rs.WebApplicationException
import javax.ws.rs.core.Response
import javax.ws.rs.core.Response.Status

import org.apache.spark.util.Utils

private[v1] class SimpleDateParam(val originalValue: String) {

val timestamp: Long = {
Expand All @@ -31,7 +33,7 @@ private[v1] class SimpleDateParam(val originalValue: String) {
} catch {
case _: ParseException =>
val gmtDay = new SimpleDateFormat("yyyy-MM-dd", Locale.US)
gmtDay.setTimeZone(TimeZone.getTimeZone("GMT"))
gmtDay.setTimeZone(Utils.getTimeZone)
try {
gmtDay.parse(originalValue).getTime()
} catch {
Expand Down
7 changes: 6 additions & 1 deletion core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ import java.nio.ByteBuffer
import java.nio.channels.{Channels, FileChannel}
import java.nio.charset.StandardCharsets
import java.nio.file.{Files, Paths}
import java.util.{Locale, Properties, Random, UUID}
import java.util.{Locale, Properties, Random, TimeZone, UUID}
import java.util.concurrent._
import java.util.concurrent.atomic.AtomicBoolean
import java.util.zip.GZIPInputStream
Expand Down Expand Up @@ -2742,6 +2742,11 @@ private[spark] object Utils extends Logging {
}
}

def getTimeZone: TimeZone = {

@dbolshak dbolshak Nov 6, 2017

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please image that this method will be used outside scope of the current issue?
What's about users who does not have spark history at all?

The name of method, the class and the package does not tell us that this method somehow related ONLY to history server, such API makes me easy to do wrong assumptions.

val sparkConf = new SparkConf(false).loadFromSystemProperties(true)

@jiangxb1987 jiangxb1987 Nov 7, 2017

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make SparkConf as a input param instead of create a new instance for every function call?

TimeZone.getTimeZone(sparkConf.get("spark.history.timeZone", "GMT"))
}

}

private[util] object CallerContext extends Logging {
Expand Down