[SPARK-31756][WEBUI] Add real headless browser support for UI test #28578

sarutak · 2020-05-19T07:06:03Z

What changes were proposed in this pull request?

This PR mainly adds two things.

Real headless browser support for UI test
A test suite using headless Chrome as one instance of those browsers.

Also, for environment where Chrome and Chrome driver is not installed, ChromeUITest tag is added to filter out the test suite.

Why are the changes needed?

In the current master, there are two problems for UI test.

Lots of tests especially JavaScript related ones are done manually.
Appearance is better to be confirmed by our eyes but logic should be tested by test cases ideally.
Compared to the real web browsers, HtmlUnit doesn't seem to support JavaScript enough.
I added a JavaScript related test before for SPARK-31534 using HtmlUnit which is simple library based headless browser for test.
The test I added works somehow but some JavaScript related error is shown in unit-tests.log.

======= EXCEPTION START ========
Exception class=[net.sourceforge.htmlunit.corejs.javascript.JavaScriptException]
com.gargoylesoftware.htmlunit.ScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:904)
        at net.sourceforge.htmlunit.corejs.javascript.Context.call(Context.java:628)
        at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.call(ContextFactory.java:515)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:835)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine.callFunction(JavaScriptEngine.java:807)
        at com.gargoylesoftware.htmlunit.InteractivePage.executeJavaScriptFunctionIfPossible(InteractivePage.java:216)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptFunctionJob.runJavaScript(JavaScriptFunctionJob.java:52)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptExecutionJob.run(JavaScriptExecutionJob.java:102)
        at com.gargoylesoftware.htmlunit.javascript.background.JavaScriptJobManagerImpl.runSingleJob(JavaScriptJobManagerImpl.java:426)
        at com.gargoylesoftware.htmlunit.javascript.background.DefaultJavaScriptExecutor.run(DefaultJavaScriptExecutor.java:157)
        at java.lang.Thread.run(Thread.java:748)
Caused by: net.sourceforge.htmlunit.corejs.javascript.JavaScriptException: Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)". (http://192.168.1.209:60724/static/jquery-3.4.1.min.js#2)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpretLoop(Interpreter.java:1009)
        at net.sourceforge.htmlunit.corejs.javascript.Interpreter.interpret(Interpreter.java:800)
        at net.sourceforge.htmlunit.corejs.javascript.InterpretedFunction.call(InterpretedFunction.java:105)
        at net.sourceforge.htmlunit.corejs.javascript.ContextFactory.doTopCall(ContextFactory.java:413)
        at com.gargoylesoftware.htmlunit.javascript.HtmlUnitContextFactory.doTopCall(HtmlUnitContextFactory.java:252)
        at net.sourceforge.htmlunit.corejs.javascript.ScriptRuntime.doTopCall(ScriptRuntime.java:3264)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$4.doRun(JavaScriptEngine.java:828)
        at com.gargoylesoftware.htmlunit.javascript.JavaScriptEngine$HtmlUnitContextAction.run(JavaScriptEngine.java:889)
        ... 10 more
JavaScriptException value = Error: TOOLTIP: Option "sanitizeFn" provided type "window" but expected type "(null|function)".
== CALLING JAVASCRIPT ==
  function () {
      throw e;
  }
======= EXCEPTION END ========

I tried to upgrade HtmlUnit to 2.40.0 but what is worse, the test become not working even though it works on real browsers like Chrome, Safari and Firefox without error.

[info] UISeleniumSuite:
[info] - SPARK-31534: text for tooltip should be escaped *** FAILED *** (17 seconds, 745 milliseconds)
[info]   The code passed to eventually never returned normally. Attempted 2 times over 12.910785232 seconds. Last failure message: com.gargoylesoftware.htmlunit.ScriptException: ReferenceError: Assignment to undefined "regeneratorRuntime" in strict mode (http://192.168.1.209:62132/static/vis-timeline-graph2d.min.js#52(Function)#1)

To resolve those problems, it's better to support headless browser for UI test.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

I tested with following patterns. Both Chrome and Chrome driver should be installed to test.

sbt / with chromedriver / include tag (expect to succeed)
build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite"
sbt / with chromedriver / exclude tag (expect to be ignored)
build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.ui.ChromeUISeleniumSuite -l org.apache.spark.tags.ChromeUITest"
sbt / without chromedriver / include tag (expect to be failed)
build/sbt "testOnly org.apache.spark.ui.ChromeUISeleniumSuite"
sbt / without chromedriver / exclude tag (expect to be skipped)
build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ChromeUITest "testOnly org.apache.spark.ui.ChromeUISeleniumSuite"
Maven / wth chromedriver / include tag (expect to succeed)
build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test
Maven / with chromedriver / exclude tag (expect to be skipped)
build/mvn -Dtest.exclude.tags="org.apache.spark.tags.ChromeUITest" -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test
Maven / without chromedriver / include tag (expect to be failed)
build/mvn -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test
Maven / without chromedriver / exclude tag (expect to be skipped)
build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ChromeUITest -Dtest=none -DwildcardSuites=org.apache.spark.ui.ChromeUISeleniumSuite test

…tests.py

sarutak · 2020-05-19T07:09:01Z

dev/run-tests.py

    print("[info] Found the following changed modules:",
          ", ".join(x.name for x in changed_modules))

+    excluded_tags.extend(always_excluded_tags)


For environments where Chrome and Chrome driver is not installed, including CI environment, tests tagged as ChromeUITest is filtered out when tests are run through run-tests.py.

SparkQA · 2020-05-19T10:25:45Z

Test build #122835 has finished for PR 28578 at commit e701dc9.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2020-05-20T03:11:21Z

Test build #122863 has finished for PR 28578 at commit 6ccbc39.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

srowen · 2020-05-20T14:21:05Z

Seems OK to me.

HyukjinKwon · 2020-05-21T09:55:16Z

cc @gengliangwang FYI. Looks fine to me too

gengliangwang · 2020-05-21T18:47:37Z

@sarutak seems there is code conflict after #28585 is merged.

srowen · 2020-05-22T13:24:41Z

Merged to master

dongjoon-hyun · 2020-05-23T23:25:45Z

Hi, All.
This seems to break Hadoop 3.2/Hive 2.3 profile Jenkins Maven jobs in master branch.

ChromeUISeleniumSuite:
org.apache.spark.ui.ChromeUISeleniumSuite *** ABORTED ***
  java.lang.IllegalStateException: The driver executable does not exist: /home/jenkins/workspace/spark-master-test-maven-hadoop-3.2-hive-2.3@2/core/null
  at com.google.common.base.Preconditions.checkState(Preconditions.java:176)
  at org.openqa.selenium.remote.service.DriverService.checkExecutable(DriverService.java:121)
  at org.openqa.selenium.remote.service.DriverService.findExecutable(DriverService.java:116)
  at org.openqa.selenium.chrome.ChromeDriverService.access$000(ChromeDriverService.java:32)
  at org.openqa.selenium.chrome.ChromeDriverService$Builder.findDefaultExecutable(ChromeDriverService.java:137)
  at org.openqa.selenium.remote.service.DriverService$Builder.build(DriverService.java:296)
  at org.openqa.selenium.chrome.ChromeDriverService.createDefaultService(ChromeDriverService.java:88)
  at org.openqa.selenium.chrome.ChromeDriver.<init>(ChromeDriver.java:148)
  at org.apache.spark.ui.ChromeUISeleniumSuite.beforeAll(ChromeUISeleniumSuite.scala:37)
  at org.scalatest.BeforeAndAfterAll.liftedTree1$1(BeforeAndAfterAll.scala:212)

Hadoop 3.2 / Hive 2.3 / JDK11 Jenkins Maven job is also failing together consistently.

https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-master-test-maven-hadoop-3.2-hive-2.3-jdk-11/

sarutak · 2020-05-23T23:35:45Z

Ah, some profiles directly run tests without run-tests.py.
It might be better to disable the new suite by default.

dongjoon-hyun · 2020-05-23T23:36:15Z

Thanks for taking a look at that. Could you make a follow-up PR, @sarutak ?

sarutak · 2020-05-23T23:38:20Z

I'll try it.

dongjoon-hyun · 2020-05-23T23:57:06Z

After investigating more, it seems to break all Maven jobs. The following Hadoop 2.7 jobs are broken too.

In short, this PR only passes SBT environment.

dongjoon-hyun · 2020-05-23T23:59:34Z

Since it's a weekend, can we simple revert this first to recover the master branch first? Then, we can make a PR with [test-maven] title to make it sure at PR stage.

sarutak · 2020-05-24T00:02:18Z

Agree. I'll make a revert PR.

dongjoon-hyun · 2020-05-24T00:14:57Z

Thank you so much.

dongjoon-hyun · 2020-05-24T00:15:32Z

You can revert directly since this commit is pretty new.

dongjoon-hyun · 2020-05-24T00:16:17Z

Thank you. I saw the commit.

8441e93

…ver tests ### What changes were proposed in this pull request? This PR adds two things. Real headless browser support for HistoryServer tests. A test suite using headless Chrome as one instance of those browsers. ### Why are the changes needed? The motivation is same as #28578 . In the current master, there is a testcase for HistoryServer which uses Ajax so we need the support for HistoryServer tests. Also this change is necessary to upgrade HtmlUnit (See #28585) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? I tested with following patterns. Both Chrome and Chrome driver should be installed to test. 1. sbt / with default excluded tags (ChromeUIHistoryServerSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/sbt -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 2. sbt / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/sbt -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 3. sbt / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/sbt -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver "testOnly org.apache.spark.deploy.history.ChromeUIHistoryServerSuite org.apache.spark.sql.SQLQueryTestSuite" 4. Maven / with default excluded tags (ChromeUIHistoryServerSuite is expected to be skipped and SQLQueryTestSuite is expected to succeed) `build/mvn -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` 5. Maven / overwrite default excluded tags as empty string (Both suites are expected to succeed) `build/mvn -Dtest.default.exclude.tags= -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` 6. Maven / set `test.exclude.tags` to `org.apache.spark.tags.ExtendedSQLTest` (Both suites are expected to be skipped) `build/mvn -Dtest.exclude.tags=org.apache.spark.tags.ExtendedSQLTest -Dspark.test.webdriver.chrome.driver=/path/to/chromedriver -Dtest=none -DwildcardSuites=org.apache.spark.deploy.history.ChromeUIHistoryServerSuite,org.apache.spark.sql.SQLQueryTestSuite test` Closes #28622 from sarutak/headless-browser-support-for-historyserver. Authored-by: Kousuke Saruta <[email protected]> Signed-off-by: Sean Owen <[email protected]>

sarutak added 5 commits May 19, 2020 14:11

Add RealBrowserUISeleniumSuite.scala

424092a

Add ChromeUITest.java

4fe3e80

Add ChromeUISeeniumSuite.scala

760eb0a

Remove JavaScript related tests from UISeleniumSuite.scala

c2f6a6d

Disable tests taged as 'ChromeUITest' when tests are run through run-…

e701dc9

…tests.py

probot-autolabeler bot added BUILD CORE WEB UI labels May 19, 2020

sarutak commented May 19, 2020

View reviewed changes

Fixed RealBrowserUISeleniumSuite.

6ccbc39

sarutak mentioned this pull request May 20, 2020

[SPARK-31765][WEBUI][test-maven] Upgrade HtmlUnit >= 2.37.0 #28585

Closed

srowen closed this in d955708 May 22, 2020

sarutak mentioned this pull request May 23, 2020

[SPARK-31804][WEBUI] Add real headless browser support for HistoryServer tests #28622

Closed

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28578

[SPARK-31756][WEBUI] Add real headless browser support for UI test #28578

Uh oh!

Conversation

sarutak commented May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

sarutak May 19, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SparkQA commented May 19, 2020

Uh oh!

SparkQA commented May 20, 2020

Uh oh!

srowen commented May 20, 2020

Uh oh!

HyukjinKwon commented May 21, 2020

Uh oh!

gengliangwang commented May 21, 2020

Uh oh!

srowen commented May 22, 2020

Uh oh!

dongjoon-hyun commented May 23, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sarutak commented May 23, 2020

Uh oh!

dongjoon-hyun commented May 23, 2020

Uh oh!

sarutak commented May 23, 2020

Uh oh!

dongjoon-hyun commented May 23, 2020

Uh oh!

dongjoon-hyun commented May 23, 2020

Uh oh!

sarutak commented May 24, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dongjoon-hyun commented May 24, 2020

Uh oh!

dongjoon-hyun commented May 24, 2020

Uh oh!

dongjoon-hyun commented May 24, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

sarutak commented May 19, 2020 •

edited

Loading

sarutak May 19, 2020 •

edited

Loading

dongjoon-hyun commented May 23, 2020 •

edited

Loading

sarutak commented May 24, 2020 •

edited

Loading