[APM] e2e are failing when running them locally with some `snapshot`

[v1v]

Kibana version:
master branch

commit a108b23c27635ee90382f86de2cc8206bdb18d8d (grafted, HEAD -> master, origin/master, origin/HEAD)
Author: Ryan Keairns <contactryank@gmail.com>
Date:   Wed May 6 01:39:29 2020 -0500

Elasticsearch version:
master branch

{
name: "122804b8ce33",
cluster_name: "docker-cluster",
cluster_uuid: "v-rouCyHShWZI5Ykl-t4qw",
version: {
number: "8.0.0-SNAPSHOT",
build_flavor: "default",
build_type: "docker",
build_hash: "84116c5132e7ea5a1e63b6a4c4736ea0a7111e4f",
build_date: "2020-05-06T03:08:57.223715Z",
build_snapshot: true,
lucene_version: "8.5.1",
minimum_wire_compatibility_version: "7.8.0",
minimum_index_compatibility_version: "7.0.0"
},
tagline: "You Know, for Search"
}

Server OS version:
macosx 10.15.4

Browser version:
Cypress e2e UI

Browser OS version:
N/A

Original install method (e.g. download page, yum, from source, etc.):
From source
node --version - v10.19.0
nvm --version - 0.34.0

Description of the problem including expected versus actual behavior:

e2e tests are failing with

Steps to reproduce:

Run all the below steps from scratch:

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
docker ps -a
cd kibana
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

Since there is an issue when running from scratch, see #61942
I recommend to change the x-pack/plugins/apm/e2e/run-e2e.sh with something like the below diff:

diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..aa1aecd8 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -111,15 +111,26 @@ fi
 ##################################################
 echo "\n${bold}Waiting for Kibana to start...${normal}"
 echo "Note: you need to start Kibana manually. Find the instructions at the top."
-yarn wait-on -i 500 -w 500 http://localhost:$KIBANA_PORT > /dev/null
+yarn wait-on -i 500 -w 500 http://localhost:$KIBANA_PORT/status > /dev/null
 
 echo "\n✅ Setup completed successfully. Running tests...\n"
 
 #
-# run cypress tests
+# run cypress tests a few times
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
-
+n=0
+until [ $n -ge 20 ]
+  do
+    ## for debugging purposes only
+    curl --silent --user admin:changeme \
+          "localhost:${KIBANA_PORT}/api/status" | \
+            jq .status.overall.state | \
+            grep 'green'
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=$[$n+1]
+  done
+fi

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="0" tests="1" failures="1">
  <testsuite name="Root Suite" timestamp="2020-05-06T07:39:14" tests="0" file="cypress/integration/apm.feature" failures="0" time="0">
  </testsuite>
  <testsuite name="APM" timestamp="2020-05-06T07:39:14" tests="1" failures="1" time="0">
    <testcase name="APM Transaction duration charts" time="0" classname="Transaction duration charts">
      <failure message="Snapshot difference. To update, delete snapshot and rerun test.
&quot;500 ms&quot; =&gt; &quot;350 ms&quot;" type="Error"><![CDATA[Error: Snapshot difference. To update, delete snapshot and rerun test.
"500 ms" => "350 ms"
    at http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22407:15
    at InternalConstructor.orElse (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:2433:14)
    at cyRaiser (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22396:14)
    at setOrCheckValue (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22583:7)
    at snapShotCore (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22595:14)
    at setSnapshot (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22412:5)
    at Context.snapshot (http://localhost:5701/__cypress/tests?p=cypress/support/index.ts-869:22431:5)]]></failure>
    </testcase>
  </testsuite>
</testsuites>

Describe the feature:

[elasticmachine]

Pinging @elastic/apm-ui (Team:apm)

[sorenlouv]

Try deleting the snapshots.js file. I think something could have happened to that during the NP migration.

[v1v]

Do you mean x-pack/plugins/apm/e2e/cypress/integration/snapshots.js ? If so I did delete it and the error is now something else:

What did I do?

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
cd kibana

## Apply patch
cat <<EOT > tweak.patch
diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
deleted file mode 100644
index a462f4a5..00000000
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ /dev/null
@@ -1,16 +0,0 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-
-module.exports = {
-  APM: {
-    'Transaction duration charts': {
-      '1': '500 ms',
-      '2': '250 ms',
-      '3': '0 ms'
-    }
-  },
-  __version: '4.2.0'
-};
diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..7aad1412 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -118,7 +118,12 @@ echo "\n✅ Setup completed successfully. Running tests...\n"
 #
 # run cypress tests
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
+n=0
+until [  -ge 20 ] ; do
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=\$[\$n+1]
+done
 
 #
 # Run interactively
EOT

git apply  --stat tweak.patch 
git apply tweak.patch 
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

[sorenlouv]

What happens if you create it as an empty file:

module.exports = {};

[v1v]

That's interesting, I actually created the file with the empty content as suggested, and after running the scripts, the file got changed as below:

diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
index dc164f3e..07782866 100644
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
@@ -1 +1,10 @@
-module.exports = { }
\ No newline at end of file
+module.exports = {
+  "__version": "4.2.0",
+  "APM": {
+    "Transaction duration charts": {
+      "1": "350 ms",
+      "2": "175 ms",
+      "3": "0 ms"
+    }
+  }
+}

It works with the retry tweak after 6 tries :)

How did I run the e2e?

cd /tmp
git clone https://github.com/elastic/kibana --depth 1
cd kibana

## Apply patch
cat <<EOT > tweak.patch
diff --git a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
index a462f4a5..dc164f3e 100644
--- a/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
+++ b/x-pack/plugins/apm/e2e/cypress/integration/snapshots.js
@@ -1,16 +1 @@
-/*
- * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
- * or more contributor license agreements. Licensed under the Elastic License;
- * you may not use this file except in compliance with the Elastic License.
- */
-
-module.exports = {
-  APM: {
-    'Transaction duration charts': {
-      '1': '500 ms',
-      '2': '250 ms',
-      '3': '0 ms'
-    }
-  },
-  __version: '4.2.0'
-};
+module.exports = { }
\ No newline at end of file
diff --git a/x-pack/plugins/apm/e2e/run-e2e.sh b/x-pack/plugins/apm/e2e/run-e2e.sh
index 818d45ab..c6fceaaf 100755
--- a/x-pack/plugins/apm/e2e/run-e2e.sh
+++ b/x-pack/plugins/apm/e2e/run-e2e.sh
@@ -118,7 +118,12 @@ echo "\n✅ Setup completed successfully. Running tests...\n"
 #
 # run cypress tests
 ##################################################
-yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true
+n=0
+until [ \$n -ge 20 ] ; do
+    yarn cypress run --config pageLoadTimeout=100000,watchForFileChanges=true && break || true
+    sleep 10
+    n=\$[\$n+1]
+done
 
 #
 # Run interactively
EOT

git apply  --stat tweak.patch 
git apply tweak.patch 
nvm use $(cat .node-version)
cd x-pack/plugins/apm/e2e
yarn install
cd -
yarn kbn clean && yarn kbn bootstrap
nohup node ./scripts/kibana --no-base-path --dev --no-dev-config --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml > /tmp/kibana.log 2>&1 &
x-pack/plugins/apm/e2e/run-e2e.sh

Do you need any further details?

[sorenlouv]

after running the scripts, the file got changed as below:

Great, that's also what I'd expect (it should recreate the snapshot).
It's a shame this only happens when the file is empty - it should also happen when the file doesn't exist. But we can look into that another time.

It works with the retry tweak after 6 tries :)

Ouch. Do you know why it failed the first 5? Is it still because we can't reliably determine when Kibana is up and running?
If so, can you try running Kibana in prod mode instead of dev?

node ./scripts/kibana --no-base-path --config x-pack/plugins/apm/e2e/ci/kibana.e2e.yml

[v1v]

Is it still because we can't reliably determine when Kibana is up and running?
If so, can you try running Kibana in prod mode instead of dev?

That's already reported in #61942

A bit of context, I've already tried running Kibana in prod mode instead of dev a few times and did not work:

• 2802f68
• 81f5c92

That's the reason I enabled the retry tweak in #61803 and it does work, the good thing, what people run locally with the e2e it's also validated in the CI, therefore no running discrepancies in the CI and locally.

[sorenlouv]

That's already reported in #61942

I'm not following. Are you saying that the retries are still necessary after #61942 is fixed?

[sorenlouv]

I think we should find out why the retries are necessary so we can get rid of them.

[v1v]

I'll close this done since it's now solved in #66373