Find a stable setup to run simulation tests locally #1995

sangaman · 2020-11-17T07:15:56Z

I get this every time I try to run simulation tests locally:

=== RUN   TestIntegration
2020/11/17 07:04:10 xud: creating network
2020/11/17 07:04:13 ltcd: launching node...
2020/11/17 07:04:14 ltcd: 200 blocks generated
2020/11/17 07:04:14 lnd-ltc: launching network...
2020/11/17 07:04:27 btcd: launching node...
2020/11/17 07:04:27 btcd: 200 blocks generated
2020/11/17 07:04:27 lnd-btc: launching network...
2020/11/17 07:04:38 connext: launching network...
2020/11/17 07:04:41 xud: launching network...
2020/11/17 07:05:01 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Bob/tls.cert
exit status 1
FAIL	github.com/ExchangeUnion/xud-simulation	51.112s

docker-run.sh: test failed, re-running... (1/3)

Starting simulation_gomod_1   ... done
Starting simulation_btcd_1    ... done
Starting simulation_connext_1 ... done
Starting simulation_lnd_1     ... done
Starting simulation_xud_1     ... done
=== RUN   TestIntegration
2020/11/17 07:05:31 xud: creating network
2020/11/17 07:05:34 ltcd: launching node...
2020/11/17 07:05:35 ltcd: 200 blocks generated
2020/11/17 07:05:35 lnd-ltc: launching network...
2020/11/17 07:05:49 btcd: launching node...
2020/11/17 07:05:49 btcd: 200 blocks generated
2020/11/17 07:05:49 lnd-btc: launching network...
2020/11/17 07:06:01 connext: launching network...
2020/11/17 07:06:04 xud: launching network...
2020/11/17 07:06:24 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Dave/tls.cert

Attempting to run on a resource constrained (but clean/fresh) VM results in similar timeout errors.

The text was updated successfully, but these errors were encountered:

sangaman · 2020-11-17T08:20:05Z

Oddly I get this even after setting the tls cert timeout periods to 300 seconds, implying xud isn't starting at all, but I can't figure out why. I think at it would be good to get some error logging here to detect if xud isn't starting if possible and print any error messages.

kilrau · 2020-11-18T12:24:42Z

Building images with npm fails instantly for me, probably something wrong with my node/npm which I just installed. bash docker-build.sh failed at the step Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory.

kilrau · 2020-11-18T12:35:19Z

Rebooted and tried again, exact same outcome as above.

michael1011 · 2020-11-18T12:41:10Z

Building and running them worked fine on my machine:

Ubuntu 20.10
Node v14.15.1
NPM v6.14.8
Docker v19.03.13

LePremierHomme · 2020-11-18T20:29:57Z

Building images with npm fails instantly for me, probably something wrong with my node/npm which I just installed. bash docker-build.sh failed at the step Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory.

Looks like the problem is sh: 1: tsc: not found. Try to run npm install -g typescript.

LePremierHomme · 2020-11-18T20:37:53Z

Oddly I get this even after setting the tls cert timeout periods to 300 seconds, implying xud isn't starting at all, but I can't figure out why. I think at it would be good to get some error logging here to detect if xud isn't starting if possible and print any error messages.

Indeed the problem is probably that xud fails to start. If no log file exist, we can print xud's stdout. I haven't done that on default because it can be annoying.
In case you're working on a custom branch which refactored xud's cmd args, then that might be the problem. Please let me know if so, and if not, i'll add support for printing xud's stdout.

kilrau · 2020-11-19T11:34:11Z

I finally got simulation tests to build successfully with the exact same setup that worked for @michael1011 :

    Ubuntu 20.10
    Node v14.15.1
    NPM v6.14.8
    Docker v19.03.13

What I did:

npm run test:sim:build still first failed instantly for me just like before (and yes I installed typescript via npm install -g typescript beforehand).
bash docker-build.sh fails at the same step as above Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory
then I ran npm run test:sim:build again and it finally went through fine:

Successfully built 44c81800e23a
Successfully tagged simulation_test:latest

Running them fails with:

kilrau@K-Yoga:~/xud/test/simulation$ npm run test:sim:run

> [email protected] test:sim:run /home/kilrau/xud
> (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/start-indra.sh
Using docker images indra_name:latest 
Can't find image redis:5-alpine locally, attempting to pull redis:5-alpine
5-alpine: Pulling from library/redis
188c0c94c7c5: Pull complete 
fb6015f7c791: Pull complete 
f8890a096979: Pull complete 
4d275bc77206: Pull complete 
b6ead6773e5a: Pull complete 
8375f47daa52: Pull complete 
Digest: sha256:e0256846c84d07095778890a8c481eceaca9a46a9d292a499555f40d248dfb29
Status: Downloaded newer image for redis:5-alpine
docker.io/library/redis:5-alpine
Proxy configured
Node configured
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
make: *** [Makefile:48: start] Error 1
~/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/stop.sh indra
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: indra
bash ops/stop.sh daicard
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: daicard
bash ops/stop.sh testnet
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: testnet
docker container prune -f
Total reclaimed space: 0B
docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true
docker secret rm indra_database_dev 2> /dev/null || true
docker volume rm indra_database_dev  2> /dev/null || true
docker volume rm `docker volume ls -q -f name=indra_database_test_*` 2> /dev/null || true
rm -rf .chaindata/*
rm -rf modules/bot/.bot-store
~/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-19T11_32_50_983Z-debug.log

kilrau · 2020-11-19T11:35:46Z

2020-11-19T11_32_50_983Z-debug.log:

 info it worked if it ends with ok
1 verbose cli [
1 verbose cli   '/home/kilrau/.nvm/versions/node/v14.15.1/bin/node',
1 verbose cli   '/home/kilrau/.nvm/versions/node/v14.15.1/bin/npm',
1 verbose cli   'run',
1 verbose cli   'test:sim:run'
1 verbose cli ]
2 info using [email protected]
3 info using [email protected]
4 verbose run-script [ 'pretest:sim:run', 'test:sim:run', 'posttest:sim:run' ]
5 info lifecycle [email protected]~pretest:sim:run: [email protected]
6 info lifecycle [email protected]~test:sim:run: [email protected]
7 verbose lifecycle [email protected]~test:sim:run: unsafe-perm in lifecycle true
8 verbose lifecycle [email protected]~test:sim:run: PATH: /home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
9 verbose lifecycle [email protected]~test:sim:run: CWD: /home/kilrau/xud
10 silly lifecycle [email protected]~test:sim:run: Args: [ '-c', '(cd test/simulation && ./docker-run.sh)' ]
11 silly lifecycle [email protected]~test:sim:run: Returned: code: 1  signal: null
12 info lifecycle [email protected]~test:sim:run: Failed to exec test:sim:run script
13 verbose stack Error: [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
13 verbose stack Exit status 1
13 verbose stack     at EventEmitter.<anonymous> (/home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
13 verbose stack     at EventEmitter.emit (events.js:315:20)
13 verbose stack     at ChildProcess.<anonymous> (/home/kilrau/.nvm/versions/node/v14.15.1/lib/node_modules/np>
13 verbose stack     at ChildProcess.emit (events.js:315:20)
13 verbose stack     at maybeClose (internal/child_process.js:1048:16)
13 verbose stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:288:5)
14 verbose pkgid [email protected]
15 verbose cwd /home/kilrau/xud/test/simulation
16 verbose Linux 5.8.0-29-generic
17 verbose argv "/home/kilrau/.nvm/versions/node/v14.15.1/bin/node" "/home/kilrau/.nvm/versions/node/v14.15.1/>
18 verbose node v14.15.1
19 verbose npm  v6.14.8
20 error code ELIFECYCLE
21 error errno 1
22 error [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
22 error Exit status 1
23 error Failed at the [email protected] test:sim:run script.
23 error This is probably not a problem with npm. There is likely additional logging output above.
24 verbose exit [ 1, true ]

kilrau · 2020-11-19T11:58:14Z

Should we also bump simulation tests to current versions? E.g. to go 1.14 and lnd to 0.11.1 , guess there is more to be updated.

https://github.com/ExchangeUnion/xud/blob/master/test/simulation/go.mod

kilrau · 2020-11-19T12:01:04Z

Tried a couple of more runs with docker system prune -a before, no luck: always failing with the Network indra declared as external, but could not be found as above.

kilrau · 2020-11-19T12:02:50Z

Can you set up a VM with this "default developer setup":

    Ubuntu 20.10
    Node v14.15.1 (via nvm)
    NPM v6.14.8 (via nvm)
    Docker v19.03.13 (installed via `apt install docker.io`)

And harden simulation tests for this? You should be able to reproduce the issues above without problems. Our goal is to have the setup above with these specific version work reliably @LePremierHomme

ghost · 2020-11-19T12:28:12Z

Can you try:

docker system prune -a
npm run test:sim:clean
npm run test:sim:clean:indra
npm run test:sim

kilrau · 2020-11-19T13:15:31Z

Did that, same:

> [email protected] test:sim:run /home/kilrau/xud
> (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/start-indra.sh
Using docker images indra_name:latest 
Can't find image redis:5-alpine locally, attempting to pull redis:5-alpine
5-alpine: Pulling from library/redis
188c0c94c7c5: Pull complete 
fb6015f7c791: Pull complete 
f8890a096979: Pull complete 
4d275bc77206: Pull complete 
b6ead6773e5a: Pull complete 
8375f47daa52: Pull complete 
Digest: sha256:e0256846c84d07095778890a8c481eceaca9a46a9d292a499555f40d248dfb29
Status: Downloaded newer image for redis:5-alpine
docker.io/library/redis:5-alpine
Proxy configured
Node configured
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
make: *** [Makefile:48: start] Error 1
~/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
~/xud/test/simulation/temp/indra ~/xud/test/simulation
bash ops/stop.sh indra
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: indra
bash ops/stop.sh daicard
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: daicard
bash ops/stop.sh testnet
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: testnet
docker container prune -f
Total reclaimed space: 0B
docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true
docker secret rm indra_database_dev 2> /dev/null || true
docker volume rm indra_database_dev  2> /dev/null || true
docker volume rm `docker volume ls -q -f name=indra_database_test_*` 2> /dev/null || true
rm -rf .chaindata/*
rm -rf modules/bot/.bot-store
~/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-19T12_48_25_297Z-debug.log
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test:sim: `(npm run test:sim:build && npm run test:sim:run)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test:sim script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-19T12_48_25_318Z-debug.log

michael1011 · 2020-11-19T13:18:23Z

Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

I also had these issues. A reboot somehow solved them

kilrau · 2020-11-19T13:19:47Z

Let me try that, also good hint for @LePremierHomme

kilrau · 2020-11-19T13:33:17Z

Reboot didn't solve it for me.

LePremierHomme · 2020-11-19T22:43:03Z

What I did:
npm run test:sim:build still first failed instantly for me just like before (and yes I installed typescript via npm install -g typescript beforehand).

It still just looks as if you don’t have access to tsc executable. I verified it and it should work fine if you installed it via npm install -g typescript (and it’s accessible on your $PATH), or you run npm install (if you didn’t, you’ll have more issues, but that will be the first one).

bash docker-build.sh fails at the same step as above Step 7/10 : COPY xud ./xud ERROR: Service 'xud' failed to build: COPY failed: stat /var/lib/docker/tmp/docker-builder981589250/xud: no such file or directory

Why do you run this directly? It should always be run only as part of npm run test:sim:build. It failed because the compilation failed, and xud dir files were not copied to the expected location. So this is not a problem, and we just need to solve the previous one.

Regarding your running failure - something is wrong with indra’s docker setup (“This node is not a swarm manager…”). I’m trying to reproduce it on gcloud VM now.

michael1011 · 2020-11-19T22:59:48Z

Regarding your running failure - something is wrong with indra’s docker setup (“This node is not a swarm manager…”)

@erkarl do you think debugging the existing Indra setup is worth it since Connext v2 is just around the corner?

ghost · 2020-11-20T05:39:08Z

@erkarl do you think debugging the existing Indra setup is worth it since Connext v2 is just around the corner?

No, disable it.

LePremierHomme · 2020-11-20T10:28:46Z

Ubuntu 20.10 has an issue with docker installation (“Package 'docker-ce' has no installation candidate”), so I used Ubuntu 20.04 LTS instead. Looks like it had no issue the swarm manager. This is what I got:

> [email protected] test:sim:run /home/premier_gmail_com/xud
> (cd test/simulation && ./docker-run.sh)

/home/premier_gmail_com/xud/test/simulation/temp/indra /home/premier_gmail_com/xud/test/simulation
bash ops/start-indra.sh
Swarm initialized: current node (vr1m05tumic9rzks9mzy5375t) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join --token SWMTKN-1-62m6b062m40s8cmxqsa1cyp33tg6aioncmkss3akvnjw6peqko-4b1q6jatui1jsgw99fjtyj7jd 10.128.0.19:2377

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Requirements:

Ubuntu 20.04 LTS
25 GB disk size
Good internet connection, 4 CPU, 16 GB memory (otherwise random failure may occur)

Setup script (some sudo permissions are required due to indra’s docker setup):

curl -o- https://raw.githubusercontent.com/creationix/nvm/v0.33.0/install.sh | bash
export NVM_DIR="$HOME/.nvm"    
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion"

nvm install 14.15.1

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

sudo curl -L "https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose

sudo apt-get install build-essential python npm jq

https://github.com/ExchangeUnion/xud.git
cd xud
npm i
sudo npm run test:sim:build
sudo npm run test:sim:run

TODO (@LePremierHomme):

Update simulation/README.md with the script.
Fix simulation/docker-run.sh to exit before running the test if indra start failed.
Update Go version (LND v0.11.1-beta is already the latest).

kilrau · 2020-11-20T11:48:44Z

Ubuntu 20.10 has an issue with docker installation (“Package 'docker-ce' has no installation candidate”)

Docker install scripts are always behind. Please install docker on ubuntu 20.10 via ubuntu repo: sudo apt install docker.io which should get you Docker version 19.03.13, build 4484c46.

kilrau · 2020-11-20T11:49:01Z

That looks like A LOT of additional dependencies that are needed. Are we sure we need all of these?

kilrau@K-Yoga:~$ bash test.sh 
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 11329  100 11329    0     0  80921      0 --:--:-- --:--:-- --:--:-- 80921
=> nvm is already installed in /home/kilrau/.nvm, trying to update using git
=> error: pathspec 'v0.33.0' did not match any file(s) known to git
=> Compressing and cleaning up git repository
Enumerating objects: 317, done.
Counting objects: 100% (317/317), done.
Delta compression using up to 8 threads
Compressing objects: 100% (292/292), done.
Writing objects: 100% (317/317), done.
Total 317 (delta 49), reused 268 (delta 0), pack-reused 0

=> nvm source string already in /home/kilrau/.bashrc
=> bash_completion source string already in /home/kilrau/.bashrc
=> You currently have modules installed globally with `npm`. These will no
=> longer be linked to the active version of Node when you install a new node
=> with `nvm`; and they may (depending on how you construct your `$PATH`)
=> override the binaries of modules installed with `nvm`:

/home/kilrau/.nvm/versions/node/v14.15.1/lib
└── [email protected]
=> If you wish to uninstall them at a later point (or re-install them under your
=> `nvm` Nodes), you can remove them from the system Node as follows:

     $ nvm use system
     $ npm uninstall -g a_module

=> Close and reopen your terminal to start using nvm or run the following to use it now:

export NVM_DIR="/home/kilrau/.nvm"
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh"  # This loads nvm
v14.15.1 is already installed.
Now using node v14.15.1 (npm v6.14.8)
# Executing docker install script, commit: 26ff363bcf3b3f5a00498ac43694bf1c7d9ce16c
Warning: the "docker" command appears to already exist on this system.

If you already have Docker installed, this script can cause trouble, which is
why we're displaying this warning and provide the opportunity to cancel the
installation.

If you installed the current Docker package using this script and are using it
again to update Docker, you can safely ignore this message.

You may press Ctrl+C now to abort this script.
+ sleep 20
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/ubuntu/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)).
+ sh -c echo "deb [arch=amd64] https://download.docker.com/linux/ubuntu groovy stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
E: Package 'docker-ce' has no installation candidate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   651  100   651    0     0   5761      0 --:--:-- --:--:-- --:--:--  5761
100 11.6M  100 11.6M    0     0  7778k      0  0:00:01  0:00:01 --:--:-- 12.2M
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'python-is-python2' instead of 'python'
build-essential is already the newest version (12.8ubuntu3).
build-essential set to manually installed.
The following additional packages will be installed:
  gyp libjq1 libjs-inherits libjs-is-typedarray libjs-psl libjs-typedarray-to-buffer libnode-dev libnode72 libonig5 libpython2-stdlib libpython2.7-minimal
  libpython2.7-stdlib libssl-dev libuv1-dev node-abbrev node-ajv node-ansi node-ansi-align node-ansi-regex node-ansi-styles node-ansistyles node-aproba node-archy
  node-are-we-there-yet node-asap node-asn1 node-assert-plus node-asynckit node-aws-sign2 node-aws4 node-balanced-match node-bcrypt-pbkdf node-bl node-bluebird node-boxen
  node-brace-expansion node-builtin-modules node-builtins node-cacache node-call-limit node-camelcase node-caseless node-chalk node-chownr node-ci-info node-cli-boxes
  node-cliui node-clone node-co node-color-convert node-color-name node-colors node-columnify node-combined-stream node-concat-map node-concat-stream node-config-chain
  node-configstore node-console-control-strings node-copy-concurrently node-core-util-is node-cross-spawn node-crypto-random-string node-cyclist node-dashdash
  node-debbundle-es-to-primitive node-debug node-decamelize node-decompress-response node-deep-extend node-defaults node-define-properties node-delayed-stream
  node-delegates node-detect-indent node-detect-newline node-dot-prop node-duplexer3 node-duplexify node-ecc-jsbn node-editor node-encoding node-end-of-stream
  node-err-code node-errno node-es6-promise node-escape-string-regexp node-execa node-extend node-extsprintf node-fast-deep-equal node-find-up node-flush-write-stream
  node-forever-agent node-form-data node-from2 node-fs-vacuum node-fs-write-stream-atomic node-fs.realpath node-function-bind node-gauge node-genfun node-get-caller-file
  node-get-stream node-getpass node-glob node-got node-graceful-fs node-gyp node-har-schema node-har-validator node-has-flag node-has-symbol-support-x
  node-has-to-string-tag-x node-has-unicode node-hosted-git-info node-http-signature node-iconv-lite node-iferr node-import-lazy node-imurmurhash node-inflight
  node-inherits node-ini node-invert-kv node-ip node-ip-regex node-is-npm node-is-obj node-is-object node-is-path-inside node-is-plain-obj node-is-retry-allowed
  node-is-stream node-is-typedarray node-isarray node-isexe node-isstream node-isurl node-jsbn node-json-parse-better-errors node-json-schema node-json-schema-traverse
  node-json-stable-stringify node-json-stringify-safe node-jsonify node-jsonparse node-jsonstream node-jsprim node-latest-version node-lazy-property node-lcid node-libnpx
  node-locate-path node-lockfile node-lodash node-lodash-packages node-lowercase-keys node-lru-cache node-make-dir node-mem node-mime node-mime-types node-mimic-fn
  node-mimic-response node-minimatch node-minimist node-mississippi node-mkdirp node-move-concurrently node-ms node-mute-stream node-nopt node-normalize-package-data
  node-npm-bundled node-npm-package-arg node-npm-run-path node-npmlog node-number-is-nan node-oauth-sign node-object-assign node-once node-opener node-os-locale
  node-os-tmpdir node-osenv node-p-cancelable node-p-finally node-p-is-promise node-p-limit node-p-locate node-p-timeout node-package-json node-parallel-transform
  node-path-exists node-path-is-absolute node-path-is-inside node-performance-now node-pify node-prepend-http node-process-nextick-args node-promise-inflight
  node-promise-retry node-promzard node-proto-list node-prr node-pseudomap node-psl node-pump node-pumpify node-punycode node-qs node-qw node-rc node-read
  node-read-package-json node-readable-stream node-registry-auth-token node-registry-url node-request node-require-directory node-require-main-filename node-resolve
  node-resolve-from node-retry node-rimraf node-run-queue node-safe-buffer node-semver node-semver-diff node-set-blocking node-sha node-shebang-command node-shebang-regex
  node-signal-exit node-slash node-slide node-sorted-object node-spdx-correct node-spdx-exceptions node-spdx-expression-parse node-spdx-license-ids node-sshpk node-ssri
  node-stream-each node-stream-iterate node-stream-shift node-strict-uri-encode node-string-decoder node-string-width node-strip-ansi node-strip-eof
  node-strip-json-comments node-supports-color node-tar node-term-size node-text-table node-through node-through2 node-timed-out node-tough-cookie node-tunnel-agent
  node-tweetnacl node-typedarray node-typedarray-to-buffer node-uid-number node-unique-filename node-unique-string node-unpipe node-uri-js node-url-parse-lax
  node-url-to-options node-util-deprecate node-uuid node-validate-npm-package-license node-validate-npm-package-name node-verror node-wcwidth.js node-which
  node-which-module node-wide-align node-widest-line node-wrap-ansi node-wrappy node-write-file-atomic node-xdg-basedir node-xtend node-y18n node-yallist node-yargs
  node-yargs-parser nodejs nodejs-doc python2 python2-minimal python2.7 python2.7-minimal
Suggested packages:
  libssl-doc python2-doc python-tk python2.7-doc
The following packages will be REMOVED:
  python-is-python3
The following NEW packages will be installed:
  gyp jq libjq1 libjs-inherits libjs-is-typedarray libjs-psl libjs-typedarray-to-buffer libnode-dev libnode72 libonig5 libpython2-stdlib libpython2.7-minimal
  libpython2.7-stdlib libssl-dev libuv1-dev node-abbrev node-ajv node-ansi node-ansi-align node-ansi-regex node-ansi-styles node-ansistyles node-aproba node-archy
  node-are-we-there-yet node-asap node-asn1 node-assert-plus node-asynckit node-aws-sign2 node-aws4 node-balanced-match node-bcrypt-pbkdf node-bl node-bluebird node-boxen
  node-brace-expansion node-builtin-modules node-builtins node-cacache node-call-limit node-camelcase node-caseless node-chalk node-chownr node-ci-info node-cli-boxes
  node-cliui node-clone node-co node-color-convert node-color-name node-colors node-columnify node-combined-stream node-concat-map node-concat-stream node-config-chain
  node-configstore node-console-control-strings node-copy-concurrently node-core-util-is node-cross-spawn node-crypto-random-string node-cyclist node-dashdash
  node-debbundle-es-to-primitive node-debug node-decamelize node-decompress-response node-deep-extend node-defaults node-define-properties node-delayed-stream
  node-delegates node-detect-indent node-detect-newline node-dot-prop node-duplexer3 node-duplexify node-ecc-jsbn node-editor node-encoding node-end-of-stream
  node-err-code node-errno node-es6-promise node-escape-string-regexp node-execa node-extend node-extsprintf node-fast-deep-equal node-find-up node-flush-write-stream
  node-forever-agent node-form-data node-from2 node-fs-vacuum node-fs-write-stream-atomic node-fs.realpath node-function-bind node-gauge node-genfun node-get-caller-file
  node-get-stream node-getpass node-glob node-got node-graceful-fs node-gyp node-har-schema node-har-validator node-has-flag node-has-symbol-support-x
  node-has-to-string-tag-x node-has-unicode node-hosted-git-info node-http-signature node-iconv-lite node-iferr node-import-lazy node-imurmurhash node-inflight
  node-inherits node-ini node-invert-kv node-ip node-ip-regex node-is-npm node-is-obj node-is-object node-is-path-inside node-is-plain-obj node-is-retry-allowed
  node-is-stream node-is-typedarray node-isarray node-isexe node-isstream node-isurl node-jsbn node-json-parse-better-errors node-json-schema node-json-schema-traverse
  node-json-stable-stringify node-json-stringify-safe node-jsonify node-jsonparse node-jsonstream node-jsprim node-latest-version node-lazy-property node-lcid node-libnpx
  node-locate-path node-lockfile node-lodash node-lodash-packages node-lowercase-keys node-lru-cache node-make-dir node-mem node-mime node-mime-types node-mimic-fn
  node-mimic-response node-minimatch node-minimist node-mississippi node-mkdirp node-move-concurrently node-ms node-mute-stream node-nopt node-normalize-package-data
  node-npm-bundled node-npm-package-arg node-npm-run-path node-npmlog node-number-is-nan node-oauth-sign node-object-assign node-once node-opener node-os-locale
  node-os-tmpdir node-osenv node-p-cancelable node-p-finally node-p-is-promise node-p-limit node-p-locate node-p-timeout node-package-json node-parallel-transform
  node-path-exists node-path-is-absolute node-path-is-inside node-performance-now node-pify node-prepend-http node-process-nextick-args node-promise-inflight
  node-promise-retry node-promzard node-proto-list node-prr node-pseudomap node-psl node-pump node-pumpify node-punycode node-qs node-qw node-rc node-read
  node-read-package-json node-readable-stream node-registry-auth-token node-registry-url node-request node-require-directory node-require-main-filename node-resolve
  node-resolve-from node-retry node-rimraf node-run-queue node-safe-buffer node-semver node-semver-diff node-set-blocking node-sha node-shebang-command node-shebang-regex
  node-signal-exit node-slash node-slide node-sorted-object node-spdx-correct node-spdx-exceptions node-spdx-expression-parse node-spdx-license-ids node-sshpk node-ssri
  node-stream-each node-stream-iterate node-stream-shift node-strict-uri-encode node-string-decoder node-string-width node-strip-ansi node-strip-eof
  node-strip-json-comments node-supports-color node-tar node-term-size node-text-table node-through node-through2 node-timed-out node-tough-cookie node-tunnel-agent
  node-tweetnacl node-typedarray node-typedarray-to-buffer node-uid-number node-unique-filename node-unique-string node-unpipe node-uri-js node-url-parse-lax
  node-url-to-options node-util-deprecate node-uuid node-validate-npm-package-license node-validate-npm-package-name node-verror node-wcwidth.js node-which
  node-which-module node-wide-align node-widest-line node-wrap-ansi node-wrappy node-write-file-atomic node-xdg-basedir node-xtend node-y18n node-yallist node-yargs
  node-yargs-parser nodejs nodejs-doc npm python-is-python2 python2 python2-minimal python2.7 python2.7-minimal
0 upgraded, 300 newly installed, 1 to remove and 0 not upgraded.
Need to get 20.7 MB of archives.
After this operation, 102 MB of additional disk space will be used.
Do you want to continue? [Y/n]

LePremierHomme · 2020-11-20T12:02:41Z

Docker install scripts are always behind. Please install docker on ubuntu 20.10 via ubuntu repo: sudo apt install docker.io which should get you Docker version 19.03.13, build 4484c46.

I thought I tried that first and it didn't work, will try it again.

That looks like A LOT of additional dependencies that are needed. Are we sure we need all of these?

These dependencies are installed due to sudo apt-get install npm which is used because nvm doesn't put npm to be accessible via sudo (needed for indra docker. setup). I tried to solve it differently without success, but i'll look into it again.

kilrau · 2020-11-20T12:44:04Z

Todov2:

you can safely remove the docker installation script from the script (it fails on many OSes anyways due to "no installation candidate") and just provide a specific version as requirement in README.md instead
I accepted installing all these dependencies above and ran the simulation tests via your script again, tried multiple times. Still failing with the same:

bash ops/start-indra.sh
Using docker images indra_name:latest 
Proxy configured
Node configured
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
make: *** [Makefile:48: start] Error 1
/home/kilrau/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
/home/kilrau/xud/test/simulation/temp/indra /home/kilrau/xud/test/simulation
bash ops/stop.sh indra
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: indra
bash ops/stop.sh daicard
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: daicard
bash ops/stop.sh testnet
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.
No stack, service or running container names match: testnet
docker container prune -f
Total reclaimed space: 0B
docker network rm indra indra_cf_tester indra_node_tester indra_test_store 2> /dev/null || true
docker secret rm indra_database_dev 2> /dev/null || true
docker volume rm indra_database_dev  2> /dev/null || true
docker volume rm `docker volume ls -q -f name=indra_database_test_*` 2> /dev/null || true
rm -rf .chaindata/*
rm -rf modules/bot/.bot-store
/home/kilrau/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2020-11-20T12_43_15_758Z-debug.log

kilrau · 2020-11-21T17:53:56Z

And I have my first successful run. What I did:

docker swarm init
npm run test:sim:clean
docker system prune
make sure the /test/simulation/temp folder is not there
npm run test:sim:build

And it went through! Full log: https://paste.ubuntu.com/p/HBZysWs4HV/

kilrau · 2020-11-21T18:26:38Z

Second run which I started immediately after without cleaning anything succeeded too.

kilrau · 2020-11-21T19:21:55Z

Third run which I started after reboot (without cleaning or any further actions) succeeded too.

kilrau · 2020-11-21T19:23:53Z

Let's finalize the todo's @LePremierHomme and then let @sangaman do an acceptance test - he's working on xud mainly and needs to run simulation tests locally.

ghost · 2020-11-23T08:15:14Z

I suggest to disable connext v1/indra simulation tests if they are causing issues. We'll be switching to V2 this week.

kilrau · 2020-11-23T09:59:45Z

Well they are 100% reliable for me now after #1995 (comment), I opened #2001 to close this issue for now adding hints how to prep your environment to make tests pass reliably. If I manage, everyone should ;)

We'll anyways take another look at simulation tests after you did the switch to vector @erkarl

kilrau · 2020-11-24T09:14:13Z

I cleaned my environment with steps as per steps of my PR. Now the script fails to create the docker indra network again and the only solution seems to be to create it manually:

kilrau@K-Yoga:~/xud$ npm run test:sim:run

> [email protected] test:sim:run /home/kilrau/xud
> (cd test/simulation && ./docker-run.sh)

~/xud/test/simulation/temp/indra ~/xud/test/simulation
make: *** No rule to make target 'start'.  Stop.
~/xud/test/simulation
ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (1/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (2/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.

docker-run.sh: test failed, re-running... (3/3)

ERROR: Network indra declared as external, but could not be found. Please create the network manually using `docker network create indra` and try again.
~/xud/test/simulation/temp/indra ~/xud/test/simulation
make: *** No rule to make target 'reset'.  Stop.
~/xud/test/simulation
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] test:sim:run: `(cd test/simulation && ./docker-run.sh)`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the [email protected] test:sim:run script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR!     /home/kilrau/.npm/_logs/2020-11-24T09_12_06_185Z-debug.log
kilrau@K-Yoga:~/xud$ docker network create indra
707122d065aa464ea8e753cb701f943a65a8453e7b9bd35213facd587f88e9d9

sangaman · 2020-11-27T19:21:43Z

Following all the steps in the README in #2001 still gives me the error in the OP here:

2020/11/27 19:18:52 cannot start xud network: timeout waiting for TLS cert file to be created after 20 seconds: /app/temp/xuddatadir-Dave/tls.cert
exit status 1

Should we reopen?

kilrau · 2020-11-29T12:54:39Z

Well it doesn't fail for anyone else with that, thus isn't easy to debug for others. Rather open a separate issue and I guess you'll have to look into it yourself :/

Only thing that comes to mind are resources/Internet: You made sure Internet was stable and you have 10GB+ RAM free when you ran the tests?

LePremierHomme · 2020-11-29T21:44:05Z

@sangaman are you on master or a feature branch?

sangaman added the automated tests label Nov 17, 2020

kilrau assigned LePremierHomme Nov 17, 2020

kilrau added the P2 mid priority label Nov 17, 2020

kilrau changed the title ~~Simulation tests "cannot start xud network: timeout waiting for TLS cert file..."~~ Find a reliable way to run simulation tests locally Nov 19, 2020

kilrau changed the title ~~Find a reliable way to run simulation tests locally~~ Find a stable setup to run simulation tests locally Nov 19, 2020

kilrau mentioned this issue Nov 23, 2020

feat: enhance simulation test README #2001

Merged

kilrau closed this as completed in #2001 Nov 25, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Find a stable setup to run simulation tests locally #1995

Find a stable setup to run simulation tests locally #1995

sangaman commented Nov 17, 2020

sangaman commented Nov 17, 2020

kilrau commented Nov 18, 2020 •

edited

Loading

kilrau commented Nov 18, 2020

michael1011 commented Nov 18, 2020

LePremierHomme commented Nov 18, 2020

LePremierHomme commented Nov 18, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020 •

edited

Loading

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020 •

edited

Loading

ghost commented Nov 19, 2020

kilrau commented Nov 19, 2020

michael1011 commented Nov 19, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020

LePremierHomme commented Nov 19, 2020

michael1011 commented Nov 19, 2020 •

edited

Loading

ghost commented Nov 20, 2020

LePremierHomme commented Nov 20, 2020

kilrau commented Nov 20, 2020

kilrau commented Nov 20, 2020 •

edited

Loading

LePremierHomme commented Nov 20, 2020

kilrau commented Nov 20, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

ghost commented Nov 23, 2020

kilrau commented Nov 23, 2020

kilrau commented Nov 24, 2020 •

edited

Loading

sangaman commented Nov 27, 2020

kilrau commented Nov 29, 2020

LePremierHomme commented Nov 29, 2020

Find a stable setup to run simulation tests locally #1995

Find a stable setup to run simulation tests locally #1995

Comments

sangaman commented Nov 17, 2020

sangaman commented Nov 17, 2020

kilrau commented Nov 18, 2020 • edited Loading

kilrau commented Nov 18, 2020

michael1011 commented Nov 18, 2020

LePremierHomme commented Nov 18, 2020

LePremierHomme commented Nov 18, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020 • edited Loading

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020 • edited Loading

ghost commented Nov 19, 2020

kilrau commented Nov 19, 2020

michael1011 commented Nov 19, 2020

kilrau commented Nov 19, 2020

kilrau commented Nov 19, 2020

LePremierHomme commented Nov 19, 2020

michael1011 commented Nov 19, 2020 • edited Loading

ghost commented Nov 20, 2020

LePremierHomme commented Nov 20, 2020

kilrau commented Nov 20, 2020

kilrau commented Nov 20, 2020 • edited Loading

LePremierHomme commented Nov 20, 2020

kilrau commented Nov 20, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

kilrau commented Nov 21, 2020

ghost commented Nov 23, 2020

kilrau commented Nov 23, 2020

kilrau commented Nov 24, 2020 • edited Loading

sangaman commented Nov 27, 2020

kilrau commented Nov 29, 2020

LePremierHomme commented Nov 29, 2020

kilrau commented Nov 18, 2020 •

edited

Loading

kilrau commented Nov 19, 2020 •

edited

Loading

kilrau commented Nov 19, 2020 •

edited

Loading

michael1011 commented Nov 19, 2020 •

edited

Loading

kilrau commented Nov 20, 2020 •

edited

Loading

kilrau commented Nov 24, 2020 •

edited

Loading