Document how to debug integration tests

[johnsaigle]

Describe the bug
There isn't a document (that I know of) describing how to setup integration tests locally.

To Reproduce
Steps to reproduce the behaviour (attach screenshots if applicable):

1. On a development instance of LORIS, run npm run tests:integration
2. You'll probably get an error.
3. Search the repo for help.
4. I don't think you'll find any.

What did you expect to happen?
We should have documentation explaining this part of our test suite.

Browser Environment (please complete the following information):

• N/A

Server Environment (if known):

• N/A

Additional context
This was encountered by @driusan recently and discussed on Slack. I've also had the same issue.

[ridz1208]

@johnsaigle does github have a bump feature ?

[johnsaigle]

🤷‍♂

[maltheism]

@johnsaigle I think it's a great idea to have a guide on everything about the testing. I think an example of how to best approach a single module of LORIS for running tests after development of the module would be amazingly useful. So we don't have to run every test before getting to the tests that we want to see work or not.

Other thoughts: My experience with trying the npm run tests:integration command (in the past) had the output telling me to install docker on my mac. I believe after installing docker I found out I needed to run composer install and after the npm run tests:integration worked for me. Although this is on my local development machine and might be different with how VMs are setup with mysql not necessarily on the VM.

[johnsaigle]

To clarify there is actually some documentation in test/README.md. It explains what command to run, how to install Docker, etc.

This ticket was created in response to Dave's comment asking how to debug a Travis report like this:

Waiting for Selenium...
Selenium is alive
PHPUnit 7.0.0 by Sebastian Bergmann and contributors.
Runtime: PHP 7.2.25
Configuration: /app/test/phpunit.xml
.....................................I......................... 63 / 511 ( 12%)
............................................................... 126 / 511 ( 24%)
.......................................SSSS.................... 189 / 511 ( 36%)
....EEE....F...................I.......I....................... 252 / 511 ( 49%)
............................................................... 315 / 511 ( 61%)
..............................EE.EEEEEEEEEEEEEEEE
The job exceeded the maximum time limit for jobs, and has been terminated.

It's not clear what is causing those errors or what to do about them.

The only bit of info I could find in the linked document is this:

You can see the integration tests in action by connecting your VNC viewer to :5900 and supplying the password secret.

However I don't have anything like that set up and it's not clear whether that would help with a report like this.

[johnsaigle]

@maltheism I just wrote an automated testing guide #5859. Hopefully that will help. Between that document and also the CONTRIBUTING.md document I think that should give a rough idea on approaching testing on a more general level.

I also think there's a document on "how to create a module" buried in the Wiki somewhere. however I think it was created for LORIS v 14 or 15 so it would need updating.

[johnsaigle]

So we don't have to run every test before getting to the tests that we want to see work or not.

I think it'll always be important to run every test. Generally speaking anyway. A lot of things in LORIS aren't all that "modular" and so changing something in one place can affect other parts of the code.

[maltheism]

The only bit of info I could find in the linked document is this:

You can see the integration tests in action by connecting your VNC viewer to :5900 and supplying the password secret.

However I don't have anything like that set up and it's not clear whether that would help with a report like this.

I got this to work one day after asking @kongtiaowang some questions.
My notes were to launch Selenium Server with doing this:

cd ~/
java -jar selenium-server-standalone-2.53.1.jar

Start tests:
npm run tests:integration
View in VNC Viewer:
127.0.0.1
Password:
secret
Then I could see the browser performing all the tests (one by one) in realtime.

I think it'll always be important to run every test. Generally speaking anyway. A lot of things in LORIS aren't all that "modular" and so changing something in one place can affect other parts of the code.

I mean like instead of wasting 15 minutes or however long it takes for the tests to get to the specific test of seeing one specific thing fail. We can just go directly to that test and I mean like frontend tests after the module has been updated to react. I can see php code being modified possibly breaking all of LORIS.

[johnsaigle]

Would you mind updating test/README.md to include that information?

We can just go directly to that test and I mean like frontend tests

That's an interesting idea. It may be possible if you look into the phpunit command flags.

npm run tests:integration is an alias and just runs test/dockerized-integration-tests.sh. This in turn runs the command:

docker-compose run -T --rm ${CONTAINER} vendor/bin/phpunit --configuration test/phpunit.xml --testsuite LorisModuleIntegrationTests $*

test/phpunit.xml contains a list of all the integration tests that this command will run.

The tool might have something like an "exclude" or "include" flag that could be used to run just the tests for one module.

[johnsaigle]

@maltheism I also found this but it seems out of date. https://github.com/aces/Loris/wiki/LORIS-Module-Testing

[maltheism]

@johnsaigle I recall viewing that guide (many months ago) when I was trying to get the VNC Viewer to display the test running. I'll try to replicate everything with what the guide explains and see whats outdated.

[johnsaigle]

I think almost all of it is outdated. We're using Docker to set up most of what it's describing now.

It also seems to have two concerns going on. "Creating tests for a new module" and "Installing the integration test suite". I think these should be separate files.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.