Visual regression testing using Galata

[mbektas]

• adds visual regression testing support using https://github.com/jupyterlab/galata
• test suite uploads and runs a notebook, that has code cells creating different widget instances, in JupyterLab using headless browser
• cell output image captures are taken and compared to reference images
• test output artifacts are saved and accessible on workflow page

(mbektas#4 demonstrates detecting visual regression caused by change to widget label rendering)

@ellisonbg @blink1073 @afshin FYI, this is the first integration of Galata in a Jupyter project! More to come.

[github-actions]

👈 Launch a binder notebook on branch mbektasbbg/ipywidgets/galata

[mbektas]

Sample UI test result report generated as artifact

and diff related to failed test

[blink1073]

Love it!

[jtpio]

Looks great, thanks @mbektasbbg for this!

[jtpio]

Wondering whether this new package should be added to the existing lerna packages? Then we might be able to do without ui-tests/package-lock.json and keep dependencies in a top-level yarn.lock.

Or is it better to keep the ui-tests package a bit more separated from the rest?

[mbektas]

@jtpio good point. Initially I tried adding as dependency to top level package.json. I had some build issues then since rest of the project is using chai for unit testing and that was causing name clashes with jest that Galata uses (names like it, test, expect). Also, having a separate ui-tests directory & package decouples things nicely. Only the developers that add UI tests need to install the dependencies (which include browser binaries) and UI tests can work with different jupyterlab versions than ipywidgets is dependent on.

[mbektas]

I will add some documentation to guide on adding new UI test cases, building and testing locally.

[jtpio]

Ah right, there can indeed be some clashes (also since galata is built for a specific version of lab).

[jtpio]

Thanks!