Add e2e tests for new model overview experience (part 1) #1467

romanlutz · 2022-06-06T20:48:48Z

Description

copy of #1462 but targeted at main instead of postbuild

While adding e2e tests for the new model overview experience I encountered several challenges that needed to be addressed first. Therefore, this PR contains somewhat more than just the tests. Whatever I could separate out into other PRs I've already separated out and merged (#1505 #1507 #1478)

e2e-widget.js now has the ability to generate notebooks with flights injected! We just pass feature_flights to the dashboard constructor and voila we can run e2e tests on the flighted UI. Right now there's only 1 flight, but if there were multiple we'd need to specify which combinations we want to have e2e tests for and add spec files accordingly (more on spec files below).
added a very basic set of tests for the new model overview experience in the e2e package (describeNewModelOverview under v2) and moved the existing ones into a v1 folder. Tests cover all tasks and notebooks, but will need to be expanded in a follow-up PR.
lots of spec files for the flighted notebooks. These aren't necessarily very different. The only file that actually differs is the one about model overview (using describeNewModelOverview). You can easily imagine future scenarios where there is quite a bit of difference so this duplication may end up being less duplication than it looks right now. Fortunately, I found a nice way to avoid duplicating the "expected" values (see modelAssessmentDatasets.ts).

For the record, I fully expect to delete everything v1/old as soon as we are confident about v2/new. This will not linger around like other "old" parts that we've seen in the past.

Checklist

I have added screenshots above for all UI changes.
I have added e2e tests for all UI changes.
Documentation was updated if it was needed.

…verview

…ebook execution, more ID tags added to model overview component

…olbox into romanlutz/test_model_overview

…ai-toolbox into romanlutz/test_model_overview

* fix error on machines with pyspark installed where passed dataframe is not spark pandas (#1415) * add postbuild branch trigger (#1417) * Fix causal UI strings according to classification/regression tasks (#1419) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Fix description for model overview (#1425) * fix description for model overview * keep new description for new model overview * fix failing to create error report when filter_features is empty list (#1421) * filter out missing values from what if dropdown to prevent explanation dashboard from crashing (#1418) * Remove |Set Value| blurb in case it is not availble in counterfactual panel (#1426) Signed-off-by: Gaurav Gupta <[email protected]> * Add y-axis description to counterfactual feature importance chart (#1423) Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * Add the user class name to causal UI strings (#1422) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Add the user class name to causal UI strings Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * fix math.min / max for array size more than 10^7 (#1427) Signed-off-by: Ke Xu <[email protected]> * upgrade pytest and lightgbm to try to fix random pytest segfault test failures (#1424) s * fix flaky notebook causing build failures by adding retry logic (#1431) * Upper bound SciKit-Learn to address freeze in causal (#1432)  ## Description Replaces #1429 to address #1430 . Causal analysis is getting stuck with the latest release of SciKit-Learn. This contains: - Test case which gets stuck with SciKit-Learn 1.1.0 - Upper bound on SciKit-Learn in `requirements.txt` ## Checklist  - [x] I have added screenshots above for all UI changes. - [x] Documentation was updated if it was needed. - [x] New tests were added or changes were manually verified. Signed-off-by: Richard Edgar <[email protected]> * fix dependency chart axis updating with incorrect values in explanation dashboard (#1437) * fix codecov and widget test screenshot uploads (#1428) * release raiwidgets and responsibleai v0.18.2 (#1439) * fix (#1441) Signed-off-by: vinutha karanth <[email protected]> * Fix cohort name conflict and not run few tests for AML (#1442) * fix Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * Few e2e tests changes to accommodate AML static tests (#1445) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * Fix locators logic for string features - data explorer and model statistics components (#1446) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * Add more unittests RAI dashboard input class (#1448) * Add unit tests for ResponsibleAIDashboardInput Signed-off-by: Gaurav Gupta <[email protected]> * Add more tests Signed-off-by: Gaurav Gupta <[email protected]> * Fix imports Signed-off-by: Gaurav Gupta <[email protected]> * Address code review comments Signed-off-by: Gaurav Gupta <[email protected]> * Update the way to get the length of elements obtained in e2e tests (#1450) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * skip what-if create tests for AML Signed-off-by: vinutha karanth <[email protected]> Co-authored-by: Ilya Matiach <[email protected]> Co-authored-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> Co-authored-by: Richard Edgar <[email protected]> Co-authored-by: Vinutha Karanth <[email protected]>

…ategorical')" (#1452) * fix error on machines with pyspark installed where passed dataframe is not spark pandas (#1415) * add postbuild branch trigger (#1417) * Fix causal UI strings according to classification/regression tasks (#1419) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Fix description for model overview (#1425) * fix description for model overview * keep new description for new model overview * fix failing to create error report when filter_features is empty list (#1421) * filter out missing values from what if dropdown to prevent explanation dashboard from crashing (#1418) * Remove |Set Value| blurb in case it is not availble in counterfactual panel (#1426) Signed-off-by: Gaurav Gupta <[email protected]> * Add y-axis description to counterfactual feature importance chart (#1423) Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * Add the user class name to causal UI strings (#1422) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Add the user class name to causal UI strings Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * fix math.min / max for array size more than 10^7 (#1427) Signed-off-by: Ke Xu <[email protected]> * upgrade pytest and lightgbm to try to fix random pytest segfault test failures (#1424) s * fix flaky notebook causing build failures by adding retry logic (#1431) * Upper bound SciKit-Learn to address freeze in causal (#1432)  ## Description Replaces #1429 to address #1430 . Causal analysis is getting stuck with the latest release of SciKit-Learn. This contains: - Test case which gets stuck with SciKit-Learn 1.1.0 - Upper bound on SciKit-Learn in `requirements.txt` ## Checklist  - [x] I have added screenshots above for all UI changes. - [x] Documentation was updated if it was needed. - [x] New tests were added or changes were manually verified. Signed-off-by: Richard Edgar <[email protected]> * fix dependency chart axis updating with incorrect values in explanation dashboard (#1437) * fix codecov and widget test screenshot uploads (#1428) * release raiwidgets and responsibleai v0.18.2 (#1439) * fix (#1441) Signed-off-by: vinutha karanth <[email protected]> * Fix cohort name conflict and not run few tests for AML (#1442) * fix Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * Few e2e tests changes to accommodate AML static tests (#1445) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * Fix locators logic for string features - data explorer and model statistics components (#1446) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * Add more unittests RAI dashboard input class (#1448) * Add unit tests for ResponsibleAIDashboardInput Signed-off-by: Gaurav Gupta <[email protected]> * Add more tests Signed-off-by: Gaurav Gupta <[email protected]> * Fix imports Signed-off-by: Gaurav Gupta <[email protected]> * Address code review comments Signed-off-by: Gaurav Gupta <[email protected]> * more update Signed-off-by: vinutha karanth <[email protected]> * Update the way to get the length of elements obtained in e2e tests (#1450) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * skip what-if create tests for AML Signed-off-by: vinutha karanth <[email protected]> Co-authored-by: Ilya Matiach <[email protected]> Co-authored-by: Roman Lutz <[email protected]> Co-authored-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> Co-authored-by: Richard Edgar <[email protected]>

…ai-toolbox into romanlutz/test_model_overview

…el (#1458) * fix error on machines with pyspark installed where passed dataframe is not spark pandas (#1415) * add postbuild branch trigger (#1417) * Fix causal UI strings according to classification/regression tasks (#1419) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Fix description for model overview (#1425) * fix description for model overview * keep new description for new model overview * fix failing to create error report when filter_features is empty list (#1421) * filter out missing values from what if dropdown to prevent explanation dashboard from crashing (#1418) * Remove |Set Value| blurb in case it is not availble in counterfactual panel (#1426) Signed-off-by: Gaurav Gupta <[email protected]> * Add y-axis description to counterfactual feature importance chart (#1423) Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * Add the user class name to causal UI strings (#1422) * Fix causal UI strings according to classification/regression tasks Signed-off-by: Gaurav Gupta <[email protected]> * Fix lint error Signed-off-by: Gaurav Gupta <[email protected]> * Fix UI test Signed-off-by: Gaurav Gupta <[email protected]> * Add the user class name to causal UI strings Signed-off-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> * fix math.min / max for array size more than 10^7 (#1427) Signed-off-by: Ke Xu <[email protected]> * upgrade pytest and lightgbm to try to fix random pytest segfault test failures (#1424) s * fix flaky notebook causing build failures by adding retry logic (#1431) * Upper bound SciKit-Learn to address freeze in causal (#1432)  ## Description Replaces #1429 to address #1430 . Causal analysis is getting stuck with the latest release of SciKit-Learn. This contains: - Test case which gets stuck with SciKit-Learn 1.1.0 - Upper bound on SciKit-Learn in `requirements.txt` ## Checklist  - [x] I have added screenshots above for all UI changes. - [x] Documentation was updated if it was needed. - [x] New tests were added or changes were manually verified. Signed-off-by: Richard Edgar <[email protected]> * fix dependency chart axis updating with incorrect values in explanation dashboard (#1437) * fix codecov and widget test screenshot uploads (#1428) * release raiwidgets and responsibleai v0.18.2 (#1439) * fix (#1441) Signed-off-by: vinutha karanth <[email protected]> * Fix cohort name conflict and not run few tests for AML (#1442) * fix Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * Few e2e tests changes to accommodate AML static tests (#1445) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * Fix locators logic for string features - data explorer and model statistics components (#1446) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * lintfix Signed-off-by: vinutha karanth <[email protected]> * fix Signed-off-by: vinutha karanth <[email protected]> * Add more unittests RAI dashboard input class (#1448) * Add unit tests for ResponsibleAIDashboardInput Signed-off-by: Gaurav Gupta <[email protected]> * Add more tests Signed-off-by: Gaurav Gupta <[email protected]> * Fix imports Signed-off-by: Gaurav Gupta <[email protected]> * Address code review comments Signed-off-by: Gaurav Gupta <[email protected]> * Update the way to get the length of elements obtained in e2e tests (#1450) * update Signed-off-by: vinutha karanth <[email protected]> * update Signed-off-by: vinutha karanth <[email protected]> * skip what-if create tests for AML Signed-off-by: vinutha karanth <[email protected]> * update cf predicted class to non-editable text Co-authored-by: Ilya Matiach <[email protected]> Co-authored-by: Roman Lutz <[email protected]> Co-authored-by: Gaurav Gupta <[email protected]> Co-authored-by: xuke444 <[email protected]> Co-authored-by: Richard Edgar <[email protected]> Co-authored-by: Vinutha Karanth <[email protected]>

… in e2e-widget.js

…ai-toolbox into romanlutz/test_model_overview

…olbox into romanlutz/test_model_overview

codecov-commenter · 2022-06-06T20:52:57Z

Codecov Report

Merging #1467 (3ac1c73) into main (b4ba63c) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main    #1467   +/-   ##
=======================================
  Coverage   87.27%   87.27%           
=======================================
  Files         108      108           
  Lines        5108     5108           
=======================================
  Hits         4458     4458           
  Misses        650      650

Flag	Coverage Δ
unittests	`87.27% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b4ba63c...3ac1c73. Read the comment docs.

github-actions · 2022-06-06T21:56:57Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-06T22:00:03Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

…olbox into romanlutz/test_model_overview

github-actions · 2022-06-23T18:59:42Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-23T19:01:27Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

…olbox into romanlutz/test_model_overview

github-actions · 2022-06-24T03:09:31Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-24T03:11:52Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-24T03:42:35Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-24T03:45:03Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

github-actions · 2022-06-24T04:17:53Z

https://responsibleai.blob.core.windows.net/pullrequest/microsoft/responsible-ai-toolbox/romanlutz/test_model_overview/dashboard/index.html

romanlutz and others added 21 commits May 17, 2022 15:19

execute e2e tests even with flighted versions of dashboard

3d54e2b

Merge branch 'romanlutz/postbuild_update' into romanlutz/test_model_o…

1babf25

…verview

rename model stats to model overview, add v1 and v2 for new experience

e5ef474

basic test harness for individual notebook execution and flighted not…

35ca493

…ebook execution, more ID tags added to model overview component

more ids in model overview

a8fd08d

Merge branch 'main' of https://github.com/microsoft/responsible-ai-to…

25ff1ef

…olbox into romanlutz/test_model_overview

Merge branch 'postbuild' of https://github.com/microsoft/responsible-…

7a62ee4

…ai-toolbox into romanlutz/test_model_overview

remove randomness from notebooks

7cfa76d

small fixes

b95fd62

lots of new IDs and tests

acc46f8

make first set of tests work for first dataset

4aeeb58

Merge branch 'postbuild' of https://github.com/microsoft/responsible-…

2a53ea9

…ai-toolbox into romanlutz/test_model_overview

document e2e-widget options in CONTRIBUTING.md and flightCombinations…

8e792f3

… in e2e-widget.js

Merge branch 'postbuild' of https://github.com/microsoft/responsible-…

f39aa43

…ai-toolbox into romanlutz/test_model_overview

lintfix

901ce58

fix torch bytetensor input type

609a0d5

torch manual seed instead of set_rng_state

cc7e056

Merge branch 'main' of https://github.com/microsoft/responsible-ai-to…

6e29a34

…olbox into romanlutz/test_model_overview

romanlutz requested review from xuke444, gaugup, tongyu-microsoft and vinuthakaranth as code owners June 6, 2022 20:48

romanlutz changed the title ~~Romanlutz/test model overview~~ Add e2e tests for new model overview experience (part 1) Jun 6, 2022

Merge branch 'main' into romanlutz/test_model_overview

a8cb77a

tongyu-microsoft approved these changes Jun 21, 2022

View reviewed changes

romanlutz added 11 commits June 22, 2022 00:14

fix heatmap cell test

058f677

use lodash

84c17d6

lintfix

9203e0f

Merge branch 'main' of https://github.com/microsoft/responsible-ai-to…

b7d0866

…olbox into romanlutz/test_model_overview

delete spec files that aren't required

f3bc17d

simplify modelAssessmentDatasets.ts using lodash

acf740f

make e2e-widget.js work with empty -f arg

d6b6431

add error throwing back

de378e6

move id inside component and remove extra div

dce982f

lintfix

2f4fb15

Merge branch 'main' into romanlutz/test_model_overview

8208594

xuke444 approved these changes Jun 23, 2022

View reviewed changes

romanlutz added 2 commits June 23, 2022 22:00

Merge branch 'main' of https://github.com/microsoft/responsible-ai-to…

01283df

…olbox into romanlutz/test_model_overview

adjust for aria-label in regression case

50e4c88

romanlutz requested a review from imatiach-msft as a code owner June 24, 2022 02:17

romanlutz added 3 commits June 23, 2022 22:19

add multiclass aria-label

38af34d

lintfix

3bc3f36

lintfix

8ca4e60

Merge branch 'main' into romanlutz/test_model_overview

3ac1c73

romanlutz merged commit f52539a into main Jun 24, 2022

romanlutz deleted the romanlutz/test_model_overview branch June 24, 2022 04:51

imatiach-msft mentioned this pull request Jul 18, 2022

release raiwidgets and responsibleai 0.20.0 #1563

Merged

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add e2e tests for new model overview experience (part 1) #1467

Add e2e tests for new model overview experience (part 1) #1467

romanlutz commented Jun 6, 2022 •

edited

Loading

codecov-commenter commented Jun 6, 2022 •

edited

Loading

github-actions bot commented Jun 6, 2022

github-actions bot commented Jun 6, 2022

github-actions bot commented Jun 23, 2022

github-actions bot commented Jun 23, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

Add e2e tests for new model overview experience (part 1) #1467

Add e2e tests for new model overview experience (part 1) #1467

Conversation

romanlutz commented Jun 6, 2022 • edited Loading

Description

Checklist

codecov-commenter commented Jun 6, 2022 • edited Loading

Codecov Report

github-actions bot commented Jun 6, 2022

github-actions bot commented Jun 6, 2022

github-actions bot commented Jun 23, 2022

github-actions bot commented Jun 23, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

github-actions bot commented Jun 24, 2022

romanlutz commented Jun 6, 2022 •

edited

Loading

codecov-commenter commented Jun 6, 2022 •

edited

Loading