Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable dropped_features for error analysis, data explorer and explainer, counterfactual and causal managers #1824

Merged
merged 74 commits into from
Dec 7, 2022

Conversation

tongyu-microsoft
Copy link
Contributor

@tongyu-microsoft tongyu-microsoft commented Dec 1, 2022

This PR adds dropped_features and pass dropped_features to error analysis, data explorer and explainer managers if there is any.

GIF:
in the below regression example, we dropped ["s1", "s5"] features:
dropped_features_RA

for counterfactual:
dropped_features_cf

for causal:
dropped_features_causal

Description

  1. for error analysis, we display dropped_features, since feature importances are the correlation between the feature column and the error column, and any feature column can be computed.
  2. for explainer, we hide dropped_features, since displaying dropped_features that aren't fed to the model doesn't make sense.
  3. for data explorer, we still show the dropped_features.
  4. for counterfactual, we hide the dropped_features.
  5. for causal, we hide the dropped_features.

We will update the example notebooks, doc and add more tests in my next PR.

Checklist

  • I have added screenshots above for all UI changes.
  • I have added e2e tests for all UI changes.
  • Documentation was updated if it was needed.

gaugup and others added 30 commits August 19, 2022 16:19
Signed-off-by: Gaurav Gupta <[email protected]>
Signed-off-by: Gaurav Gupta <[email protected]>
Signed-off-by: Gaurav Gupta <[email protected]>
Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
* Add model wrapper for wrapping predictions and test data

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix failing tests

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
…ortance (#1770)

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
* Port tests

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix rai_insights

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
* Simply tests in test_cohort_filter.py

Signed-off-by: Gaurav Gupta <[email protected]>

* Change test name

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
* Move cohort re-labelling logic to Cohort.ts

Signed-off-by: Gaurav Gupta <[email protected]>

* Lint fix

Signed-off-by: Gaurav Gupta <[email protected]>

* Lint fix

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
…nceView (#1775)

* Refactor DataAnalysisTab into Chartview/DataAnalysisView and DataBalanceView

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix lint

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
… highcharts (#1783)

* enable csv xls download in hightcharts

* localization
* Add rai_test_utils package

Signed-off-by: Gaurav Gupta <[email protected]>

* Lint fix and namespace changes

Signed-off-by: Gaurav Gupta <[email protected]>

* Fix duplicated cohorts not retaining the selection in combobox (#1785)

* fix and e2e

Signed-off-by: vinutha karanth <[email protected]>

* update filter

Signed-off-by: vinutha karanth <[email protected]>

Signed-off-by: vinutha karanth <[email protected]>

* release raiwidgets and responsibleai 0.23.0 (#1787)

* fix release github action for raiwidgets and responsibleai by making it same as in gated build (#1788)

* Add e2e for dependency plot Avg of abs value option (#1768)

* fix dep plot

Signed-off-by: vinutha karanth <[email protected]>

* add e2e

Signed-off-by: vinutha karanth <[email protected]>

* lintfix

Signed-off-by: vinutha karanth <[email protected]>

* lintfix

Signed-off-by: vinutha karanth <[email protected]>

Signed-off-by: vinutha karanth <[email protected]>

* update e2e visit function (#1792)

* Fix code review comments

Signed-off-by: Gaurav Gupta <[email protected]>

Signed-off-by: Gaurav Gupta <[email protected]>
Signed-off-by: vinutha karanth <[email protected]>
Co-authored-by: Vinutha Karanth <[email protected]>
Co-authored-by: Ilya Matiach <[email protected]>
Co-authored-by: tongy-msft <[email protected]>
Signed-off-by: vinutha karanth <[email protected]>

Signed-off-by: vinutha karanth <[email protected]>
Copy link
Contributor

@imatiach-msft imatiach-msft left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved with some nits

@codecov-commenter
Copy link

codecov-commenter commented Dec 2, 2022

Codecov Report

Merging #1824 (e0b0110) into main (b733264) will increase coverage by 5.77%.
The diff coverage is 94.36%.

@@            Coverage Diff             @@
##             main    #1824      +/-   ##
==========================================
+ Coverage   82.82%   88.60%   +5.77%     
==========================================
  Files          55       96      +41     
  Lines        2935     5547    +2612     
==========================================
+ Hits         2431     4915    +2484     
- Misses        504      632     +128     
Flag Coverage Δ
unittests 88.60% <94.36%> (+5.77%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...sibleai/responsibleai/rai_insights/rai_insights.py 93.57% <93.02%> (ø)
...i/responsibleai/managers/error_analysis_manager.py 89.39% <95.45%> (ø)
...idgets/raiwidgets/responsibleai_dashboard_input.py 91.50% <100.00%> (+0.05%) ⬆️
...ibleai/responsibleai/managers/explainer_manager.py 89.55% <100.00%> (ø)
...leai/responsibleai/modelanalysis/model_analysis.py 94.64% <100.00%> (ø)
...onsibleai/responsibleai/managers/causal_manager.py 95.51% <0.00%> (ø)
...eai/responsibleai/managers/data_balance_manager.py 97.60% <0.00%> (ø)
...sibleai/responsibleai/_tools/shared/base_result.py 95.65% <0.00%> (ø)
responsibleai/responsibleai/_config/base_config.py 92.30% <0.00%> (ø)
... and 36 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

@tongyu-microsoft tongyu-microsoft changed the title Enable dropped_features for error analysis, data explorer and explainer managers Enable dropped_features for error analysis, data explorer and explainer, counterfactual and causal managers Dec 5, 2022
Copy link
Contributor

@romanlutz romanlutz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving since these are all minor comments.

@@ -82,6 +82,35 @@ def as_error_config(json_dict):
return json_dict


class MetadataRemovalModelWrapper():
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will this go into mlwrappers eventually? @imatiach-msft
Note: this is not a blocker for this PR regardless of the answer.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants