Add ability to set attributions in WIT #2252

jameswex · 2019-05-20T19:37:42Z

Motivation for features / changes

Enabling WIT to be able to display feature-level attributions for individual predictions if the model provides it.

Technical description of changes

Update visuals of attribution(saliency) viewing in vz-example-viewer, including better layout/coloring, and removing some unnecessary controls.
Update inference_utils.py logic to be able to handle inference results with new, optional attribution values.
Update WitWidget custom_predict_functions to be able to optionally return attributions in addition to predictions, by returning a dict containing both. Update parsing of those results to handle that case.
Update AI platform custom predict function to pass any attributions it generates back through that newly-defined dict.
Update WitWidget colab and jupyter code to pass those optional attributions to javascript when inferences are made.
Update WIT element to handle attributions passed to it by updating the visdata for the elements that have attribution information to include that information so it is visualized by vz-example-viewer, and usable in Facets Dive.

Screenshots of UI changes

Detailed steps to verify changes work correctly (as executed by you)

Created custom predict function that created fake attributions and inferences, tested it in colab and jupyter notebooks to see correct visual handling of those attributions.

Alternate designs / implementations considered

jameswex · 2019-05-20T19:38:55Z

@tolga-b please review

tolga-b

Added some comments.

tolga-b · 2019-05-22T17:28:37Z

tensorboard/components/vz_example_viewer/vz-example-viewer.ts

@@ -397,7 +397,7 @@ Polymer({
          (d: {}, i: number) => this.getColorForSaliency(val[i]) :
          () => this.getColorForSaliency(val);
      this.selectAll(
-            `input.${this.sanitizeFeature(feat.name)}.value-pill`)
+            `iron-autogrow-textarea.${this.sanitizeFeature(feat.name)}.value-pill`)


General comment, it would be better if this code did not depend on the type of text box (iron-autogrow-textarea or input). It looks a bit fragile right now in case we forget to change this when html side is updated.

This code actually works correctly without the selector for the element this, so i removed it. Thanks for catching this.

tolga-b · 2019-05-22T18:01:55Z

tensorboard/plugins/interactive_inference/witwidget/notebook/base.py

@@ -122,8 +123,11 @@ def infer_impl(self):
      self.estimator_and_spec.get('estimator'),
      self.estimator_and_spec.get('feature_spec'),
      self.custom_predict_fn)
-    infer_objs.append(inference_utils.run_inference_for_inference_results(
+    (preidctions, attributions) = (


tolga-b · 2019-05-22T18:08:33Z

tensorboard/plugins/interactive_inference/witwidget/notebook/visualization.py

@@ -413,6 +413,24 @@ def set_custom_predict_fn(self, predict_fn):
      - For regression: A 1D list of numbers, with a regression score for each
        example being predicted.

+    Optionally, if attributions can be returned by the model with each
+    prediction, then this method can return a dict with the key 'predictions'
+    containing the preidctions result list described above, and with the key


typo preidctions, this happens in a lot of places so mass replace should fix it.

tolga-b · 2019-05-22T18:14:44Z

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

+        for (let i = 0; i < attributions.indices.length; i++) {
+          const idx = attributions.indices[i];
+          const datapoint = this.visdata[idx];
+          // For now, we only display attribution from the first model, if WIT


Add TODO here for two models

tolga-b · 2019-05-22T18:16:12Z

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

@@ -2186,6 +2186,27 @@ <h2>Show similarity to selected datapoint</h2>
          observer: 'newInferences_',
          value: () => ({})
        },
+        // Attributions from inference. A dict with two fields: 'indices' and
+        // 'attributions'. Inidicies contains a list of example indices that


typo Inidicies

jameswex · 2019-05-22T20:30:55Z

@stephanwlee please review thx

stephanwlee · 2019-05-22T20:41:33Z

tensorboard/plugins/interactive_inference/interactive_inference_plugin.py

-        infer_objs.append(inference_utils.run_inference_for_inference_results(
-          examples_to_infer, serving_bundle))
+        (predictions, _) = inference_utils.run_inference_for_inference_results(
+          examples_to_infer, serving_bundle)


+2 whitespace per python style guide.

stephanwlee · 2019-05-23T19:06:53Z

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

+        attributions: {
+          type: Object,
+          observer: 'newAttributions_',
+          value: () => ({})


null might be better.

stephanwlee · 2019-05-23T19:10:12Z

...ractive_inference/tf_interactive_inference_dashboard/tf-interactive-inference-dashboard.html

+              datapoint[attribSparseValueKey] = attribs[keys[j]][1];
+            } else {
+              const attribKey = attributionPrefix + keys[j];
+              datapoint[attribKey] = attribs[keys[j]];


it feels un-Polymermic to mutate data like this and then trigger a draw (as opposed to using prop changes to trigger data binding). Can you consider creating another object and do this.set(`visdata[${index}]`, newObject)?

jameswex and others added 29 commits March 19, 2018 19:38

Add flags to TBContext

d84a0eb

Add flags to TBContext in application

ab0e9cf

Add flags to tbcontext

e72cbcf

gitignore changes

3803ab7

Merge branch 'master' of https://github.com/tensorflow/tensorboard

286df31

Merge remote-tracking branch 'upstream/master'

ab7ed2d

Merge remote-tracking branch 'upstream/master'

ccb962a

fix merge issue

32bf7e8

Merge remote-tracking branch 'upstream/master'

67c0e07

Merge remote-tracking branch 'upstream/master'

5c6aa65

Merge remote-tracking branch 'upstream/master'

f1f67e6

Merge remote-tracking branch 'upstream/master'

aae6cea

Merge remote-tracking branch 'upstream/master'

ac0ff40

Merge remote-tracking branch 'upstream/master'

da58894

Merge remote-tracking branch 'upstream/master'

9499cdf

Merge remote-tracking branch 'upstream/master'

6a1f6a8

Merge remote-tracking branch 'upstream/master'

1e52fd9

Merge remote-tracking branch 'upstream/master'

5ac08a5

Merge remote-tracking branch 'upstream/master'

f99b90f

Merge remote-tracking branch 'upstream/master'

45ae026

Merge remote-tracking branch 'upstream/master'

66ba5d8

Merge remote-tracking branch 'upstream/master'

e53cc99

Merge remote-tracking branch 'upstream/master'

21333bc

experiment

e352c78

fix attributions

487a0d5

fix sparse attributions

4c349a2

line

01d4f9a

handle attributions

28fb67b

cleanup

243827b

jameswex added the plugin:what-if-tool label May 20, 2019

tolga-b reviewed May 22, 2019

View reviewed changes

review comments

fe42fc1

jameswex requested a review from stephanwlee May 22, 2019 20:30

tolga-b approved these changes May 23, 2019

View reviewed changes

stephanwlee reviewed May 23, 2019

View reviewed changes

review comments

4fd0342

stephanwlee approved these changes May 23, 2019

View reviewed changes

jameswex merged commit 49fed7e into tensorflow:master May 24, 2019

jameswex deleted the attribution branch May 24, 2019 12:55

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ability to set attributions in WIT #2252

Add ability to set attributions in WIT #2252

jameswex commented May 20, 2019 •

edited

Loading

jameswex commented May 20, 2019

tolga-b left a comment

tolga-b May 22, 2019

jameswex May 22, 2019

tolga-b May 22, 2019

jameswex May 22, 2019

tolga-b May 22, 2019

jameswex May 22, 2019

tolga-b May 22, 2019

jameswex May 22, 2019

tolga-b May 22, 2019

jameswex May 22, 2019

jameswex commented May 22, 2019

stephanwlee May 22, 2019

jameswex May 23, 2019

stephanwlee May 23, 2019

stephanwlee May 23, 2019

jameswex May 23, 2019

Add ability to set attributions in WIT #2252

Add ability to set attributions in WIT #2252

Conversation

jameswex commented May 20, 2019 • edited Loading

jameswex commented May 20, 2019

tolga-b left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameswex commented May 22, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jameswex commented May 20, 2019 •

edited

Loading