include a mask for ece calculation #77
Conversation
| sample_weight = tf.reshape(sample_weight, [-1]) | ||
|
|
||
| probabilities = tf.boolean_mask(probabilities, sample_weight) | ||
| labels = tf.boolean_mask(labels, sample_weight) |
There was a problem hiding this comment.
Thinking about this further, if we added a sample_weight parameter to this method, to follow the convention in tf.keras.metrics.Metric, we'd need sample_weight to be a (broadcastable) tensor of continuous values that are multiplied with the samples in some manner. It's a simple weighted average for Accuracy, and more involved for something like AUC. For ECE, we'd need to think more about how to add a general sample_weight, i.e., how to compute ECE with continuously-weighted samples. Binary masking is a subset of the cases that should be handled to follow the Keras API.
For the case of masking out a subset of samples, I'd suggest that we move that logic to the calling site, so in the training code that calls this metric rather than inside the metric.
This code includes a sample_weight variable to mask the logits used to calculate the ECE.