Design suggestions of _IterationBuilder._best_loss from adanet.core.iteration

[linjing-lab]

I have designed some operators for model loss monitoring, like the following continuous fragments:

https://github.com/linjing-lab/easy-pytorch/blob/9651774dcc4581104f914980baf2ebc05f96fd85/released_box/perming/_utils.py#L269-L281

and I achieved it for this approach taking a small proportion in CPU runtime, I don't want to burden the CPU runtime without the need to detect the optimal model after training runtime.

adanet/adanet/core/iteration.py

Lines 743 to 747 in 0364cc4

	best_candidate_index = self._best_candidate_index(
	candidates, best_ensemble_index_override)
	best_predictions = self._best_predictions(candidates,
	best_candidate_index)
	best_loss = self._best_loss(candidates, best_candidate_index, mode)

I don't think the underlined function is a quick response code because it selects the optimal model at least O(n) runtime from all trained candidate ensembles only by loss evaluation. It's an implementation based on the tf API, but it doesn't take into account how users can get the ideal combination based on the strategy they are using in as few trials as possible.

adanet/adanet/core/iteration.py

Lines 1089 to 1109 in 0364cc4

	def _best_loss(self, candidates, best_candidate_index, mode):
	"""Returns the best loss from a set of candidates.
	Args:
	candidates: List of `_Candidate` instances to compare.
	best_candidate_index: `Tensor` index of the best candidate in the list.
	mode: Defines whether this is training, evaluation or inference. Loss is
	always None during inference. See `ModeKeys`.
	Returns:
	Float `Tensor` of the best candidate's loss.
	"""
	if mode == tf.estimator.ModeKeys.PREDICT:
	return None
	if len(candidates) == 1:
	return candidates[0].ensemble_spec.loss
	with tf_compat.v1.variable_scope("best_loss"):
	losses = [c.ensemble_spec.loss for c in candidates]
	loss = tf.slice(tf.stack(losses), [best_candidate_index], [1])
	return tf.reshape(loss, [])