Incorrect CRPS calculation

[c-lyu]

1. System Info

PyPOTS version: 0.8.1

This bug is independent of operating environment.

2. Information

• The official example scripts
• My own created scripts

3. Reproduction

The CRPS calculation seems to be incorrect. Line 318, targets is passed as the first argument, and q_pred is the second; whereas according to Line 257, the first argument should be predictions and the second should be targets.

PyPOTS/pypots/utils/metrics/error.py

Lines 257 to 320 in 7239b51

	def calc_quantile_loss(predictions, targets, q: float, eval_points) -> float:
	quantile_loss = 2 * torch.sum(
	torch.abs((predictions - targets) * eval_points * ((targets <= predictions) * 1.0 - q))
	)
	return quantile_loss
	def calc_quantile_crps(
	predictions: Union[np.ndarray, torch.Tensor],
	targets: Union[np.ndarray, torch.Tensor],
	masks: Union[np.ndarray, torch.Tensor],
	scaler_mean=0,
	scaler_stddev=1,
	) -> float:
	"""Continuous rank probability score for distributional predictions.
	Parameters
	----------
	predictions :
	The prediction data to be evaluated.
	targets :
	The target data for helping evaluate the predictions.
	masks :
	The masks for filtering the specific values in inputs and target from evaluation.
	Only values at corresponding positions where values ==1 in ``masks`` will be used for evaluation.
	scaler_mean:
	Mean value of the scaler used to scale the data.
	scaler_stddev:
	Standard deviation value of the scaler used to scale the data.
	Returns
	-------
	CRPS :
	Value of continuous rank probability score.
	"""
	# check shapes and values of inputs
	_ = _check_inputs(predictions, targets, masks, check_shape=False)
	if isinstance(predictions, np.ndarray):
	predictions = torch.from_numpy(predictions)
	if isinstance(targets, np.ndarray):
	targets = torch.from_numpy(targets)
	if isinstance(masks, np.ndarray):
	masks = torch.from_numpy(masks)
	targets = targets * scaler_stddev + scaler_mean
	predictions = predictions * scaler_stddev + scaler_mean
	quantiles = np.arange(0.05, 1.0, 0.05)
	denominator = torch.sum(torch.abs(targets * masks))
	CRPS = torch.tensor(0.0)
	for i in range(len(quantiles)):
	q_pred = []
	for j in range(len(predictions)):
	q_pred.append(torch.quantile(predictions[j : j + 1], quantiles[i], dim=1))
	q_pred = torch.cat(q_pred, 0)
	q_loss = calc_quantile_loss(targets, q_pred, quantiles[i], masks)
	CRPS += q_loss / denominator
	return CRPS.item() / len(quantiles)

4. Expected behavior

Line 318 should be q_loss = calc_quantile_loss(q_pred, targets, quantiles[i], masks).

[github-actions]

Hi there 👋,

Thank you so much for your attention to PyPOTS! You can follow me on GitHub
to receive the latest news of PyPOTS. If you find PyPOTS helpful to your work, please star⭐️ this repository.
Your star is your recognition, which can help more people notice PyPOTS and grow PyPOTS community.
It matters and is definitely a kind of contribution to the community.

I have received your message and will respond ASAP. Thank you for your patience! 😃

Best,
Wenjie