Definition of Miss Rate

[a-pru]

Hi,

In your prediction evaluation code, a prediction is deemed a miss if the maximum distance between any predicted trajectory and the ground truth at any time t exceeds the defined threshold.

In contrast, Argoverse and WOMD use the following definitions:
Argoverse 1/2: The number of scenarios where none of the forecasted trajectories are within 2.0 meters of ground truth according to endpoint error.
WOMD: A miss is defined as the state when none of the individual K predictions for an object are within a given lateral and longitudinal threshold of the ground truth trajectory at a given time T.

Could you elaborate on the rationale of using maximum displacement instead of minimum displacement to compute the miss rate?
My intuition aligns rather with AV1/2 and WOMD: diverse trajectories are desirable (not all predictions should collapse to the same mode), and if at least one predicted trajectory matches the ground truth, I would typically not consider it a miss.

Thank you and all the best,
Alex

[whyekit-motional]

@a-pru if you take the max over the last two dimensions of an array of shape [num_modes, horizon_length, state_dim], then the resulting array will have shape [num_modes]:

import numpy as np


stacked_trajs = np.random.rand(5, 4, 3)
stacked_ground_truth = np.random.rand(5, 4, 3)
result = np.max(np.linalg.norm(stacked_trajs - stacked_ground_truth, axis=-1), axis=-1)
print(result.shape)  # (5,)

So the max is over the horizon length (and not the modes)

[a-pru]

Of course, you're right—thanks for the clarification and sorry for the confusion.