Bug in infractions/km calculation in compute_global_statistics

[Kait0]

I think there is a bug in the computation of the infractions / km metrics in the current leaderboard repository (master):
The file is leaderboard/utils/statistics_manager.py
Function def compute_global_statistics(self, total_routes):

...
if self._registry_route_records:
        for route_record in self._registry_route_records:
                 global_record.scores['score_route'] += route_record.scores['score_route']
                 global_record.scores['score_penalty'] += route_record.scores['score_penalty']
                 global_record.scores['score_composed'] += route_record.scores['score_composed']

                 for key in global_record.infractions.keys():
                     route_length_kms = max(route_record.scores['score_route'] / 100 * route_record.meta['route_length'] / 1000.0, 0.001)
                     if isinstance(global_record.infractions[key], list):
                          global_record.infractions[key] = len(route_record.infractions[key]) / route_length_kms
                     else:
                          global_record.infractions[key] += len(route_record.infractions[key]) / route_length_kms
 
                 if route_record.status is not 'Completed':
                     global_record.status = 'Failed'
                     if 'exceptions' not in global_record.meta:
                         global_record.meta['exceptions'] = []
                     global_record.meta['exceptions'].append((route_record.route_id,
                                                              route_record.index,
                                                              route_record.status))
 
             for key in global_record.scores.keys():
                 global_record.scores[key] /= float(total_routes)
             ...

In the line:
global_record.infractions[key] += len(route_record.infractions[key]) / route_length_kms

The infractions/km from all the individual routes simply get added together, leading to nonsense value that are dependent on the number of routes.
The current code implements the following formula:

where c_i is the number of collisions in route, km_i is the number of km driven in route i. N is the number of routes.

A naive fix would be to divide by the number of routes:

However this is not exactly the correct calculation.
Slicing the driven km into route segments changes the result of the infraction / km metric and in the worst case can lead to a simpsons paradox.

What we want to compute is the total number of collisions / total number of km driven so the correct formula is:

which in code would looks something like adding the infractions (per type) as well as adding the driven (!) km to variables outside the for_loop and after the route for loop dividing the counts by the total number of driven kms.

[Kin-Zhang]

leading to nonsense value that are dependent on the number of routes.

I knew what you mean here, but the infraction on global record may want to the result as the formula here.

The next one actually cannot seem like the average of collisions on routes kilometers. but average on all kilometers with fading the route concept?

But I agree with your opinion. since the first one will cause some number really big when the route_length_kms on one route are really small.

[Kait0]

The next one actually cannot seem like the average of collisions on routes kilometers. but average on all kilometers with fading the route concept?

Precisely, the concept of a route is not part of the infractions / km metrics.
If you look at the official descriptions on the leaderboard website, they say for example:
Collision Vehicles: "Number of collisions with other vehicles, normalized per km."

No route concept is mentioned there.
It is not a metric that averages (the routes) like for example the route completion which says:
Route completion: "Average percentage of routes completed."

[Kait0]

As a sidenote one needs to take additional care with the off-road infraction metric.
Right now I think the leaderboard always reports it as one infraction adding together the total amount of meters driven off road.
So counting does not work for that infraction since it is either 1 or 0.
With the current system one can sum up the (kilo!) meters driven off road (and divide by the total number of km driven) to get a meaningful estimate (maybe multiply by 100 because that is a percentage).

The other option would be to change how off-road is reported and report every single instance where the agent drove off the road (and then just count as usual).

[glopezdiest]

Hey @Kait0, thanks for the detailed explanation. I'll take a look at it and report back

[glopezdiest]

After some inspection, it does seem to be misscalculated. We are indeed using

to calculate the infractions per km, while in reality we want

I'll update the Leaderboard as soon as I can with this fix, thanks 🙂

[glopezdiest]

With the coming release of the LB 2.0, this issue has been fixed. It is currently only available in the leaderboard-2.0 branch, but for those interested, the fix is done in this PR https://github.com/carla-simulator/leaderboard/pull/130/files (along with many other things).