collator-proto/metrics: Fix blindspot in collation fetch latency metrics

[lexnv]

The current implementation of the collation_fetch_latency metric contains a critical observability blindspot due to insufficient histogram bucket resolution.

Currently, the the collation fetching is capped at an upper bound of 5 seconds. This effectively creates a black box for investigating latency events. Any fetch operation exceeding the 5s threshold is aggregated into the final bucket, regardless if the fetching took 30s or 1h. This obscures the true distribution of network delays and prevents accurate performance profiling for high-latency scenarios.

The discrepancy was identified with https://github.com/lexnv/block-confidence-monitor and confirmed via manual analysis of the logs. Without granular visibility into these outliers, we cannot effectively measure the success of our block confidence / debug bottlenecks in validator-side protocols.

While at it, have increased the granularity of other buckets which might be relevant.

Part of the block confidence work:

• Lower Block Confidence on Polkadot 96% vs Kusama parachains 99% #11377

cc @sandreim @skunert

[skunert]

So with @sandreim comments, does this change even make sense? If the candidates drop anyway, we should never see entries in the buckets > 10 right?

[lexnv]

This is mostly needed for collation_fetch_latency - How much time collations spend waiting to be fetched. We can already see this on polkadot-yap 3428: the 2 spikes to 10s would have been capped at 5s (ie the blindspot is real here):

https://grafana.teleport.parity.io/goto/aQvXSItDg?orgId=1

So with @sandreim comments, does this change even make sense? If the candidates drop anyway, we should never see entries in the buckets > 10 right?

Have reverted the metrics where I bumped the block times, yep.

[sandreim]

I'd remove some of the lower end buckets, anything lower than 250ms is excellent considering some reasonable latency validator/collator.

[lexnv]

Graph showing the latency spikes to 30s that were captured by the block-confidence-monitoring tool, we should have better insights into these issues.

We have also deployed a trace version with extra monitoring on validators, which should give us more details into what is happening on the validator-side of the code 🙏

Thanks a lot for the review 🙏

[lexnv]

/cmd prdoc --audience node_dev --bump patch