wrong event ordering when used with HTTP/2

[RoadRunnr]

I'm not totally sure whether this is a bug in cowboy_telemetry or in the way the telemetry events are use by opentelemetry_cowboy.

telemetry:execute seems to have the assumption the all events send from a process belong to the same request or span.

The way cowboy handles HTTP/2 and invokes it stream handlers results in a violation of that assumption.

cowboy uses a single process for each TCP connection. All stream handlers will be invoked from that process. However, a single HTTP/2 connection can have multiple HTTP/2 streams. That means that events from different HTTP/2 requests can all be invoked from the same process. Requests on different HTTP/2 streams are not synchronized. That can result in interleaving calls to telemetry:execute.

The interleaving might not be a problem for metrics. But when the requests are used to start and stop tracing spans with only the information from the process itself, this will go wrong.