Rethink logging

[vcolin7]

From here:

In a perfect world, I'd rethink how we do HTTP logging. I'd use one policy for tracing and logs and would by default print formatted traces to stdout adding all the headers/content. This would reduce number of policies, instrumentation overhead, provide consistent property names and will have other benefits.

Happy to prototype this if there is interest (and in the meantime it can be hidden from users and just added in a defaultPipeline factory method).

[samvaity]

Follow up discussions:

Liudmila Molkova
" to have logging tracing and metrics in the same policy which could remove some duplication (like wrapping response to detect if it fails after response code is received) and make log correlation so much better."

Srikanta Nagaraja
Interesting idea - we could have a policy that combines logging, metrics and traces but I think there are still use cases where they can be used individually?
so, have separate policies but also have another policy to combine them?

Liudmila Molkova
It would also work, essentially I want them to be aware of each other and work together (e.g. tracing and logging would share the same response wrapper, logging would have trace-id, span-id available, logging could reuse otel semantic conventions, they could share config like allowed headers, etc)
I guess I can prototype with otel-api and we can independently decide if we need an abstraction layer

[vcolin7]

Related issue: #39606.

[lmolkova]

The remaining work affecting public API and overall design:

• implement logging/other instrumentation inside HTTP clients - this way we have it guaranteed (enabling the policy and the order are now parts of client lib logic). It also makes it more clear during debugging if the operation has ever reached the HTTP client.
• implement tracing/metrics hooks in the same places as logging. Implementing custom request/response logger should be done in the same way as custom tracing/metrics.
• nice-to-haves: support explicit correlation for logs
• unify as much as possible between different instrumentation signals. They should still be consumable independently, but work best together with minimal overhead for individual signal.

[samvaity]

Putting related content together: https://gist.github.com/alzimmermsft/bb86b046c4b70abd52ef9d1ecb05bc54

[samvaity]

#43678