Guide for configuring tracing

[R-Lawton]

WHAT:

Add a new docs page/guide that shows how you can enable tracing in istio, authorino and limitador to a central collector, and an example troubleshooting flow.

Depends on changes and information captured in:

• Distributed tracing on data plane architecture#48 (not all tasks are required in this issue linked to progress the guide)
• Add support to push traces to collector limitador#86
• Add limitador tracing-endpoint command limitador-operator#130
• Include the traceID (& parent trace id if available) in logs kuadrant-operator#438

Related to Kuadrant/kuadrant-operator#399, but doesn't depend on it.
The guide can be written with the current limitations called out.

Rough layout of the guide:

• show how to enable tracing & set a collector in istio via Telemetry CR
• same for the Authorino and Limitador CRs
• call out the caveat about trace ids not propegating to wasm modules and the issue Look into what it would take to start trace, and propagate to our services, from Envoy/Istio/OSSM/Sail kuadrant-operator#399 (more info on the caveat below)
• would be good to show a screenshot of a trace from the grafana or jaeger UI
• give an example troubleshooting flow where you move between traces and logs i.e. seeing the traceid in logs (of limitador at least. see outcome of Include the traceID (& parent trace id if available) in logs kuadrant-operator#438 if other logs are relevant to include)
• call out that the tracing support in kuadrant should be otel compatible

The work in https://gist.github.com/adam-cattermole/0f5097e3814f2fcf4d9eed85c6d897a3 and Kuadrant/kuadrant-operator#521 can be taken as needed for the guide. There's no need to include the complete setup of the various kuadrant and gateway api CRs though, or the setup of the tracing stack. The focus can be on the trace collector endpoint configuration in the gateway and kuadrant components, and making use of traces.

Caveat:

• The trace starting point must be outside of envoy/istio/ossm (before it hits the gateway) for tracing support in limitador to take effect.
• There are future plans to remove this limitation by implementing a feature in Envoy. If/When that lands, it will be picked up by OSSM at some future date, after which we update our documentation and examples accordingly.

Future work (no need to call this out in guide):

• There is future work to abstract the tracing endpoint configuration to a single configuration point in the Kuadrant CR. There are also discussions about how this could be automated in the context of OpenShift so that if you are using OSSM with OpenShift Distributed Tracing, you get tracing support in Kuadrant components turned out by default (or at the very least, easily enabled), sending traces to the same collector as OSSM.