[loadgen] - add baggage

[puckpuck]

Fixes #329

Changes

• fixes some routes to be the API routes instead of the Browser routes for the frontend
• Adds a synthetic_request=true baggage to all requests

• Appropriate CHANGELOG.md updated for non-trivial changes

[cartersocha]

I pulled the branch and did a fresh docker build. I'm not seeing any baggage attribute being created or passed down from the load gen

[puckpuck]

I pulled the branch and did a fresh docker build. I'm not seeing any baggage attribute being created or passed down from the load gen

baggage doesn't get exported. The only way to see it is to capture the HTTP headers and sniff out the baggage header. To test this, I put a proxy server in place and had the loadgen go through the proxy server. Then within my proxy server I was able to sniff out the baggage header on requests to the frontend.

[cartersocha]

I pulled the branch and did a fresh docker build. I'm not seeing any baggage attribute being created or passed down from the load gen

baggage doesn't get exported. The only way to see it is to capture the HTTP headers and sniff out the baggage header. To test this, I put a proxy server in place and had the loadgen go through the proxy server. Then within my proxy server I was able to sniff out the baggage header on requests to the frontend.

How are we expecting baggage to be consumed then if it’s not appearing in jaeger?

Ideally in grafana & jaeger we have filters on is synthetic

[cijothomas]

I pulled the branch and did a fresh docker build. I'm not seeing any baggage attribute being created or passed down from the load gen

baggage doesn't get exported. The only way to see it is to capture the HTTP headers and sniff out the baggage header. To test this, I put a proxy server in place and had the loadgen go through the proxy server. Then within my proxy server I was able to sniff out the baggage header on requests to the frontend.

How are we expecting baggage to be consumed then if it’s not appearing in jaeger?

Ideally in grafana & jaeger we have filters on is synthetic

I think one must write a SpanProcessor to read Baggage and add it as Span Attributes. Baggage is not automatically attached to Spans or Logs or Metrics by default.

(some old discussions from spec open-telemetry/opentelemetry-specification#867)

[puckpuck]

How are we expecting baggage to be consumed then if it’s not appearing in jaeger?

As was mentioned baggage isn't meant to be exported by OTLP. Some have written SpanProcessors that will take items out of baggage and add them as span attributes before getting exported. Here is an example of one in Go: https://github.com/honeycombio/honeycomb-opentelemetry-go/blob/main/baggage_span_processor.go

We still want people to know baggage was used to drive something about the webstore telemetry. So #332 will add a span attribute app.synthetic_request when it sees the baggage item appear. Also #335 added logic where the app.payment.charged attribute on a span event will be true/false based on that baggage value.

We should most certainly document this stuff, and how baggage is used in downstream services.

[julianocosta89]

How are we expecting baggage to be consumed then if it’s not appearing in jaeger?

If I got that right, in a real life scenario we could decide to create or not the spans based on the baggage.
Would that be a good use case for it @puckpuck ?

[austinlparker]

How are we expecting baggage to be consumed then if it’s not appearing in jaeger?

If I got that right, in a real life scenario we could decide to create or not the spans based on the baggage. Would that be a good use case for it @puckpuck ?

My 0.02, you wouldn't necessarily want to create spans or not based on baggage (although you could), but it would be useful to insert into the telemetry as an attribute. Let's say you have an SLO (or any other kind of alert) on some API route, perhaps you'd want to filter out synthetic requests from impacting that SLO since it doesn't affect "real" users.