OpenTelemetry should convert level to string

[leahneukirchen]

I'm running Telemere 1.0.0-beta14 with io.opentelemetry/opentelemetry-exporter-otlp 1.39.0, sending to Alloy 1.1.1 to Loki 3.0.

By default, the message "level" (a symbol in Clojure) ends up as a string like ":error" in Loki, which is recognized as "unknown" level. I think signal->attrs-map should pass on (name level) as "level", so "level" is "error". Then Loki colors the events properly.

[ptaoussanis]

@leahneukirchen Hi Leah, thanks for pinging about this - I'm not familiar with Loki, so it's helpful to hear about things like this 🙏

Telemere's Open Telemetry handler currently does the following:

1. It sets the severity level of the log record based on the level.
2. It otherwise uses str on all keyword log record attribute values.

So in your case 1 should be correct, but I guess that Loki is also checking certain attribute values ("level" for example)?

My initial preference was to prefer ":foo/bar" to "foo/bar" for keyword attribute values since that at least preserves some info about the type, but this'd be easy to change.

Which option makes the most sense to you?:

• A: Just change the "level" attribute value to be "foo/bar" instead of ":foo/bar"
• B: Change all attribute values to be "foo/bar" instead of ":foo/bar" (so this'd also affect the "kind", "id", "uid" keys, etc.).

[leahneukirchen]

Yes, Loki doesn't seem to use the severity level (but I'm converting with Alloy from OTLP to Loki, so it may get lost there. I'm trying direct ingestion next.)

I think B is reasonable actually, as it's unlikely people mix symbols and strings there, and expect it to be different?

[ptaoussanis]

Yes, Loki doesn't seem to use the severity level (but I'm converting with Alloy from OTLP to Loki, so it may get lost there.

👍

I think B is reasonable actually, as it's unlikely people mix symbols and strings there, and expect it to be different?

Just to confirm- where exactly are you seeing symbols? I didn't quite understand this part in your original message. I believe Telemere's log record attribute names should all be strings; are you seeing something different?

[leahneukirchen]

I switched to OTLP now, and while this sets severity_number = 5, Loki doesn't seem to parse this.

I meant values, not names. Currently, kind, data_handler_id, level, detected_level (perhaps added somewhere else) are prefixed by :.

[ptaoussanis]

I meant values, not names. Currently, kind, data_handler_id, level, detected_level (perhaps added somewhere else) are prefixed by :.

Okay, but just to confirm- the values should also be strings (not symbols).

So you're seeing: "level" (string key) mapped to ":error" (string value).
What you want: "level" (string key) mapped to "error" (string value, without the ":").

In case you're not very familiar with Clojure, it has distinct string, symbol, and keyword types - so I'm just confirming that you're not somehow encountering symbols unexpectedly.

[leahneukirchen]

Yes, it's (str symbol), not (name symbol).

[leahneukirchen]

I created confusion by saying symbol (that's what 20 years of Ruby do to you), I meant keywords of course. :)

[ptaoussanis]

No worries, thanks for circling back to clarify 👍

[ptaoussanis]

Fixed on upcoming master