Clarify kubernetes logs attributes names

[sumo-drosiek]

There are some attributes for kubernetes container logs, which can be useful (or not) and doesn't have naming convention so far:

• stream (either stdout or stderr) which can be used for example for keeping only stderr logs
• run_id - I don't know proper name of this attribute. This is run number of container (pods? maybe) (starts from 0 and it's increased with every container restart). This is name of a file log (/var/log/pods/<namespace>_<pod-name>_<pod-uid>/<container-name>/<run-id>.log)

Another question is if container-name is more like k8s.container.name or container.name

semantic convention
k8s convention
container convention

Additional context
#2606

[pmm-sumo]

I wonder if stream should be a member of Log Data Model or a dynamic attribute. What do you think @tigrannajaryan @djaglowski ? In case of the latter, perhaps we might want to have log-specific semantic conventions

As for run_id, I think we should just propose it as part of the specification

[djaglowski]

I slightly prefer not to add it to the data model itself. There are a lot situations where stream isn't applicable. If we did add it to the data model, I think there is a question of what the default value would be. Perhaps stdout is sufficiently neutral, but I can imagine scenarios where this would be confusing. Perhaps a third possible value such as default or unknown would be necessary.

[tigrannajaryan]

Note that names like stream and run_id are not compliant with Otel spec recommendations: https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/attribute-and-label-naming.md

We need to put them in some reasonable namespaces, I don't think they can be top-level fields.

[tigrannajaryan]

Seems like run_id can belong to container.* Otel semantic conventions namespace.

[pmm-sumo]

As discussed during the SIG, we might also want to put other log-related attributes, such as the path. ECS schema has a log namespace which includes log.file.path among other fields.

Perhaps we should follow that approach and define log namespace, which could include log.file.path and log.stream?

[tigrannajaryan]

I can't find the concept of run id in K8s docs. Does anyone have a link?

[sumo-drosiek]

I did some research around it and see some references in code as instance number link or container instance link

Didn't find anything useful in documentation

Based on that, I would go with container.instance or container.instance_number

[sumo-drosiek]

Also, we won't be able to ensure that container_id taken by enrichment is truly the container id related to container instance (as container can be restarted meantime)

[dmitryax]

@sumo-drosiek @tigrannajaryan @pmm-sumo Looks like the run_id part of the logs path represents number of container restarts from k8s API. I would like to suggest to call the attribute container.restart_count.

I want to settle a semantic convention for this attribute, so it can be used along with k8s.pod.uid and container.name to fetch a particular container status in this issue.

Please let me know what do you think

[sumo-drosiek]

@dmitryax for me it sounds good. more intuitive than instance/ instanceNumber/run_id

[dmitryax]

Ok, I submitted a PR to add this as a semantic convention open-telemetry/opentelemetry-specification#1945

[tigrannajaryan]

open-telemetry/opentelemetry-specification#1945 resolved the run_id attribute.

open-telemetry/opentelemetry-specification#1980 is open to clarify container.name and k8s.container.name. Please review/comment.

The stream attribute is not addressed at all yet.

[djaglowski]

This issue proposes a place for stream attribute: open-telemetry/opentelemetry-specification#1772

[dmitryax]

I think we can close this issue since all the attributes now added to the OTel conventions cc @tigrannajaryan @djaglowski

[tigrannajaryan]

Closing. Feel free to reopen if there is anything else to be done or create a new issue.