Propagation fails because trace ID value does not observe Cloud Logging format.

[afeldkamp]

The formatter in this sample correctly renames the OpenTelemetry field names to their GCP counterparts, however this does not address the format difference in the trace ID value. Cloud Logging is expecting projects/{project-id}/traces/{trace-id}, but we are only providing the raw trace ID here.

We can propagate existing Cloud Logging trace ID's by including something like this:

from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.cloud_trace_propagator import CloudTraceFormatPropagator

LoggingInstrumentor().instrument()
set_global_textmap(CloudTraceFormatPropagator())

This will attach the propagated trace ID to our logging calls correctly, however with the JsonFormatter in this sample alone the logs would not correlate properly in GCP Logs Explorer or Trace Explorer since the projects/{project-id}/traces/ prefix is still missing from the trace ID value. We can use the log_hook argument on LoggingInstrumentor to modify the field value like this:

def _log_hook(span: Span, record: logging.LogRecord) -> None:
    if span and span.is_recording():
        if hasattr(record, "otelTraceID"):
            record.otelTraceID = (
                f"projects/{PROJECT_ID}/traces/{record.otelTraceID}"
            )
LoggingInstrumentor().instrument(log_hook=_log_hook)

(Let's pretend we queried metadata.google.internal or something and already set PROJECT_ID somewhere for the sake of this example.) With this, the otelTraceID value will first be changed to the correct format by the LoggingInstrumentor hook, then the field is renamed to logging.googleapis.com/trace by the JsonFormatter in the sample, and now traces will correlate properly in Logs Explorer/Trace Explorer! Hope this helps.

[aabmass]

@afeldkamp please take a look at https://cloud.google.com/trace/docs/setup/python-ot#config-structured-logging. This sample shows how to configure a Python app to do structured logging with proper OpenTelemetry trace-log correlation

Sorry just took a closer read of your comment.

Cloud Logging is expecting projects/{project-id}/traces/{trace-id}, but we are only providing the raw trace ID here.

I think the docs are a bit outdated here. Afaik, the agents have been updated to recognize the raw trace ID directly. I just tested it in GKE and everything works correctly because the logging agent adds the projects/.../traces/ for you.

kubectl run -it busybox --image=busybox --restart=Never -- echo '{"severity":"INFO","logging.googleapis.com/trace":"3a7b0c3dbf85c424a1fc184831dc565f","logging.googleapis.com/spanId":"562cfa35c0281c38","logging.googleapis.com/trace_sampled":true,"message":"Testing 123"}'