Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Propagation fails because trace ID value does not observe Cloud Logging format. #335

Open
afeldkamp opened this issue Jun 21, 2024 · 1 comment
Labels
priority: p3 question Further information is requested

Comments

@afeldkamp
Copy link

The formatter in this sample correctly renames the OpenTelemetry field names to their GCP counterparts, however this does not address the format difference in the trace ID value. Cloud Logging is expecting projects/{project-id}/traces/{trace-id}, but we are only providing the raw trace ID here.

We can propagate existing Cloud Logging trace ID's by including something like this:

from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.cloud_trace_propagator import CloudTraceFormatPropagator

LoggingInstrumentor().instrument()
set_global_textmap(CloudTraceFormatPropagator())

This will attach the propagated trace ID to our logging calls correctly, however with the JsonFormatter in this sample alone the logs would not correlate properly in GCP Logs Explorer or Trace Explorer since the projects/{project-id}/traces/ prefix is still missing from the trace ID value. We can use the log_hook argument on LoggingInstrumentor to modify the field value like this:

def _log_hook(span: Span, record: logging.LogRecord) -> None:
    if span and span.is_recording():
        if hasattr(record, "otelTraceID"):
            record.otelTraceID = (
                f"projects/{PROJECT_ID}/traces/{record.otelTraceID}"
            )
LoggingInstrumentor().instrument(log_hook=_log_hook)

(Let's pretend we queried metadata.google.internal or something and already set PROJECT_ID somewhere for the sake of this example.) With this, the otelTraceID value will first be changed to the correct format by the LoggingInstrumentor hook, then the field is renamed to logging.googleapis.com/trace by the JsonFormatter in the sample, and now traces will correlate properly in Logs Explorer/Trace Explorer! Hope this helps.

@aabmass aabmass added question Further information is requested priority: p3 labels Jul 29, 2024
@aabmass
Copy link
Collaborator

aabmass commented Jul 29, 2024

@afeldkamp please take a look at https://cloud.google.com/trace/docs/setup/python-ot#config-structured-logging. This sample shows how to configure a Python app to do structured logging with proper OpenTelemetry trace-log correlation

Sorry just took a closer read of your comment.

Cloud Logging is expecting projects/{project-id}/traces/{trace-id}, but we are only providing the raw trace ID here.

I think the docs are a bit outdated here. Afaik, the agents have been updated to recognize the raw trace ID directly. I just tested it in GKE and everything works correctly because the logging agent adds the projects/.../traces/ for you.

kubectl run -it busybox --image=busybox --restart=Never -- echo '{"severity":"INFO","logging.googleapis.com/trace":"3a7b0c3dbf85c424a1fc184831dc565f","logging.googleapis.com/spanId":"562cfa35c0281c38","logging.googleapis.com/trace_sampled":true,"message":"Testing 123"}'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
priority: p3 question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants