Skip to content

TRACING-5814 | RHOSDT 3.9 Document google managed prometheus #104286

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
pavolloffay wants to merge 7 commits into from
Closed

TRACING-5814 | RHOSDT 3.9 Document google managed prometheus #104286

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
35a1024
TRACING-5814 | RHOSDT 3.9 Document google managed prometheus
pavolloffay Dec 23, 2025
4f37f23
TRACING-5814 | RHOSDT 3.9 Document google managed prometheus
pavolloffay Dec 23, 2025
a08a279
Google link
pavolloffay Dec 23, 2025
da06f79
Add google auth extension
pavolloffay Jan 7, 2026
c20634e
Use WIF
pavolloffay Jan 8, 2026
5c73d11
Use WIF
pavolloffay Jan 8, 2026
725d3af
Use WIF
pavolloffay Jan 8, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 38 additions & 0 deletions modules/otel-extensions-googleauth-extension.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
// Module included in the following assemblies:
//
// * observability/otel/otel-collector/otel-collector-extensions.adoc

:_mod-docs-content-type: REFERENCE
[id="otel-extensions-googleauth-extension_{context}"]
= Google Client Authentication Extension

[role="_abstract"]

The Google Client Authentication extension provides Google OAuth2 Client Credentials and Metadata for gRPC and http based exporters.

.OpenTelemetry Collector custom resource with the configured Google Client Auth Extension
[source,yaml]
----
# ...
config:
extensions:
googleclientauth:
project: "my-project" # <1>

exporters:
otlphttp:
encoding: json
endpoint: https://telemetry.googleapis.com
auth:
authenticator: googleclientauth

service:
extensions: [googleclientauth]
pipelines:
traces:
receivers: [otlp]
exporters: [debug]
# ...
----

<1> The Google Cloud Project telemetry is sent to if the `gcp.project.id` resource attribute is not set. If unspecified, this is determined using application default credentials.
Comment thread
max-cx marked this conversation as resolved.
66 changes: 66 additions & 0 deletions modules/otel-forwarding-data-to-google-managed-prometheus.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// Module included in the following assemblies:
//
// * observability/otel/otel-forwarding-telemetry-data.adoc

:_mod-docs-content-type: CONCEPT
[id="otel-forwarding-data-to-google-managed-prometheus_{context}"]
= Forwarding telemetry data to Google Managed Prometheus

[role="_abstract"]
To forward metrics to the Google Managed Prometheus, use the OpenTelemetry Collector with the OTLP exporter, metricstarttime processor and googleclientauth extension.

.OpenTelemetry Collector custom resource with a configured File Storage Extension that persists an OTLP sending queue
[source,yaml]
----
# ...
mode: sidecar
config:
extensions:
googleclientauth: {}

@pavolloffay pavolloffay Jan 7, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@IshwarKanse the enxtension has a project parameter

project - The Google Cloud Project telemetry is sent to if the gcp.project.id resource attribute is not set. If unspecified, this is determined using application default credentials.

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/extension/googleclientauthextension

If unspecified, this is determined using application default credentials.

what do they mean by this? In OCP context, can it be only specified in GOOGLE_APPLICATION_CREDENTIALS?

@IshwarKanse IshwarKanse Jan 8, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Application Default Credentials (ADC) is Google's credential discovery mechanism that searches for credentials in this order:

  1. GOOGLE_APPLICATION_CREDENTIALS environment variable (pointing to a credential JSON file)
  2. Local ADC file at ~/.config/gcloud/application_default_credentials.json (created by gcloud auth application-default login)
  3. GCP metadata server (for workloads running on Google Cloud infrastructure)

In the OpenShift/Kubernetes context, GOOGLE_APPLICATION_CREDENTIALS is the standard approach, even when running on Google Cloud, because:

  • It provides explicit control over which credentials the collector uses
  • It's portable across different environments (on-prem, AWS, Azure, GCP, etc.)
  • The local ADC file option (Added client tools install topics #2) is meant for local development, not production containers
  • While the metadata server (Initial build system #3) would work for OpenShift clusters running on GCP, explicitly mounting credentials is the Kubernetes best practice

The GOOGLE_APPLICATION_CREDENTIALS env var should point to one of these credential types:

  1. Service Account Key JSON:
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: "/etc/google-cloud-sa/sa-key.json"
  2. Workload Identity Federation config JSON:
    env:
    - name: GOOGLE_APPLICATION_CREDENTIALS
    value: "/etc/workload-identity/credential-configuration.json"

About the project parameter

When we set project in the extension config, it's actually optional because:

  • ADC will automatically extract the project ID from the credential file
  • It only serves as a fallback when the gcp.project.id resource attribute isn't set
  • In my referenced config in the previous comment, we are already setting gcp.project_id via the resource processor, so the extension's project parameter is somewhat redundant

For the documentation, I'd suggest making the project parameter optional and explaining that users can omit it if they're already setting the resource attribute or if they want ADC to auto-detect it from the credentials.


exporters:
otlphttp:
encoding: json
endpoint: https://telemetry.googleapis.com
auth:
authenticator: googleclientauth

processors:
metricstarttime:
strategy: subtract_initial_point # <1>

resource/gcp_project_id:
attributes:
- action: insert
value: project_id # <2>
key: gcp.project_id

k8sattributes: {}

transform/collision:
metric_statements:
- context: datapoint
statements:
- set(attributes["exported_location"], attributes["location"])
- delete_key(attributes, "location")
- set(attributes["exported_cluster"], attributes["cluster"])
- delete_key(attributes, "cluster")
- set(attributes["exported_namespace"], attributes["namespace"])
- delete_key(attributes, "namespace")
- set(attributes["exported_job"], attributes["job"])
- delete_key(attributes, "job")
- set(attributes["exported_instance"], attributes["instance"])
- delete_key(attributes, "instance")
- set(attributes["exported_project_id"], attributes["project_id"])
- delete_key(attributes, "project_id")

service:
extensions: [googleclientauth]
pipelines:
metrics:
processors: [k8sattributes, resource/gcp_project_id, transform/collision, metricstarttime]
exporters: [otlphttp]
# ...
----
<1> The `subtract_initial_point` strategy is stateful, requiring the Collector to run as a sidecar to maintain per-pod state. Alternative strategies available; choose the one that best fits your use case.
Comment thread
max-cx marked this conversation as resolved.
Outdated
<2> Replace with your GCP project ID.
2 changes: 2 additions & 0 deletions observability/otel/otel-collector/otel-collector-extensions.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ include::modules/otel-extensions-jaegerremotesampling-extension.adoc[leveloffset

include::modules/otel-extensions-pprof-extension.adoc[leveloffset=+1]

include::modules/otel-extensions-googleauth-extension.adoc[leveloffset=+1]

include::modules/otel-extensions-healthcheck-extension.adoc[leveloffset=+1]

include::modules/otel-extensions-zpages-extension.adoc[leveloffset=+1]
Expand Down
8 changes: 8 additions & 0 deletions observability/otel/otel-forwarding-telemetry-data.adoc
View file
Open in desktop
Comment thread
max-cx marked this conversation as resolved.
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,14 @@ include::modules/otel-forwarding-data-to-google-cloud.adoc[leveloffset=+1]
.Additional resources
* xref:../../observability/otel/otel-collector/otel-collector-exporters.adoc#otel-collector-exporters[Exporters]

include::modules/otel-forwarding-data-to-google-managed-prometheus.adoc[leveloffset=+1]

[role="_additional-resources"]
.Additional resources
* xref:../../observability/otel/otel-collector/otel-collector-processors.adoc#otel-collector-processors[Processors]
* xref:../../observability/otel/otel-collector/otel-collector-extensions.adoc#otel-collector-extensions[Extensions]
* https://docs.cloud.google.com/stackdriver/docs/managed-prometheus

[role="_additional-resources"]
[id="additional-resources_{context}"]
== Additional resources
Expand Down