Skip to content

gRPC: initial stable semconv support and testing#16304

Open
trask wants to merge 10 commits intoopen-telemetry:mainfrom
trask:grpc-semconv
Open

gRPC: initial stable semconv support and testing#16304
trask wants to merge 10 commits intoopen-telemetry:mainfrom
trask:grpc-semconv

Conversation

@trask
Copy link
Member

@trask trask commented Feb 27, 2026

No description provided.

@trask trask force-pushed the grpc-semconv branch 4 times, most recently from 2c1a363 to 0f28710 Compare February 27, 2026 03:15
@trask trask force-pushed the grpc-semconv branch 3 times, most recently from 206f9e9 to e4b6d0d Compare February 27, 2026 03:37
@trask trask marked this pull request as ready for review February 27, 2026 05:27
@trask trask requested a review from a team as a code owner February 27, 2026 05:27
Copy link
Member

@zeitlinger zeitlinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR!

One issue I noticed: the stable gRPC semconv spec specifies that rpc.response.status_code should use the named status code (OK, CANCELLED, UNAVAILABLE, etc.), not the numeric value (0, 2, 14).

Suggest changing:

attributes.put(RPC_RESPONSE_STATUS_CODE, String.valueOf(status.getCode().value()));

to:

attributes.put(RPC_RESPONSE_STATUS_CODE, status.getCode().name());

(and updating the test assertions accordingly)

Copy link
Member

@zeitlinger zeitlinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In rpc/dup mode, span attributes can only hold one value for rpc.method. Since stable semconv wins (line 42-53 of RpcCommonAttributesExtractor), old metrics would silently get the stable method value (example.Greeter/SayHello) instead of the old one (SayHello).

The dualEmitContextCustomizer from #16298 was designed to fix this — it stashes the old rpc.method in context so RpcClientMetrics/RpcServerMetrics can swap it back for old metrics. But it's not wired into GrpcTelemetryBuilder.

Suggest adding to GrpcTelemetryBuilder.build(), after addOperationMetrics for each instrumenter:

clientInstrumenterBuilder
    .addOperationMetrics(RpcClientMetrics.get())
    .addContextCustomizer(
        RpcMetricsContextCustomizers.dualEmitContextCustomizer(rpcAttributesGetter));
serverInstrumenterBuilder
    .addOperationMetrics(RpcServerMetrics.get())
    .addContextCustomizer(
        RpcMetricsContextCustomizers.dualEmitContextCustomizer(rpcAttributesGetter));

(The customizer is a no-op when not in dup mode, so it's safe to always register.)

Copy link
Member

@zeitlinger zeitlinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Follow-up on the dualEmitContextCustomizer comment: the reason this wasn't caught is that there's no rpc/dup integration test for gRPC. The testBothSemconv task in instrumentation-api-incubator exercises RpcClientMetrics/RpcServerMetrics in isolation (manually calling the context customizer), but nothing tests the full gRPC pipeline in dup mode.

Adding a testBothSemconv task to instrumentation/grpc-1.6/javaagent/build.gradle.kts with -Dotel.semconv-stability.opt-in=rpc/dup would catch this: the test assertions would see old metrics with rpc.method=example.Greeter/SayHello instead of rpc.method=SayHello and fail, revealing the missing context customizer wiring.

Copy link
Member

@zeitlinger zeitlinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two more things I noticed by comparing with #15879:

1. Captured request metadata uses old attribute key prefix in stable mode

GrpcAttributesExtractor.onEnd() always stores captured request metadata under rpc.grpc.request.metadata.<key>, even when stable semconv is active. The stable semconv spec defines the attribute as rpc.request.metadata.<key> (no grpc. infix). The big PR handles this via CapturedGrpcMetadataUtil.stableRequestAttributeKey() and branching on semconv mode — that's missing here.

2. Missing testStableSemconv task for grpc-1.6/library

The testStableSemconv Gradle task is only added to grpc-1.6/javaagent/build.gradle.kts, but the big PR also adds one to grpc-1.6/library/build.gradle.kts. The library tests exercise GrpcTelemetryBuilder.build() directly, which is the code path where things like the dualEmitContextCustomizer wiring would be caught.

@zeitlinger zeitlinger mentioned this pull request Feb 27, 2026
@zeitlinger
Copy link
Member

This PR is "PR 2" (rpc-grpc) from the split plan in #15879. PR 1 (rpc-api-incubator) was merged as #16298.

Copy link
Member

@zeitlinger zeitlinger left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done reviewing. Summary of findings:

  1. rpc.response.status_code should use named status codes (status.getCode().name()) not numeric (String.valueOf(status.getCode().value())) per the stable gRPC semconv spec
  2. dualEmitContextCustomizer from #16298 is not wired into GrpcTelemetryBuilder, so rpc/dup mode would produce wrong rpc.method values on old metrics
  3. A testBothSemconv Gradle task with rpc/dup would have caught the above
  4. Captured request metadata always uses old rpc.grpc.request.metadata.* prefix — stable semconv requires rpc.request.metadata.* (missing CapturedGrpcMetadataUtil.stableRequestAttributeKey())
  5. Missing testStableSemconv task for grpc-1.6/library module (only javaagent has one)

Compared against the original big PR #15879 to identify gaps. Items 1 and 2 are also present in the big PR, so they're pre-existing rather than regressions from the split.

trask and others added 9 commits February 27, 2026 11:08
Per the stable gRPC semconv spec, rpc.response.status_code should use
the named status code (OK, CANCELLED, UNAVAILABLE, etc.) not the
numeric value (0, 2, 14).

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix missed status code assertions in streaming and clientErrorThrown tests

Use Status.Code.name() instead of String.valueOf(Status.Code.*.value()) for
rpc.response.status_code assertions in AbstractGrpcStreamingTest and the
clientErrorThrown test in AbstractGrpcTest.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
In rpc/dup mode, span attributes can only hold one value for rpc.method.
The dualEmitContextCustomizer stashes the old rpc.method value in
context so RpcClientMetrics/RpcServerMetrics can use it for old metrics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Suppress deprecation warning for RpcMetricsContextCustomizers in build()

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Tests the full gRPC pipeline in rpc/dup mode to catch issues like
missing context customizer wiring that would produce wrong rpc.method
values on old metrics.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The stable semconv spec uses rpc.request.metadata.<key> instead of
rpc.grpc.request.metadata.<key>. Add stableRequestAttributeKey() to
CapturedGrpcMetadataUtil and use it when stable semconv is active.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The library tests exercise GrpcTelemetryBuilder.build() directly,
which is the code path where issues like missing context customizer
wiring would be caught.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Fix RPC_SYSTEM assertions for rpc/dup semconv mode in tests

Replace equalTo(maybeStable(RPC_SYSTEM), "grpc") with dual RPC_SYSTEM and
RPC_SYSTEM_NAME assertions in AbstractGrpcTest and AbstractGrpcStreamingTest,
to correctly handle rpc/dup mode where both attributes are emitted.
Also removes the now-unused maybeStable import from both test files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
- ArmeriaGrpcTest: use Status.Code.OK.name() instead of
  String.valueOf(Status.Code.OK.value()) for rpc.response.status_code,
  matching the GrpcAttributesExtractor implementation
- grpc-1.6 build.gradle.kts: add testLatestDeps system property and
  IPv4 preference to testStableSemconv and testBothSemconv tasks,
  matching the existing test task configuration

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants