Kotlin gRPC conflict with OpenTelemetry Kotlin/gRPC instrumentation

[devkanro]

Describe the bug
Our server combines Kotlin Coroutine, gRPC, and OpenTelemetry, in this situation can result in unexpected behavior when using withContext to switch the gRPC context.

Check out the image below:

• GrpcContextElement is a ThreadContextElement from grpc-kotlin, it store current grpc context instance and will recover it in coroutines.
• KotlinContextElement is a ThreadContextElement from optentelemetry, it store the otlp context snapshot and will recover it in coroutines.
• ContextStorageBridge is the override Context.Storage imple from optentelemetry, it will replace the default grpc storage and store grpc context in otlp context.

When I want assign a new key value pair to gRPC context in kotlin coroutine, I will do this:

withContext(GrpcContextElement(Context.withValue(MY_KEY, MY_VALUE))) {
    // do my job
}

It works without tracing, then something strange happens with OpenTelemetry on.

// OT will inject KotlinContextElement to current coroutine context, it stored otpl context snapshot.
// And ContextStorageBridge will override the grpc context storage and read grpc context from otpl context.
// At this moment, we assume that the grpc context state is 'A', and the otplContext state is 'a'.
// 
// A means gRPC context state
// a = (A, <other context>) means the A gRPC context stored in a otpl context with other context value
// KotlinContextElement(a) means the KotlinContextElement stored the a otpl context
// GrpcContextElement(A) means the GrpcContextElement stored the A gRPC context


// First, we append the new key value pair to current grpc context(A), make it changed to B.
// Then we create a GrpcContextElement store the B grpc context.
withContext(GrpcContextElement(Context.current().withValue(MY_KEY, MY_VALUE))) {

    // The GrpcContextElement(B) is the rightmost element in coroutine context chain, it will update the thread state.
    // B grpc context will be attached in current thread, it will be handle by ContextStorageBridge and create a new otpl context named b, b = (B, <other context>).
    // When we get values in grpc context it will get it from B grpc context which in b otpl context.
    // It works fine.

    // KotlinContextElement(a) will be handled next, it will recover the otpl context a to current thread.
    // This is where the problem arises, the a context will be made current.
    // When we get values in grpc context it will get it from A grpc context which in a otpl context.

    // do my buggy job here, the grpc context is A, the otpl context is a.
}

What version are you using?
1.23.0

[trask]

hi @devkanro! it would probably be most helpful if you could provide a small standalone repro that could be used to better understand the issue

[devkanro]

@trask Please check the devkanro/openTelemetry-issue-7837