stats/opentelemetry: record retry attempts from clientStream by vinothkumarr227 · Pull Request #8342 · grpc/grpc-go

vinothkumarr227 · 2025-05-19T09:14:38Z

RELEASE NOTES:

stats/opentelemetry: Retry attempts (grpc.previous-rpc-attempts) are now recorded as span attributes for non-transparent client retries.

codecov · 2025-05-19T09:19:34Z

Codecov Report

❌ Patch coverage is 52.63158% with 9 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.50%. Comparing base (e60a04b) to head (c0e523c).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
stats/opentelemetry/client_tracing.go	52.63%	6 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8342      +/-   ##
==========================================
- Coverage   81.64%   81.50%   -0.15%     
==========================================
  Files         413      413              
  Lines       40621    40693      +72     
==========================================
- Hits        33167    33166       -1     
- Misses       5991     6000       +9     
- Partials     1463     1527      +64

Files with missing lines	Coverage Δ
stats/opentelemetry/opentelemetry.go	`50.94% <ø> (-26.97%)`	⬇️
stats/opentelemetry/trace.go	`41.07% <ø> (-47.82%)`	⬇️
stats/opentelemetry/client_tracing.go	`55.31% <52.63%> (-32.19%)`	⬇️

... and 24 files with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

stream.go

stats/opentelemetry/trace.go

stats/opentelemetry/client_tracing.go

purnesh42H

I remember we had separate tests for retries. This change should only affect that test. This change shouldn't affect tests which are doing only single attempt.

stats/opentelemetry/client_tracing.go

stats/opentelemetry/trace.go

stats/opentelemetry/e2e_test.go

purnesh42H

@vinothkumarr227 have you tested this to ensure its working correctly? I was under impression that TestTraceSpan_WithRetriesAndNameResolutionDelay will need changes for expected values. I think its not working as expected because you are not setting the count back to ctx after incrementing.

stats/opentelemetry/client_tracing.go

stats/opentelemetry/opentelemetry.go

stats/opentelemetry/trace.go

stats/opentelemetry/opentelemetry.go

stats/opentelemetry/e2e_test.go

stats/opentelemetry/client_metrics.go

stats/opentelemetry/e2e_test.go

stats/opentelemetry/opentelemetry.go

stats/opentelemetry/trace.go

stats/opentelemetry/client_tracing.go

stats/opentelemetry/e2e_test.go

dfawley

Sorry for the delays in review here. LGTM after this one small change.

dfawley · 2025-09-05T20:50:45Z

stats/opentelemetry/client_metrics.go

-			method: determineMethod(method, opts...),
+			target:              cc.CanonicalTarget(),
+			method:              determineMethod(method, opts...),
+			previousRPCAttempts: new(atomic.Uint32),


Let's make this a non-pointer type and then we don't need the new here or the chance to get nil panics.

…s-fix-8299

dfawley · 2025-09-09T20:04:08Z

stats/opentelemetry/client_metrics.go

 			target:              cc.CanonicalTarget(),
 			method:              determineMethod(method, opts...),
-			previousRPCAttempts: new(atomic.Uint32),
+			previousRPCAttempts: atomic.Uint32{},


Please delete this line. This is the zero value so it doesn't need explicit initialization.

…rpc#8342)" This reverts commit c122250.

…8342)" (#8571) This introduced flakiness in a test - Test/TraceSpan_WithRetriesAndNameResolutionDelay Failure: https://github.com/grpc/grpc-go/actions/runs/17614152882/job/50042942932?pr=8547 Related issue: #8299 RELEASE NOTES: None

…tionDelay This commit fixes the flaky test TestTraceSpan_WithRetriesAndNameResolutionDelay which was introduced in PR grpc#8342 and caused that PR to be reverted. Root Cause: The test had race conditions related to timing: 1. The goroutine that updates resolver state could complete before or after the delayed resolution event was fully processed and recorded in spans 2. Span export timing was not synchronized with test validation, causing the test to sometimes check spans before they were fully exported Fix: 1. Added 'stateUpdated' event to synchronize between the resolver state update completing and span validation beginning 2. Added explicit wait for the stateUpdated event before validating spans 3. Added a 50ms sleep after RPC completion to give the span exporter time to process and export all spans before validation Testing: - Test now passes consistently (10+ consecutive runs) - Passes with race detector enabled (-race flag) - No data races detected Fixes grpc#8700

Fixes: grpc#8299 RELEASE NOTES: - stats/opentelemetry: Retry attempts (`grpc.previous-rpc-attempts`) are now recorded as span attributes for non-transparent client retries.

…tionDelay This commit fixes the flaky test TestTraceSpan_WithRetriesAndNameResolutionDelay which was introduced in the previous commit and caused PR grpc#8342 to be reverted. Root Cause: The test had race conditions related to timing: 1. The goroutine that updates resolver state could complete before or after the delayed resolution event was fully processed and recorded in spans 2. Span export timing was not synchronized with test validation, causing the test to sometimes check spans before they were fully exported Fix: 1. Added 'stateUpdated' event to synchronize between the resolver state update completing and span validation beginning 2. Added explicit wait for the stateUpdated event before validating spans 3. Added a 50ms sleep after RPC completion to give the span exporter time to process and export all spans before validation Testing: - Test now passes consistently (10+ consecutive runs) - Passes with race detector enabled (-race flag) - No data races detected Fixes grpc#8700

Fixes: grpc#8299 RELEASE NOTES: - stats/opentelemetry: Retry attempts (`grpc.previous-rpc-attempts`) are now recorded as span attributes for non-transparent client retries.

…tionDelay This commit fixes the flaky test TestTraceSpan_WithRetriesAndNameResolutionDelay which was introduced in the previous commit and caused PR grpc#8342 to be reverted. Root Cause: The test had race conditions related to timing: 1. The goroutine that updates resolver state could complete before or after the delayed resolution event was fully processed and recorded in spans 2. Span export timing was not synchronized with test validation, causing the test to sometimes check spans before they were fully exported Fix: 1. Added 'stateUpdated' event to synchronize between the resolver state update completing and span validation beginning 2. Added explicit wait for the stateUpdated event before validating spans 3. Added a 50ms sleep after RPC completion to give the span exporter time to process and export all spans before validation Testing: - Test now passes consistently (10+ consecutive runs) - Passes with race detector enabled (-race flag) - No data races detected Fixes grpc#8700

Fixed retry attempts in HandleRPC

3ba457e

purnesh42H reviewed May 19, 2025

View reviewed changes

stream.go Outdated Show resolved Hide resolved

purnesh42H assigned vinothkumarr227 May 19, 2025

purnesh42H added Type: Bug Area: Observability Includes Stats, Tracing, Channelz, Healthz, Binlog, Reflection, Admin, GCP Observability labels May 19, 2025

purnesh42H added this to the 1.73 Release milestone May 19, 2025

vinothkumarr227 added 2 commits May 20, 2025 06:55

Fixed the review changes

5d19779

Fixed vet issues

4245950

vinothkumarr227 requested a review from purnesh42H May 20, 2025 07:09

eshitachandwani assigned purnesh42H and unassigned vinothkumarr227 May 20, 2025

purnesh42H reviewed May 20, 2025

View reviewed changes

stats/opentelemetry/trace.go Outdated Show resolved Hide resolved

stats/opentelemetry/client_tracing.go Outdated Show resolved Hide resolved

stats/opentelemetry/client_tracing.go Outdated Show resolved Hide resolved

purnesh42H assigned vinothkumarr227 and unassigned purnesh42H May 20, 2025

Fixed the review changes

5347db1

vinothkumarr227 requested a review from purnesh42H May 21, 2025 06:08

purnesh42H reviewed May 21, 2025

View reviewed changes

stats/opentelemetry/client_tracing.go Outdated Show resolved Hide resolved

stats/opentelemetry/trace.go Outdated Show resolved Hide resolved

stats/opentelemetry/trace.go Outdated Show resolved Hide resolved

stats/opentelemetry/e2e_test.go Outdated Show resolved Hide resolved

Fixed the review changes

99e88d8

vinothkumarr227 requested a review from purnesh42H May 22, 2025 07:51

purnesh42H reviewed May 26, 2025

View reviewed changes

vinothkumarr227 added 2 commits May 26, 2025 12:58

Fixed the review changes

586cf63

Fixed the test cases

1bdad7e

eshitachandwani assigned purnesh42H and unassigned vinothkumarr227 May 27, 2025

eshitachandwani requested a review from purnesh42H May 27, 2025 09:37

purnesh42H reviewed May 28, 2025

View reviewed changes

purnesh42H assigned vinothkumarr227 and unassigned purnesh42H May 28, 2025

Fixed the review changes

39c5f0d

dfawley assigned vinothkumarr227 and unassigned dfawley Aug 12, 2025

Fixed the review changes

1660fcb

vinothkumarr227 requested a review from dfawley August 13, 2025 11:01

vinothkumarr227 removed their assignment Aug 13, 2025

Fixed the review changes

992343e

eshitachandwani assigned dfawley Aug 14, 2025

dfawley approved these changes Sep 5, 2025

View reviewed changes

dfawley assigned vinothkumarr227 and unassigned dfawley Sep 5, 2025

vinothkumarr227 added 3 commits September 8, 2025 04:45

small tweaks

25f2e3d

Merge remote-tracking branch 'origin/master' into stats-retry-attempt…

e225af4

…s-fix-8299

Merge remote-tracking branch 'origin/master' into stats-retry-attempt…

f32e117

…s-fix-8299

vinothkumarr227 removed their assignment Sep 9, 2025

arjan-bal assigned dfawley Sep 9, 2025

dfawley approved these changes Sep 9, 2025

View reviewed changes

dfawley assigned vinothkumarr227 and unassigned dfawley Sep 9, 2025

small tweaks

c0e523c

vinothkumarr227 removed their assignment Sep 10, 2025

eshitachandwani merged commit c122250 into grpc:master Sep 10, 2025
15 checks passed

eshitachandwani added a commit to eshitachandwani/grpc-go that referenced this pull request Sep 10, 2025

Revert "stats/opentelemetry: record retry attempts from clientStream (g…

66b7615

…rpc#8342)" This reverts commit c122250.

easwars mentioned this pull request Nov 11, 2025

otel: retries must be tracked per-call and not per-attempt #8700

Open

eshitachandwani mentioned this pull request Nov 20, 2025

stats/opentelemetry: Track retry counts per-call instead of per-attempt #8715

Open

Conversation

vinothkumarr227 commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented May 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purnesh42H left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

purnesh42H left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dfawley left a comment

Choose a reason for hiding this comment

Uh oh!

dfawley Sep 5, 2025

Choose a reason for hiding this comment

Uh oh!

vinothkumarr227 Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

dfawley Sep 9, 2025

Choose a reason for hiding this comment

Uh oh!

vinothkumarr227 Sep 10, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

vinothkumarr227 commented May 19, 2025 •

edited

Loading

codecov bot commented May 19, 2025 •

edited

Loading