Skip to content

internal/telemetry: fix undersized span queue causing dropped spans#33927

Merged
fjl merged 1 commit into
ethereum:masterfrom
jrhea:otel-span-queue-fix
Mar 4, 2026
Merged

internal/telemetry: fix undersized span queue causing dropped spans#33927
fjl merged 1 commit into
ethereum:masterfrom
jrhea:otel-span-queue-fix

Conversation

@jrhea
Copy link
Copy Markdown
Contributor

@jrhea jrhea commented Mar 2, 2026

The BatchSpanProcessor queue size was incorrectly set to
DefaultMaxExportBatchSize (512) instead of DefaultMaxQueueSize (2048).

I noticed the issue on bloatnet when analyzing the block building traces. During a particular run, the miner was including 1000 transactions in a single block. When telemetry is enabled, the miner creates a span for each transaction added to the block. With the queue capped at 512, spans were silently dropped when production outpaced the span export, resulting in incomplete traces with orphaned spans. While this doesn't eliminate the possibility of drops under extreme
load, using the correct default restores the 4x buffer between queue
capacity and export batch size that the SDK was designed around.

@jrhea jrhea requested review from fjl and lightclient March 3, 2026 19:01
@jrhea jrhea added this to the 1.17.2 milestone Mar 3, 2026
@fjl fjl merged commit 402c71f into ethereum:master Mar 4, 2026
6 of 8 checks passed
@jrhea jrhea deleted the otel-span-queue-fix branch March 4, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants