Skip to content

fix(otel): use async reqwest client so OTLP export works in goose serve mode - #10100

Merged
dorien-koelemeijer merged 1 commit into
mainfrom
fix/otlp-serve-mode-batch-panic
Jun 30, 2026
Merged

fix(otel): use async reqwest client so OTLP export works in goose serve mode#10100
dorien-koelemeijer merged 1 commit into
mainfrom
fix/otlp-serve-mode-batch-panic

Conversation

@dorien-koelemeijer

@dorien-koelemeijer dorien-koelemeijer commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

Swaps the opentelemetry-otlp feature from reqwest-blocking-client to the async reqwest-client (one line in Cargo.toml). This fixes OTLP export panicking and silently dropping all telemetry when goosed runs in goose serve (ACP) mode.

The bug

The OTLP exporter wraps each export in a block_on fallback for when there's no ambient Tokio runtime (the BatchProcessor runs in a raw std::thread). The blocking reqwest client creates and drops its own nested runtime per request, and dropping a runtime inside block_on panics:
thread 'OpenTelemetry.Logs.BatchProcessor' panicked: Cannot drop a runtime in a context where blocking is not allowed.

The exporter thread dies ~1s in, then every log is dropped with AfterShutdown warnings. goosed agent is unaffected (exports run in the ambient axum runtime → .await path, no block_on), which is why production telemetry works today. Only goose serve hits the broken path.

The fix

Switch to the async reqwest-client. The block_on wrapper was designed for an async client (per its own comment) - the async client reuses the reactor the wrapper provides, so no nested runtime is created or dropped. otlp.rs unchanged.

Testing

  • cargo test -p goose --features otel --lib otel::otlp → 54 passed; clippy/fmt clean.
  • goose serve export: 0 panics / 0 AfterShutdown (was 1 panic + ~1,580 drops).
  • goosed agent export: unchanged, no regression.

End-to-end via goose-internal (the real goose serve consumer)

goose-internal runs the backend as goose serve and exports security telemetry (prompt/command-injection scan findings) over OTLP - it was getting nothing downstream because of this panic. Verified the fix end-to-end:

  1. Built goose from this branch and pointed goose-internal's dev app at it via GOOSE_BIN=.../target/debug/goose just dev.
  2. Confirmed the spawned process: goose serve mode, OTLP logs-only export enabled, correct endpoint/service name.
  3. Ran a uniquely-tagged tool call in the app (echo GOOSE_INTERNAL_TEST_RUN_100) and kept the app open ~60s so the batch processor flushed during the session.
  4. In the goosed log for that PID: the security scan fired and the exporter stayed healthy - 0 panics, 0 AfterShutdown, no OTLP export errors (vs. an immediate panic + thousands of dropped records on the unfixed binary).
  5. Queried the collector’s backing store for the unique marker → row present. The same query returned nothing before the fix.

Risk

Low — single-line feature swap that restores the async client the wrapper was built for; agent-mode path verified non-regressed.

@shellz-n-stuff shellz-n-stuff left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@michaelneale michaelneale left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

huh neat fix for that.

@dorien-koelemeijer
dorien-koelemeijer added this pull request to the merge queue Jun 30, 2026
Merged via the queue into main with commit 66e981e Jun 30, 2026
25 checks passed
@dorien-koelemeijer
dorien-koelemeijer deleted the fix/otlp-serve-mode-batch-panic branch June 30, 2026 07:18
lifeizhou-ap added a commit that referenced this pull request Jul 1, 2026
* main: (26 commits)
  Fix MCP app sandbox bridge lifecycle (#10064)
  fix(bedrock): send inference config (max_tokens, temperature) on Converse (#9889)
  feat(providers): support OpenRouter request parameters (#9276)
  Migrate local inference model management to ACP (#10124)
  (attempt to) fix disk space errors in linux release builds (#10024)
  feat: add --edit session flag to edit conversation before forking (#9799)
  feat: add iFlytek Spark and Astron MaaS providers (#9837)
  fix(desktop): dedupe Nostr session deep link imports (#9918)
  [codex] Add SessionStart hook parity outside CLI (#9970)
  feat(providers): add Fireworks AI declarative provider (#9990)
  fix(providers): don't retry deterministically-permanent 400s (thinking-block immutability) (#10005)
  fix(deps): downgrade pkcs8 to v0.10 to match sec1/pkcs1 v0.7 (#10119)
  chore(deps): bump actions/cache from 5.0.2 to 6.0.0 (#10051)
  Make OpenAI Responses API store param configurable (#10040)
  remove unsupported model (#10121)
  chore(release): bump version to 1.40.0 (minor) (#10099)
  move ollama provider into goose-providers (#9986)
  UI acp migratoin: Decouple desktop UI types from generated OpenAPI types (#10109)
  fix(otel): use async reqwest client so OTLP export works in `goose serve` mode (#10100)
  feat (acp): exposed available tools in acp schema (#10097)
  ...
lifeizhou-ap added a commit that referenced this pull request Jul 1, 2026
* main: (42 commits)
  Fix MCP app sandbox bridge lifecycle (#10064)
  fix(bedrock): send inference config (max_tokens, temperature) on Converse (#9889)
  feat(providers): support OpenRouter request parameters (#9276)
  Migrate local inference model management to ACP (#10124)
  (attempt to) fix disk space errors in linux release builds (#10024)
  feat: add --edit session flag to edit conversation before forking (#9799)
  feat: add iFlytek Spark and Astron MaaS providers (#9837)
  fix(desktop): dedupe Nostr session deep link imports (#9918)
  [codex] Add SessionStart hook parity outside CLI (#9970)
  feat(providers): add Fireworks AI declarative provider (#9990)
  fix(providers): don't retry deterministically-permanent 400s (thinking-block immutability) (#10005)
  fix(deps): downgrade pkcs8 to v0.10 to match sec1/pkcs1 v0.7 (#10119)
  chore(deps): bump actions/cache from 5.0.2 to 6.0.0 (#10051)
  Make OpenAI Responses API store param configurable (#10040)
  remove unsupported model (#10121)
  chore(release): bump version to 1.40.0 (minor) (#10099)
  move ollama provider into goose-providers (#9986)
  UI acp migratoin: Decouple desktop UI types from generated OpenAPI types (#10109)
  fix(otel): use async reqwest client so OTLP export works in `goose serve` mode (#10100)
  feat (acp): exposed available tools in acp schema (#10097)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants