feat(telemetry): capture tool results in Level 3 content and emit execute_tool spans

docs/ADRs/0050-distributed-tracing-instrumentation.md

@@ -171,3 +171,9 @@ beside telemetry when at least one new score is produced (tool-agnostic). Distin
 spans only; the root span keeps `fullsend.cost_usd` and `fullsend.tool_calls`
 (custom-namespaced, not auto-summed by MLflow). This prevents MLflow from
 double-counting token usage across the trace.
+
+**2026-09-03 — Tool-call span topology ([ADR 0108](0108-tool-call-span-topology.md)):**
+each tool call the runtime reports becomes an `execute_tool` child of its
+iteration's `agent` span, metadata only; the message record on the `agent`
+span stays the content carrier. Sub-agent nesting (deferred item 1 above)
+remains deferred.

docs/ADRs/0108-tool-call-span-topology.md

@@ -0,0 +1,110 @@
+---
+title: "108. Tool-call span topology"
+status: Accepted
+relates_to:
+  - operational-observability
+topics:
+  - observability
+  - telemetry
+  - opentelemetry
+---
+
+# 108. Tool-call span topology
+
+Date: 2026-09-03
+
+## Status
+
+Accepted
+
+## Context
+
+[ADR 0050](0050-distributed-tracing-instrumentation.md) chose OpenTelemetry
+and a three-level opt-in but never named the spans: the
+`run → sandbox_create → agent` tree lives only in the guides, and granularity
+was left to [#294](https://github.com/fullsend-ai/fullsend/issues/294).
+Level 3 content ([#6429](https://github.com/fullsend-ai/fullsend/pull/6429),
+[#6603](https://github.com/fullsend-ai/fullsend/pull/6603)) put tool calls
+and results on the `agent` span as `gen_ai.output.messages` parts — at
+semconv v1.37.0 the `execute_tool` span is metadata-only; its content
+attributes (`gen_ai.tool.call.arguments`, `.result`) arrive in v1.38.0 as
+Opt-In and Development. Review of #6603 asked why tool calls are not spans.
+
+fullsend observes the runtime's stream rather than executing tools. The
+normalized `ToolUseEvent`/`ToolResultEvent` pairs carry a call id, a tool
+name and an `is_error` flag; Claude Code's `assistant` (tool_use) and `user`
+(tool_result) stream lines each carry a sandbox-clock `timestamp` the parser
+does not decode; pi's tool_execution lines — the only pi lines the parser
+reads for calls — carry none (its session header and message lines do) and
+codex's items carry none; several calls are open at once (parallel sub-agent
+dispatch); `parent_tool_use_id` is dropped at decode; the pi and codex
+parsers pass no call ids through.
+
+## Options
+
+1. **Message record only** (status quo): tool calls stay parts of
+   `gen_ai.output.messages`; no per-call timing or status in the span tree.
+2. **`execute_tool` child spans from the normalized events**: one span per
+   call under the iteration's `agent` span, metadata only; content stays on
+   the message record.
+3. **The runtime's native OpenTelemetry**: Claude Code emits
+   `claude_code.tool` spans (beta) and honours inbound `TRACEPARENT` — true
+   timing, but redaction would leave fullsend's pipeline, the threat model
+   keeps runtime telemetry out of scope, and the sandbox would need egress.
+4. **Per-tool content on the spans**: needs v1.38.0's Opt-In attributes,
+   and the scorers read the message record today.
+
+## Decision
+
+Option 2. `toolSpanTracker` (`internal/cli/tool_spans.go`) opens an
+`execute_tool <tool name>` span (kind Internal) when the runtime reports a
+call and ends it when the result arrives. Both timestamps are runner-side
+receipt instants: the parent `agent` span's clock, so a child never falls
+outside its parent through cross-host skew, and the one source every runtime
+provides. The cost is looser bracketing — receipt trails the sandbox's own
+timestamps by the pipe latency — and the start is arguments-complete.
+Decoding Claude Code's timestamps into `trace.WithTimestamp` would tighten
+that runtime alone and is left open; upstream documents the assistant-line
+field as optional, host-clock and display-only, and the user-line field as
+optional with a receive-time fallback, so a decode must fall back to receipt
+time. Attributes follow semconv v1.37.0:
+`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
+`gen_ai.tool.call.id`; a result flagged `is_error` sets
+`error.type=tool_error` and status Error. Calls that never get a result
+close as `error.type=unanswered`, results for calls never reported are
+marked `fullsend.tool.unmatched`, and events without a call id (pi, codex,
+server-side tools) get no span — the edge cases are specified in the [dev
+guide](../guides/dev/tracing.md#execute_tool-spans). Names and call ids pass
+through the same sanitizer as span content — names bounded, ids dropped on
+any finding; at most 1,024 spans are recorded per iteration, so an agent-
+controlled burst cannot fill the OTLP batch queue and evict the `agent`
+span, with the overflow counted in `fullsend.tool_spans.dropped`; the cap
+assumes the default `OTEL_BSP_MAX_QUEUE_SIZE` (2048), and a smaller queue
+lowers the protection. The spans are Level 1 metadata, emitted regardless of
+the content gate. Tool content — results now, full arguments next — stays on
+the `agent` span's `gen_ai.output.messages` record, which is the scorer
+contract ([ADR 0087](0087-eval-measurements-online-trace-scoring.md)).
+
+Option 3 is the candidate to revisit once the runtime's tracing is stable and
+a redaction stage outside fullsend is designed; option 4 waits for the
+convention bump.
+
+## Consequences
+
+- The span tree gains one level; readers that select spans by name
+  (`evalmeasure`) are unaffected, and the `execute_tool` count can be below
+  `fullsend.tool_calls`, which counts every reported call, id or not.
+- Tool-heavy iterations add hundreds of spans, each one synchronous write to
+  `run-telemetry.jsonl` and one OTLP batch entry.
+- Full per-call volume is intended: the spans are the unit of tool-level
+  observability, the cap is a pathological-case backstop rather than a knob,
+  and no switch or sampler is added — sampling is per trace and would drop
+  whole runs, not thin these spans; the levers stay the standard ones
+  (endpoint unset keeps the file only; `OTEL_SDK_DISABLED`).
+- Sub-agent calls are flat children of the `agent` span; nesting them under
+  the dispatching `Agent` call is a small follow-up now that the parent span
+  exists while its children run (ADR 0050's deferred item 1 stays deferred).
+- `execute_tool` spans are Claude-only until the pi and codex parsers pass
+  their streams' call ids through.
+- This settles the span-granularity question in #294; retention and access
+  remain open there.

docs/architecture.md

@@ -373,6 +373,8 @@ Observability is a cross-cutting concern that touches every other component. Eac
 - Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
 - Eval measurements: the concept of scoring traces ([fail-open](glossary.md#fail-open)). [OTEL primary facts](glossary.md#otel-primary-facts) stay on the run trace (`run-telemetry.jsonl`); [OTEL derived products](glossary.md#otel-derived-products) are the scores (`eval-measurements.jsonl`) ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). See [Eval Measurements](guides/infrastructure/eval-measurements.md). When `OTEL_EXPORTER_OTLP_*` is set, scores also export as `gen_ai.evaluation.result` span events on the same TraceID (same OTLP path as agent traces; fail-open).
 
+- Tool-call span topology: every tool call the runtime reports becomes an `execute_tool` child span of its iteration's `agent` span — semantic-convention metadata only, timed at runner receipt; tool content stays on the `agent` span's message record ([ADR 0108](ADRs/0108-tool-call-span-topology.md)).
+
 **Open questions:**
 
 - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?

docs/guides/dev/tracing.md

@@ -91,12 +91,13 @@ silently.
 
 ## Span lifecycle in run.go
 
-`run.go` creates three span types arranged in a parent-child hierarchy:
+`run.go` creates four span types arranged in a parent-child hierarchy:
 
 ```
 run (root)
 ├── sandbox_create    (gen_ai.operation.name=create_agent)
 └── agent             (one per iteration; gen_ai.operation.name=invoke_agent)
+    └── execute_tool  (one per tool call; gen_ai.operation.name=execute_tool)
 ```
 
 ### Root span
@@ -132,6 +133,39 @@ build the attribute slices. Start attributes: `iteration`,
 `exit_code`, `gen_ai.system`, model, token counts, `fullsend.cost_usd`,
 `fullsend.tool_calls`.
 
+### execute_tool spans
+
+One per tool call the runtime reports, a child of that iteration's agent
+span, named `execute_tool <tool name>`. `toolSpanTracker`
+(`internal/cli/tool_spans.go`) starts the span when the `ToolUseEvent`
+arrives and ends it when the matching `ToolResultEvent` arrives, so both
+timestamps are runner-side receipt instants on one clock — the start is
+arguments-complete, not execution start. Attributes:
+`gen_ai.operation.name=execute_tool`, `gen_ai.tool.name`,
+`gen_ai.tool.call.id`; a result flagged `is_error` sets
+`error.type=tool_error` and status Error. A call still open when the
+iteration ends — the runtime was stopped, or its result line exceeded the
+parser's 1 MiB cap — is closed by `Finish()` as `error.type=unanswered`; a
+call superseded by a second `tool_use` with the same id is ended the same
+way at the reuse; a result with no matching call (its `tool_use` line was skipped) becomes a
+near-zero-duration span marked `fullsend.tool.unmatched=true`. Events without
+an id — pi and codex emit none — produce no span, and a `server_tool_use`
+block on an `assistant` line produces no event at all (its result never
+arrives as a `tool_result`), so the child count can be below
+`fullsend.tool_calls`. The name passes through `security.OutputPipeline()`
+— Unicode normalization, then secret redaction, the same pipeline as span
+content — and is bounded to 256 bytes before it becomes the attribute; the
+span name keeps at most 128 bytes of it. The call id is scanned through the
+same pipeline and dropped from the span on any finding — never substituted,
+since a masked id could collide with another call's — while the raw bounded
+id still keys the open-call map, so correlation is unaffected.
+The tracker records at most `maxToolSpansPerIteration` (1,024) spans per
+iteration and reports the overflow, which `runAgent` records as
+`fullsend.tool_spans.dropped` on the agent span — a burst of agent-controlled
+calls must not fill the OTLP batch queue and evict the agent span that ends
+right after `Finish()`. These spans are metadata: they carry no tool content
+and are emitted whether or not the Level 3 gate is on.
+
 ### Level 3 content on agent spans
 
 When the content-capture gate is on
@@ -143,18 +177,26 @@ spans — and tees the runtime's normalized event stream to it through
 
 **The tee trap:** supplying any `OnEvent` replaces the runtime's default
 console renderer (`internal/runtime/claude.go`), so the handler built by
-`contentEventHandler` always calls the renderer first and the collector
-second. With the gate off the collector is nil and `contentEventHandler`
-returns nil, leaving the default renderer path byte-identical to before
-Level 3 existed.
+`iterationEventHandler` always calls the renderer first, then the
+collector, then the tool-span tracker. The handler is always set — tool
+spans are emitted with the gate off — and with the gate off the collector
+is nil and inert, so console output stays byte-identical to the default
+renderer path.
 
 The collector (`internal/cli/content_collector.go`) coalesces contiguous
-text/reasoning deltas, maps tool use to `tool_call` parts, redacts every
+text/reasoning deltas, maps tool use to `tool_call` parts and tool
+results to `tool_call_response` parts (correlated by `id` when the
+runtime's stream provides one; the schema's required result field is
+`response`), redacts every
 part through `security.OutputPipeline()` at assembly (redaction runs
 before the size budget — truncating first could split a secret past
 recognition), enforces a 256 KiB ordered-suffix budget (the ending survives — the
-final answer is what consumers judge) with exact dropped-byte accounting
-across content, tool names, and summaries, and emits
+final answer is what consumers judge) plus an 8 KiB per-tool-result
+bound (tail-kept, redacted before the cut, the part marked
+`fullsend.truncated`), with exact dropped-byte
+accounting across content, tool names, summaries, responses, and part
+ids (a third, earlier boundary — the parser's 1 MiB stream-line cap —
+drops oversized lines before any event exists), and emits
 `gen_ai.output.messages` JSON following the GenAI output-messages schema,
 including the schema-required `finish_reason` from the iteration outcome. `attachContent` records the
 content and its marker attributes on the span before either