docs: ADR 0050 — distributed tracing instrumentation with OpenTelemetry

docs/ADRs/0021-jsonl-reasoning-trace-exposure.md

@@ -162,4 +162,4 @@ it suppresses JSONL for nearly all useful runs on private repos.
 - Raw JSONL serves per-run consumers (retro agent, session resumption,
   human debugging). Complementary structured extraction via OpenTelemetry
   could power aggregate analysis at scale (pattern detection across many
-  runs) — a future decision, not in scope here.
+  runs) — subsequently decided in [ADR 0050](0050-distributed-tracing-instrumentation.md).

docs/ADRs/0050-distributed-tracing-instrumentation.md

@@ -0,0 +1,143 @@
+---
+title: "50. Framework-native distributed tracing with OpenTelemetry"
+status: Accepted
+relates_to:
+  - operational-observability
+topics:
+  - observability
+  - telemetry
+  - opentelemetry
+---
+
+# 50. Framework-native distributed tracing with OpenTelemetry
+
+Date: 2026-05-23
+
+## Status
+
+Accepted
+
+## Context
+
+Fullsend agent runs are opaque. When a multi-agent pipeline dispatches
+triage → code → review, operators have no structured way to understand what
+happened, how long each step took, or where a failure occurred. The
+[operational observability](../problems/operational-observability.md) problem
+doc identifies this as a first-order concern.
+
+Fullsend is distributed to many organizations — not just our team. The
+tracing design must be safe by default without requiring any configuration
+from adopters. Setting an OTLP endpoint must never accidentally expose
+sensitive content (prompts, source code, PII) to shared or SaaS backends.
+
+Prior decisions that inform this one:
+
+- [ADR 0021](0021-jsonl-reasoning-trace-exposure.md) — JSONL reasoning trace
+  exposure (what traces contain, who can access them)
+- [ADR 0018](0018-scripted-pipeline-for-multi-agent-orchestration.md) —
+  scripted multi-agent pipeline whose cross-run correlation this enables
+- [ADR 0022](0022-harness-level-output-schema-enforcement.md) — structured
+  output schemas that `run-summary.json` complements
+
+## Options
+
+### A. Post-hoc parsing (rejected)
+
+External tooling parses CLI stdout after runs to construct spans. Fragile:
+stdout is not a stable contract, timing is approximate, and intermediate
+state is lost. The early Arize Phoenix experiment confirmed this.
+
+### B. Framework-native OpenTelemetry (accepted)
+
+CLI emits OTEL spans at source. Zero-infrastructure baseline (local files),
+one env var enables OTLP export. Backend-agnostic. Content capture requires
+explicit opt-in per OTEL GenAI semantic conventions.
+
+### C. Vendor-specific trace format (rejected)
+
+A runtime-locked trace builder (e.g., Claude-specific). Breaks when fullsend
+adds support for other runtimes (OpenCode, Gemini CLI). Not portable.
+
+## Decision
+
+Fullsend instruments the CLI natively using OpenTelemetry with a three-level
+opt-in model:
+
+**Level 1 — Local baseline (every install, zero config):**
+- Every run produces `run-telemetry.jsonl` and `run-summary.json` in the output
+  directory (uploaded as GHA artifacts alongside transcripts)
+- Metadata only: span hierarchy, timing, token counts, tool names, errors
+- No data leaves the runner. No backend required.
+
+**Level 2 — OTLP export (org opts in by setting endpoint):**
+- When `OTEL_EXPORTER_OTLP_ENDPOINT` is set, metadata spans export via
+  OTLP/HTTP to the org's chosen backend
+- Still metadata only — safe for any backend including shared/SaaS platforms
+- Spans follow [OTEL GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)
+  (`gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.request.model`,
+  `gen_ai.system`)
+- W3C `TRACEPARENT` propagation enables cross-run correlation for dispatched
+  pipelines; separate workflow runs require manual propagation
+
+**Level 3 — Content capture (org explicitly opts in):**
+- When `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is set, full
+  prompt/completion content is included in spans
+- Org is responsible for ensuring their backend's access controls are
+  appropriate for the content sensitivity
+- Enables LLM-judge evaluation scorers that need to read agent reasoning
+
+**Additional design properties:**
+- Runtime-agnostic: any runtime satisfying a transcript contract (turns,
+  tools, tokens, model, stop reason) gets span promotion
+- If the OTLP endpoint is unreachable, the CLI continues normally — local
+  files still produced, run is not affected
+- Simultaneous export to multiple backends is achieved by deploying an
+  [OTEL Collector](https://opentelemetry.io/docs/collector/) as the endpoint;
+  the CLI exports to one OTLP endpoint, the Collector fans out
+
+**Scope boundary:** This ADR decides how traces are *generated* and how
+content sensitivity is handled. Agent quality evaluation (scoring, regression
+detection, baselines) *consumes* trace data but is a separate architectural
+concern. Choice of backend is an adopter decision, not a platform decision.
+
+## Consequences
+
+- Every org gets structured observability with zero configuration (local files)
+- OTLP export is always safe to enable (metadata only by default)
+- Content capture is an explicit second opt-in — prevents accidental exposure
+  of proprietary code or PII to shared/SaaS backends
+- Any OTLP-compatible backend works (Jaeger, Tempo, MLflow, Phoenix,
+  Langfuse, SigNoz, Honeycomb, Datadog)
+- Cross-run correlation via `TRACEPARENT` for dispatched pipelines
+- GenAI-aware backends get agent dashboards without CLI changes
+- Runtime-agnostic: adding new runtimes doesn't require new trace formats
+- The `gen_ai.*` attributes follow experimental OTEL semantic conventions
+  and may change in future OTEL releases
+
+## Deferred to implementation
+
+These items are in scope for the implementation phase, not this architectural
+decision:
+
+1. **Sub-agent recursive span expansion** — When an agent dispatches sub-agents
+   via `tool:Agent` (e.g., review agent's 6 sub-agents), their turns should
+   become nested span subtrees, not flat spans. The transcript contract must
+   handle recursive agent invocations.
+
+2. **Pre/post script span instrumentation** — Pre-scripts, post-scripts, and
+   validation scripts do significant work but aren't addressed in span
+   structure. Define whether the framework instruments their execution
+   automatically or provides a contract for scripts to emit spans.
+
+## Related issues
+
+- [#294](https://github.com/fullsend-ai/fullsend/issues/294) — Define trace
+  granularity and retention policy
+- [#295](https://github.com/fullsend-ai/fullsend/issues/295) — Define
+  quality metrics for autonomous software factory
+- [#296](https://github.com/fullsend-ai/fullsend/issues/296) — Evaluate
+  Langfuse deployment threshold vs structured logging
+- [#2367](https://github.com/fullsend-ai/fullsend/issues/2367) — Add
+  `fullsend.runtime` trace attribute for multi-runtime observability
+- [#2368](https://github.com/fullsend-ai/fullsend/issues/2368) — Add
+  `fullsend.harness.content_sha` trace attribute for config change correlation

docs/architecture.md

@@ -197,11 +197,12 @@ Observability is a cross-cutting concern that touches every other component. Eac
 
 - JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
 - Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
+- Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` and `run-summary.json` locally; optional OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
 
 **Open questions:**
 
 - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?
-- How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?
+- ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source.
 - What is the retention and access model for agent logs? Who can see what? (JSONL trace access model decided in [ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md); retention policy and broader log access remain open.)
 - How does observability interact with the security requirement that "every action is logged, attributable, and reviewable"? (See [security-threat-model.md](problems/security-threat-model.md).)
 - Is there a real-time monitoring requirement (agent is stuck, agent is behaving anomalously), or is observability primarily forensic?

docs/guides/README.md

@@ -17,6 +17,7 @@ Advanced guides for platform operators who deploy and manage the GCP-side infras
 - [Mint service administration](infrastructure/mint-administration.md) — Deploying and managing the token mint Cloud Function
 - [Infrastructure reference](infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details
 - [Enabling fullsend on private repositories](infrastructure/private-repositories.md) — Additional guardrails and configuration for private repos
+- [Distributed tracing](infrastructure/distributed-tracing.md) — Configuring OpenTelemetry instrumentation and OTLP backends
 
 ## User guides
 

docs/guides/infrastructure/distributed-tracing.md

@@ -0,0 +1,193 @@
+# Distributed Tracing
+
+Fullsend produces structured telemetry for every agent run. This guide covers
+how to configure, consume, and extend the tracing system.
+
+Decided in [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md).
+
+## Zero-configuration baseline (Level 1)
+
+Every `fullsend run` produces two files in the run output directory with no
+configuration required:
+
+- **`run-telemetry.jsonl`** — NDJSON stream of lifecycle events (step starts,
+  completions, failures, warnings) with timestamps, durations, and trace IDs.
+- **`run-summary.json`** — Aggregated run summary including agent name, exit
+  code, step timings, total duration, and a W3C `traceparent` value for
+  downstream correlation.
+
+These files are always written, even when no OTLP backend is configured. They
+contain metadata only — no prompts, completions, or source code content.
+
+## Enabling OTLP export (Level 2)
+
+To send metadata spans to an OpenTelemetry-compatible backend, set one of the
+standard OTEL environment variables:
+
+```bash
+# Signal-specific (takes precedence, used as-is — no /v1/traces appended)
+export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-backend:4318/v1/traces"
+
+# Base URL (SDK appends /v1/traces automatically)
+export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-backend:4318"
+```
+
+**Precedence:** `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` > `OTEL_EXPORTER_OTLP_ENDPOINT`.
+Headers follow the same pattern: `OTEL_EXPORTER_OTLP_TRACES_HEADERS` > `OTEL_EXPORTER_OTLP_HEADERS`.
+
+Local files (`run-telemetry.jsonl`, `run-summary.json`) are always produced
+with no configuration needed (Level 1).
+
+When an endpoint is configured, spans are exported via OTLP/HTTP. Any backend
+that speaks OTLP works: Jaeger, Grafana Tempo, MLflow, Arize Phoenix,
+Langfuse, SigNoz, Honeycomb, Datadog, etc.
+
+If the endpoint is unreachable, the CLI continues normally — local files are
+still produced and the run is not affected.
+
+## Enabling content capture (Level 3)
+
+By default, spans contain metadata only (timing, token counts, tool names,
+errors). To include full prompt/completion content in spans:
+
+```bash
+export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true
+```
+
+This follows the [OTEL GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/v1.37.0/docs/gen-ai/gen-ai-spans.md)
+which mandate that content capture is opt-in. When enabled, spans include:
+
+- System prompts and user messages
+- Tool arguments and results (file contents, command output)
+- Agent reasoning/thinking text
+- Completion text
+
+**Warning:** Only enable content capture when your backend's access controls
+are appropriate for the sensitivity of the data. Content may include
+proprietary source code, issue descriptions with PII, or credentials visible
+in tool outputs.
+
+## Cross-run trace correlation
+
+Multi-agent pipelines (triage → code → review) propagate trace context via
+the `TRACEPARENT` environment variable (W3C Trace Context).
+
+When a workflow dispatches a child run:
+
+```yaml
+env:
+  TRACEPARENT: ${{ steps.parent.outputs.traceparent }}
+```
+
+The child run's root span becomes part of the parent trace, creating a
+unified view of the entire pipeline.
+
+For separate workflow runs on the same work item (triage → code → review as
+independent GHA workflows), `TRACEPARENT` must be propagated manually — for
+example, via hidden issue/PR comments. GitHub webhooks do not support custom
+trace headers natively.
+
+The `run-summary.json` includes the `traceparent` value so downstream
+consumers (scripts, other agents) can continue the trace chain.
+
+## Span structure
+
+A typical agent run produces this span hierarchy:
+
+```
+fullsend-run (root, SpanKind=Consumer if dispatched)
+├── load-harness
+├── setup-sandbox
+│   └── create-sandbox (gen_ai.operation.name=create_agent)
+├── agent-execution.iteration-0
+│   └── (gen_ai.operation.name=invoke_agent)
+├── agent-execution.iteration-1
+├── collect-artifacts
+├── security-scan
+└── validation
+```
+
+### GenAI semantic conventions
+
+Root and iteration spans carry [OTEL GenAI semantic convention](https://opentelemetry.io/docs/specs/semconv/gen-ai/) attributes:
+
+| Attribute | Example | Description |
+|-----------|---------|-------------|
+| `gen_ai.operation.name` | `invoke_agent` | The GenAI operation type |
+| `gen_ai.agent.name` | `triage` | The agent being executed |
+| `gen_ai.request.model` | `claude-sonnet-4-20250514` | The model configured in the harness |
+| `gen_ai.system` | `anthropic` | The LLM provider |
+
+These attributes enable LLM-aware backends to recognize fullsend spans as
+agent operations and surface them in GenAI-specific dashboards.
+
+### SpanKind
+
+- **Consumer**: The root span when `TRACEPARENT` is set (the run was
+  dispatched by an external system).
+- **Internal**: The root span for local/manual invocations.
+
+## Custom attributes
+
+Every span also carries fullsend-specific attributes:
+
+| Attribute | Description |
+|-----------|-------------|
+| `fullsend.agent` | Agent name from the harness |
+| `fullsend.harness` | Path to the harness YAML |
+| `fullsend.model` | Model identifier |
+| `fullsend.image` | Container image used |
+| `fullsend.work_item_id` | Issue/PR number being addressed |
+
+## GHA workflow configuration example
+
+Add these environment variables to workflow jobs that run `fullsend run`:
+
+```yaml
+env:
+  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "${{ secrets.OTLP_ENDPOINT }}"
+  OTEL_EXPORTER_OTLP_TRACES_HEADERS: "Authorization=Bearer ${{ secrets.OTLP_TOKEN }}"
+```
+
+The secret names and values depend on your chosen backend. Consult your
+backend's documentation for the endpoint URL and authentication mechanism.
+
+## Local development
+
+Run an agent locally with traces going to a local backend:
+
+```bash
+# Start a local Jaeger instance (OTLP-compatible)
+podman run -d --name jaeger \
+  -p 16686:16686 \
+  -p 4318:4318 \
+  jaegertracing/jaeger
+
+# Run an agent with tracing enabled
+export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
+fullsend run triage --issue 42
+
+# View traces at http://localhost:16686
+```
+
+Other lightweight local backends:
+
+| Backend | Command | UI |
+|---------|---------|-----|
+| Jaeger | `podman run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger` | `localhost:16686` |
+| Arize Phoenix | `podman run -p 6006:6006 -p 4318:4318 arizephoenix/phoenix` | `localhost:6006` |
+| MLflow | `uvx mlflow server` (with OTLP plugin) | `localhost:5000` |
+
+## Other backends
+
+Any OTLP-compatible backend works. Choosing an LLM-aware backend (MLflow,
+Phoenix, Langfuse) activates GenAI dashboards — token cost rollups,
+prompt/completion inspection, agent-specific views — without any CLI-side
+configuration change. The `gen_ai.*` span attributes are recognized
+automatically.
+
+For production deployments, consult your backend's documentation for:
+- High-availability configuration
+- Authentication and access control
+- Data retention policies
+- Cost considerations for high-volume trace ingestion

docs/problems/operational-observability.md

@@ -192,7 +192,7 @@ This works for early experimentation when the volume is low and the operators ar
 - What retention policy applies to traces? Indefinite retention supports audit requirements but increases storage cost and data sensitivity exposure. Time-bounded retention (e.g., 90 days) limits exposure but may lose traces needed for incident investigation.
 - How do we measure "is the system getting better"? What metrics constitute a meaningful quality signal for an autonomous software factory? Merge revert rate? Human override rate? Time-to-review? Cost per decision? Some composite score? The choice of metric shapes what gets optimized.
 - At what scale does a dedicated LLM observability platform justify its operational overhead (Postgres, ClickHouse, Redis, S3 for something like Langfuse)? Is there a threshold of agent activity below which structured logging suffices?
-- How do we handle the bootstrapping problem — the factory needs observability to improve, but building the observability infrastructure is itself work that competes with building the factory?
+- ~~How do we handle the bootstrapping problem — the factory needs observability to improve, but building the observability infrastructure is itself work that competes with building the factory?~~ Decided in [ADR 0050](../ADRs/0050-distributed-tracing-instrumentation.md): zero-configuration baseline (local JSONL + summary files) eliminates infrastructure requirements for initial observability; OTLP export adds backends when the org is ready.
 - Should observability data feed back into agent instructions automatically (e.g., auto-adjusting prompts when false positive rates exceed a threshold), or should it only inform human-driven instruction changes? Automatic feedback creates the risk of instruction oscillation; human-only feedback is slower but more controlled.
 - How do we build community dashboards that are useful to contributors with different levels of technical depth — from "is the agent doing a good job on my repo" to "show me the trace of this specific review"?
 - What is the cost of observability itself? Storing traces, running evaluators, maintaining dashboards — this has infrastructure cost. At what scale does it pay for itself in debugging time saved and quality improvement?