From a777a5dbded07884288e2ad2f16c7dd34273883a Mon Sep 17 00:00:00 2001 From: Adam Scerra Date: Tue, 16 Jun 2026 15:13:18 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20ADR=200048=20=E2=80=94=20distributed=20?= =?UTF-8?q?tracing=20instrumentation=20with=20OpenTelemetry?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add ADR recording the decision to instrument fullsend with OpenTelemetry using a three-level opt-in model (local files → OTLP metadata export → content capture). Separates telemetry from evaluation concerns. Key changes: - ADR 0048: three-level content sensitivity model per OTEL GenAI spec, explicit scope boundary (evals consume traces, separate concern), multi-backend via OTEL Collector (not multi-endpoint config) - Infrastructure guide: env var precedence, local dev section, content capture warning; backend-agnostic language throughout - Cross-reference annotation in ADR 0021 (OTel future → now decided) - Update cross-references in architecture.md and problem doc Addresses review feedback from ralphbean, maruiz93, and review bot. Signed-off-by: Adam Scerra Co-authored-by: Cursor Signed-off-by: Adam Scerra Co-authored-by: Cursor Signed-off-by: Adam Scerra Co-authored-by: Cursor Signed-off-by: Adam Scerra Co-authored-by: Cursor Signed-off-by: Adam Scerra Co-authored-by: Cursor Signed-off-by: Adam Scerra Co-authored-by: Cursor --- .../0021-jsonl-reasoning-trace-exposure.md | 2 +- ...050-distributed-tracing-instrumentation.md | 143 +++++++++++++ docs/architecture.md | 3 +- docs/guides/README.md | 1 + .../infrastructure/distributed-tracing.md | 193 ++++++++++++++++++ docs/problems/operational-observability.md | 2 +- 6 files changed, 341 insertions(+), 3 deletions(-) create mode 100644 docs/ADRs/0050-distributed-tracing-instrumentation.md create mode 100644 docs/guides/infrastructure/distributed-tracing.md diff --git a/docs/ADRs/0021-jsonl-reasoning-trace-exposure.md b/docs/ADRs/0021-jsonl-reasoning-trace-exposure.md index 81d5c0b9e7..062e030d59 100644 --- a/docs/ADRs/0021-jsonl-reasoning-trace-exposure.md +++ b/docs/ADRs/0021-jsonl-reasoning-trace-exposure.md @@ -162,4 +162,4 @@ it suppresses JSONL for nearly all useful runs on private repos. - Raw JSONL serves per-run consumers (retro agent, session resumption, human debugging). Complementary structured extraction via OpenTelemetry could power aggregate analysis at scale (pattern detection across many - runs) — a future decision, not in scope here. + runs) — subsequently decided in [ADR 0050](0050-distributed-tracing-instrumentation.md). diff --git a/docs/ADRs/0050-distributed-tracing-instrumentation.md b/docs/ADRs/0050-distributed-tracing-instrumentation.md new file mode 100644 index 0000000000..9a0fe4b8b0 --- /dev/null +++ b/docs/ADRs/0050-distributed-tracing-instrumentation.md @@ -0,0 +1,143 @@ +--- +title: "50. Framework-native distributed tracing with OpenTelemetry" +status: Accepted +relates_to: + - operational-observability +topics: + - observability + - telemetry + - opentelemetry +--- + +# 50. Framework-native distributed tracing with OpenTelemetry + +Date: 2026-05-23 + +## Status + +Accepted + +## Context + +Fullsend agent runs are opaque. When a multi-agent pipeline dispatches +triage → code → review, operators have no structured way to understand what +happened, how long each step took, or where a failure occurred. The +[operational observability](../problems/operational-observability.md) problem +doc identifies this as a first-order concern. + +Fullsend is distributed to many organizations — not just our team. The +tracing design must be safe by default without requiring any configuration +from adopters. Setting an OTLP endpoint must never accidentally expose +sensitive content (prompts, source code, PII) to shared or SaaS backends. + +Prior decisions that inform this one: + +- [ADR 0021](0021-jsonl-reasoning-trace-exposure.md) — JSONL reasoning trace + exposure (what traces contain, who can access them) +- [ADR 0018](0018-scripted-pipeline-for-multi-agent-orchestration.md) — + scripted multi-agent pipeline whose cross-run correlation this enables +- [ADR 0022](0022-harness-level-output-schema-enforcement.md) — structured + output schemas that `run-summary.json` complements + +## Options + +### A. Post-hoc parsing (rejected) + +External tooling parses CLI stdout after runs to construct spans. Fragile: +stdout is not a stable contract, timing is approximate, and intermediate +state is lost. The early Arize Phoenix experiment confirmed this. + +### B. Framework-native OpenTelemetry (accepted) + +CLI emits OTEL spans at source. Zero-infrastructure baseline (local files), +one env var enables OTLP export. Backend-agnostic. Content capture requires +explicit opt-in per OTEL GenAI semantic conventions. + +### C. Vendor-specific trace format (rejected) + +A runtime-locked trace builder (e.g., Claude-specific). Breaks when fullsend +adds support for other runtimes (OpenCode, Gemini CLI). Not portable. + +## Decision + +Fullsend instruments the CLI natively using OpenTelemetry with a three-level +opt-in model: + +**Level 1 — Local baseline (every install, zero config):** +- Every run produces `run-telemetry.jsonl` and `run-summary.json` in the output + directory (uploaded as GHA artifacts alongside transcripts) +- Metadata only: span hierarchy, timing, token counts, tool names, errors +- No data leaves the runner. No backend required. + +**Level 2 — OTLP export (org opts in by setting endpoint):** +- When `OTEL_EXPORTER_OTLP_ENDPOINT` is set, metadata spans export via + OTLP/HTTP to the org's chosen backend +- Still metadata only — safe for any backend including shared/SaaS platforms +- Spans follow [OTEL GenAI Semantic Conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/) + (`gen_ai.operation.name`, `gen_ai.agent.name`, `gen_ai.request.model`, + `gen_ai.system`) +- W3C `TRACEPARENT` propagation enables cross-run correlation for dispatched + pipelines; separate workflow runs require manual propagation + +**Level 3 — Content capture (org explicitly opts in):** +- When `OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT` is set, full + prompt/completion content is included in spans +- Org is responsible for ensuring their backend's access controls are + appropriate for the content sensitivity +- Enables LLM-judge evaluation scorers that need to read agent reasoning + +**Additional design properties:** +- Runtime-agnostic: any runtime satisfying a transcript contract (turns, + tools, tokens, model, stop reason) gets span promotion +- If the OTLP endpoint is unreachable, the CLI continues normally — local + files still produced, run is not affected +- Simultaneous export to multiple backends is achieved by deploying an + [OTEL Collector](https://opentelemetry.io/docs/collector/) as the endpoint; + the CLI exports to one OTLP endpoint, the Collector fans out + +**Scope boundary:** This ADR decides how traces are *generated* and how +content sensitivity is handled. Agent quality evaluation (scoring, regression +detection, baselines) *consumes* trace data but is a separate architectural +concern. Choice of backend is an adopter decision, not a platform decision. + +## Consequences + +- Every org gets structured observability with zero configuration (local files) +- OTLP export is always safe to enable (metadata only by default) +- Content capture is an explicit second opt-in — prevents accidental exposure + of proprietary code or PII to shared/SaaS backends +- Any OTLP-compatible backend works (Jaeger, Tempo, MLflow, Phoenix, + Langfuse, SigNoz, Honeycomb, Datadog) +- Cross-run correlation via `TRACEPARENT` for dispatched pipelines +- GenAI-aware backends get agent dashboards without CLI changes +- Runtime-agnostic: adding new runtimes doesn't require new trace formats +- The `gen_ai.*` attributes follow experimental OTEL semantic conventions + and may change in future OTEL releases + +## Deferred to implementation + +These items are in scope for the implementation phase, not this architectural +decision: + +1. **Sub-agent recursive span expansion** — When an agent dispatches sub-agents + via `tool:Agent` (e.g., review agent's 6 sub-agents), their turns should + become nested span subtrees, not flat spans. The transcript contract must + handle recursive agent invocations. + +2. **Pre/post script span instrumentation** — Pre-scripts, post-scripts, and + validation scripts do significant work but aren't addressed in span + structure. Define whether the framework instruments their execution + automatically or provides a contract for scripts to emit spans. + +## Related issues + +- [#294](https://github.com/fullsend-ai/fullsend/issues/294) — Define trace + granularity and retention policy +- [#295](https://github.com/fullsend-ai/fullsend/issues/295) — Define + quality metrics for autonomous software factory +- [#296](https://github.com/fullsend-ai/fullsend/issues/296) — Evaluate + Langfuse deployment threshold vs structured logging +- [#2367](https://github.com/fullsend-ai/fullsend/issues/2367) — Add + `fullsend.runtime` trace attribute for multi-runtime observability +- [#2368](https://github.com/fullsend-ai/fullsend/issues/2368) — Add + `fullsend.harness.content_sha` trace attribute for config change correlation diff --git a/docs/architecture.md b/docs/architecture.md index cb6a422519..b9c01fc51a 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -197,11 +197,12 @@ Observability is a cross-cutting concern that touches every other component. Eac - JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)). - Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)). +- Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` and `run-summary.json` locally; optional OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)). **Open questions:** - What signals matter most — cost, latency, token usage, action logs, decision traces, or something else? -- How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce? +- ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source. - What is the retention and access model for agent logs? Who can see what? (JSONL trace access model decided in [ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md); retention policy and broader log access remain open.) - How does observability interact with the security requirement that "every action is logged, attributable, and reviewable"? (See [security-threat-model.md](problems/security-threat-model.md).) - Is there a real-time monitoring requirement (agent is stuck, agent is behaving anomalously), or is observability primarily forensic? diff --git a/docs/guides/README.md b/docs/guides/README.md index b7dda2bbb9..01767e9eb2 100644 --- a/docs/guides/README.md +++ b/docs/guides/README.md @@ -17,6 +17,7 @@ Advanced guides for platform operators who deploy and manage the GCP-side infras - [Mint service administration](infrastructure/mint-administration.md) — Deploying and managing the token mint Cloud Function - [Infrastructure reference](infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details - [Enabling fullsend on private repositories](infrastructure/private-repositories.md) — Additional guardrails and configuration for private repos +- [Distributed tracing](infrastructure/distributed-tracing.md) — Configuring OpenTelemetry instrumentation and OTLP backends ## User guides diff --git a/docs/guides/infrastructure/distributed-tracing.md b/docs/guides/infrastructure/distributed-tracing.md new file mode 100644 index 0000000000..34f47bab78 --- /dev/null +++ b/docs/guides/infrastructure/distributed-tracing.md @@ -0,0 +1,193 @@ +# Distributed Tracing + +Fullsend produces structured telemetry for every agent run. This guide covers +how to configure, consume, and extend the tracing system. + +Decided in [ADR 0050](../../ADRs/0050-distributed-tracing-instrumentation.md). + +## Zero-configuration baseline (Level 1) + +Every `fullsend run` produces two files in the run output directory with no +configuration required: + +- **`run-telemetry.jsonl`** — NDJSON stream of lifecycle events (step starts, + completions, failures, warnings) with timestamps, durations, and trace IDs. +- **`run-summary.json`** — Aggregated run summary including agent name, exit + code, step timings, total duration, and a W3C `traceparent` value for + downstream correlation. + +These files are always written, even when no OTLP backend is configured. They +contain metadata only — no prompts, completions, or source code content. + +## Enabling OTLP export (Level 2) + +To send metadata spans to an OpenTelemetry-compatible backend, set one of the +standard OTEL environment variables: + +```bash +# Signal-specific (takes precedence, used as-is — no /v1/traces appended) +export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT="https://your-backend:4318/v1/traces" + +# Base URL (SDK appends /v1/traces automatically) +export OTEL_EXPORTER_OTLP_ENDPOINT="https://your-backend:4318" +``` + +**Precedence:** `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` > `OTEL_EXPORTER_OTLP_ENDPOINT`. +Headers follow the same pattern: `OTEL_EXPORTER_OTLP_TRACES_HEADERS` > `OTEL_EXPORTER_OTLP_HEADERS`. + +Local files (`run-telemetry.jsonl`, `run-summary.json`) are always produced +with no configuration needed (Level 1). + +When an endpoint is configured, spans are exported via OTLP/HTTP. Any backend +that speaks OTLP works: Jaeger, Grafana Tempo, MLflow, Arize Phoenix, +Langfuse, SigNoz, Honeycomb, Datadog, etc. + +If the endpoint is unreachable, the CLI continues normally — local files are +still produced and the run is not affected. + +## Enabling content capture (Level 3) + +By default, spans contain metadata only (timing, token counts, tool names, +errors). To include full prompt/completion content in spans: + +```bash +export OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=true +``` + +This follows the [OTEL GenAI semantic conventions](https://github.com/open-telemetry/semantic-conventions/blob/v1.37.0/docs/gen-ai/gen-ai-spans.md) +which mandate that content capture is opt-in. When enabled, spans include: + +- System prompts and user messages +- Tool arguments and results (file contents, command output) +- Agent reasoning/thinking text +- Completion text + +**Warning:** Only enable content capture when your backend's access controls +are appropriate for the sensitivity of the data. Content may include +proprietary source code, issue descriptions with PII, or credentials visible +in tool outputs. + +## Cross-run trace correlation + +Multi-agent pipelines (triage → code → review) propagate trace context via +the `TRACEPARENT` environment variable (W3C Trace Context). + +When a workflow dispatches a child run: + +```yaml +env: + TRACEPARENT: ${{ steps.parent.outputs.traceparent }} +``` + +The child run's root span becomes part of the parent trace, creating a +unified view of the entire pipeline. + +For separate workflow runs on the same work item (triage → code → review as +independent GHA workflows), `TRACEPARENT` must be propagated manually — for +example, via hidden issue/PR comments. GitHub webhooks do not support custom +trace headers natively. + +The `run-summary.json` includes the `traceparent` value so downstream +consumers (scripts, other agents) can continue the trace chain. + +## Span structure + +A typical agent run produces this span hierarchy: + +``` +fullsend-run (root, SpanKind=Consumer if dispatched) +├── load-harness +├── setup-sandbox +│ └── create-sandbox (gen_ai.operation.name=create_agent) +├── agent-execution.iteration-0 +│ └── (gen_ai.operation.name=invoke_agent) +├── agent-execution.iteration-1 +├── collect-artifacts +├── security-scan +└── validation +``` + +### GenAI semantic conventions + +Root and iteration spans carry [OTEL GenAI semantic convention](https://opentelemetry.io/docs/specs/semconv/gen-ai/) attributes: + +| Attribute | Example | Description | +|-----------|---------|-------------| +| `gen_ai.operation.name` | `invoke_agent` | The GenAI operation type | +| `gen_ai.agent.name` | `triage` | The agent being executed | +| `gen_ai.request.model` | `claude-sonnet-4-20250514` | The model configured in the harness | +| `gen_ai.system` | `anthropic` | The LLM provider | + +These attributes enable LLM-aware backends to recognize fullsend spans as +agent operations and surface them in GenAI-specific dashboards. + +### SpanKind + +- **Consumer**: The root span when `TRACEPARENT` is set (the run was + dispatched by an external system). +- **Internal**: The root span for local/manual invocations. + +## Custom attributes + +Every span also carries fullsend-specific attributes: + +| Attribute | Description | +|-----------|-------------| +| `fullsend.agent` | Agent name from the harness | +| `fullsend.harness` | Path to the harness YAML | +| `fullsend.model` | Model identifier | +| `fullsend.image` | Container image used | +| `fullsend.work_item_id` | Issue/PR number being addressed | + +## GHA workflow configuration example + +Add these environment variables to workflow jobs that run `fullsend run`: + +```yaml +env: + OTEL_EXPORTER_OTLP_TRACES_ENDPOINT: "${{ secrets.OTLP_ENDPOINT }}" + OTEL_EXPORTER_OTLP_TRACES_HEADERS: "Authorization=Bearer ${{ secrets.OTLP_TOKEN }}" +``` + +The secret names and values depend on your chosen backend. Consult your +backend's documentation for the endpoint URL and authentication mechanism. + +## Local development + +Run an agent locally with traces going to a local backend: + +```bash +# Start a local Jaeger instance (OTLP-compatible) +podman run -d --name jaeger \ + -p 16686:16686 \ + -p 4318:4318 \ + jaegertracing/jaeger + +# Run an agent with tracing enabled +export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318" +fullsend run triage --issue 42 + +# View traces at http://localhost:16686 +``` + +Other lightweight local backends: + +| Backend | Command | UI | +|---------|---------|-----| +| Jaeger | `podman run -p 16686:16686 -p 4318:4318 jaegertracing/jaeger` | `localhost:16686` | +| Arize Phoenix | `podman run -p 6006:6006 -p 4318:4318 arizephoenix/phoenix` | `localhost:6006` | +| MLflow | `uvx mlflow server` (with OTLP plugin) | `localhost:5000` | + +## Other backends + +Any OTLP-compatible backend works. Choosing an LLM-aware backend (MLflow, +Phoenix, Langfuse) activates GenAI dashboards — token cost rollups, +prompt/completion inspection, agent-specific views — without any CLI-side +configuration change. The `gen_ai.*` span attributes are recognized +automatically. + +For production deployments, consult your backend's documentation for: +- High-availability configuration +- Authentication and access control +- Data retention policies +- Cost considerations for high-volume trace ingestion diff --git a/docs/problems/operational-observability.md b/docs/problems/operational-observability.md index be84a3ac04..91d75a9761 100644 --- a/docs/problems/operational-observability.md +++ b/docs/problems/operational-observability.md @@ -192,7 +192,7 @@ This works for early experimentation when the volume is low and the operators ar - What retention policy applies to traces? Indefinite retention supports audit requirements but increases storage cost and data sensitivity exposure. Time-bounded retention (e.g., 90 days) limits exposure but may lose traces needed for incident investigation. - How do we measure "is the system getting better"? What metrics constitute a meaningful quality signal for an autonomous software factory? Merge revert rate? Human override rate? Time-to-review? Cost per decision? Some composite score? The choice of metric shapes what gets optimized. - At what scale does a dedicated LLM observability platform justify its operational overhead (Postgres, ClickHouse, Redis, S3 for something like Langfuse)? Is there a threshold of agent activity below which structured logging suffices? -- How do we handle the bootstrapping problem — the factory needs observability to improve, but building the observability infrastructure is itself work that competes with building the factory? +- ~~How do we handle the bootstrapping problem — the factory needs observability to improve, but building the observability infrastructure is itself work that competes with building the factory?~~ Decided in [ADR 0050](../ADRs/0050-distributed-tracing-instrumentation.md): zero-configuration baseline (local JSONL + summary files) eliminates infrastructure requirements for initial observability; OTLP export adds backends when the org is ready. - Should observability data feed back into agent instructions automatically (e.g., auto-adjusting prompts when false positive rates exceed a threshold), or should it only inform human-driven instruction changes? Automatic feedback creates the risk of instruction oscillation; human-only feedback is slower but more controlled. - How do we build community dashboards that are useful to contributors with different levels of technical depth — from "is the agent doing a good job on my repo" to "show me the trace of this specific review"? - What is the cost of observability itself? Storing traces, running evaluators, maintaining dashboards — this has infrastructure cost. At what scale does it pay for itself in debugging time saved and quality improvement?