feat(telemetry): forward OTEL env vars to all managed workflow agents

[rh-hemartin]

Summary

• Setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, OTEL_EXPORTER_OTLP_TRACES_HEADERS, and OTEL_RESOURCE_ATTRIBUTES now produces traces for all agents on your system (triage, code, review, fix, retro, prioritize, harness), including the ones dispatched via triggers. Previously only triage exported traces.
• Updated the distributed tracing guide to remove the "triage only" caveat.
• Expanded TestOTELHeadersSecretThreading to cover all stages and added the OTEL secret to TestReusableWorkflowsShareCommonInputs so cross-stage drift is caught automatically.

Test plan

• All pre-commit hooks pass
• TestOTELHeadersSecretThreading passes with 14 subcases (all stages, both install modes)
• TestReusableWorkflowsShareCommonInputs passes with OTEL secret in commonSecrets
• TestWorkflowCallInputAlignment passes (scaffold callers forward the new optional secret)

Closes #5528

🤖 Generated with Claude Code

[qodo-code-review]

PR Summary by Qodo

Forward OTEL env/secret to all managed workflow agent stages

✨ Enhancement ⚙️ Configuration changes 🧪 Tests 📝 Documentation 🕐 20-40 Minutes

AI Description

• Forward OTEL endpoint/headers/resource attributes to every managed workflow agent stage.
• Update scaffolded caller workflows to pass the OTEL headers secret consistently.
• Expand tests and docs to enforce and document all-stages trace exporting.

Diagram

graph TD
  cfg{{"GitHub vars/secrets (OTEL)"}} --> dispatch["reusable-dispatch.yml"] --> agent["Agent step"] --> backend[("OTLP backend")]
  cfg --> stage["reusable-{stage}.yml"] --> agent
  caller["Scaffold caller workflows"] --> dispatch
  caller --> stage
  subgraph Legend
    direction LR
    _wf["Workflow"] ~~~ _cfg{{"Vars/Secrets"}} ~~~ _db[("Backend")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Centralize OTEL env injection via a shared composite action

• ➕ Reduces repeated env blocks across many workflow files
• ➕ Makes future OTEL additions (logs/metrics) a single change
• ➖ Adds another abstraction layer and action versioning concerns
• ➖ Harder to reason about exactly which jobs get which env at review time

2. Have the agent binary read OTEL config from a standard file/parameter

• ➕ Avoids threading multiple workflow env vars/secrets through YAML
• ➕ Could unify behavior across GitHub Actions and other runners
• ➖ Requires product/runtime changes beyond workflow updates
• ➖ Still needs secure delivery of the headers secret to the runtime

Recommendation: Keep the PR’s explicit per-workflow env/secret forwarding. It is the lowest-risk change that preserves GitHub Actions security boundaries (headers as a secret) while ensuring every stage exports traces. A DRY refactor (composite action) could be a follow-up once the end-to-end behavior is stable.

Files changed (13) +76 / -20

Tests (1) +18 / -8

workflow_call_alignment_test.goExtend workflow alignment tests for OTEL secret across all stages +18/-8

Extend workflow alignment tests for OTEL secret across all stages

• Adds the OTEL headers secret to the commonSecrets set and expands TestOTELHeadersSecretThreading to validate forwarding across both install-mode chains and every reusable stage workflow.

internal/scaffold/workflow_call_alignment_test.go

Documentation (1) +7 / -9

distributed-tracing.mdDocument OTEL forwarding for all managed agent stages +7/-9

Document OTEL forwarding for all managed agent stages

• Removes the previous 'triage only' limitation and updates guidance to reflect that all stages forward OTEL configuration, including updated notes for older scaffolds needing secret forwarding.

docs/guides/infrastructure/distributed-tracing.md

Other (11) +51 / -3

reusable-code.ymlInject OTEL env/secret into the code agent stage +5/-0

Inject OTEL env/secret into the code agent stage

• Adds an optional OTEL headers input and forwards OTEL endpoint/headers/resource attributes into the code agent job environment so code runs can export traces.

.github/workflows/reusable-code.yml

reusable-dispatch.ymlForward OTEL env/secret to all dispatched agent jobs +21/-3

Forward OTEL env/secret to all dispatched agent jobs

• Updates the OTEL headers input description to reflect all stages, and injects OTEL endpoint/headers/resource attributes into each agent job env (code/review/fix/retro/prioritize and matrix-dispatched agents).

.github/workflows/reusable-dispatch.yml

reusable-fix.ymlInject OTEL env/secret into the fix agent stage +5/-0

Inject OTEL env/secret into the fix agent stage

• Adds an optional OTEL headers input and forwards OTEL endpoint/headers/resource attributes into the fix agent job environment for trace export.

.github/workflows/reusable-fix.yml

reusable-prioritize.ymlInject OTEL env/secret into the prioritize agent stage +5/-0

Inject OTEL env/secret into the prioritize agent stage

• Adds an optional OTEL headers input and forwards OTEL endpoint/headers/resource attributes into the prioritize agent job environment.

.github/workflows/reusable-prioritize.yml

reusable-retro.ymlInject OTEL env/secret into the retro agent stage +5/-0

Inject OTEL env/secret into the retro agent stage

• Adds an optional OTEL headers input and forwards OTEL endpoint/headers/resource attributes into the retro agent job environment.

.github/workflows/reusable-retro.yml

reusable-review.ymlInject OTEL env/secret into the review agent stage +5/-0

Inject OTEL env/secret into the review agent stage

• Adds an optional OTEL headers input and forwards OTEL endpoint/headers/resource attributes into the review agent job environment.

.github/workflows/reusable-review.yml

code.ymlForward OTEL headers secret from scaffolded code caller +1/-0

Forward OTEL headers secret from scaffolded code caller

• Adds OTEL_EXPORTER_OTLP_TRACES_HEADERS to the secrets passed into the reusable code workflow so authenticated exporters work without manual edits.

internal/scaffold/fullsend-repo/.github/workflows/code.yml

fix.ymlForward OTEL headers secret from scaffolded fix caller +1/-0

Forward OTEL headers secret from scaffolded fix caller

• Adds OTEL_EXPORTER_OTLP_TRACES_HEADERS to the secrets passed into the reusable fix workflow.

internal/scaffold/fullsend-repo/.github/workflows/fix.yml

prioritize.ymlForward OTEL headers secret from scaffolded prioritize caller +1/-0

Forward OTEL headers secret from scaffolded prioritize caller

• Adds OTEL_EXPORTER_OTLP_TRACES_HEADERS to the secrets passed into the reusable prioritize workflow.

internal/scaffold/fullsend-repo/.github/workflows/prioritize.yml

retro.ymlForward OTEL headers secret from scaffolded retro caller +1/-0

Forward OTEL headers secret from scaffolded retro caller

• Adds OTEL_EXPORTER_OTLP_TRACES_HEADERS to the secrets passed into the reusable retro workflow.

internal/scaffold/fullsend-repo/.github/workflows/retro.yml

review.ymlForward OTEL headers secret from scaffolded review caller +1/-0

Forward OTEL headers secret from scaffolded review caller

• Adds OTEL_EXPORTER_OTLP_TRACES_HEADERS to the secrets passed into the reusable review workflow.

internal/scaffold/fullsend-repo/.github/workflows/review.yml

[fullsend-ai-review]

🤖 Finished Review · ✅ Success · Started 2:11 PM UTC · Completed 2:28 PM UTC
Commit: f303c40 · View workflow run →

[github-actions]

Site preview

Preview: https://e7deb826-site.fullsend-ai.workers.dev

Commit: f303c4006fa765ac653de52d768d269082e9761d

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 61 rules

✅ Skills: writing-user-docs, writing-adrs

1. Tracing guide in wrong directory ✗ Dismissed 📜 Skill insight ⌂ Architecture

Description

docs/guides/infrastructure/distributed-tracing.md is a guide under docs/guides/ but is not
placed in either docs/guides/admin/ or docs/guides/user/ as required. This breaks the
documentation guide directory/audience policy and makes the guide harder to discover and maintain
consistently.

Code

docs/guides/infrastructure/distributed-tracing.md[R205-210]

+All agent stages (triage, code, review, fix, retro, prioritize, harness)
+forward OTEL configuration. To enable export, set on the org (or repo)
+that hosts the fullsend caller workflows:

1. Actions **variable** `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` — the backend's
   full traces URL (e.g. `https://mlflow.example.com/v1/traces`).

Evidence

The compliance rule requires that guide files under docs/guides/ be placed under either admin/
or user/. This PR modifies a guide located at docs/guides/infrastructure/distributed-tracing.md,
demonstrating it remains outside the allowed subdirectories.

docs/guides/infrastructure/distributed-tracing.md[205-221]
Skill: writing-user-docs

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
A documentation guide exists under `docs/guides/` but is stored in `docs/guides/infrastructure/` instead of the required `docs/guides/admin/` or `docs/guides/user/` subdirectories.

## Issue Context
Compliance requires every guide under `docs/guides/` to live in either the admin or user directory and to target a single audience.

## Fix Focus Areas
- docs/guides/infrastructure/distributed-tracing.md[205-221]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[waynesun09]

3-agent review squad pass (2 Claude-based, 1 Grok-based) — no Critical/High findings. The secret-threading chain (OTEL_EXPORTER_OTLP_TRACES_HEADERS) was verified complete end-to-end for both install-mode chains across all 6 newly-covered stages, with no logging exposure, injection path, or undeclared-secret gap.

One MEDIUM was raised — the updated doc line presents "harness" OTEL coverage as available across all install modes, but harness-run/harness-dispatch currently only exist for the per-repo install path via reusable-dispatch.yml. Non-blocking given per-org mode is being deprecated, so this scoping nuance won't matter going forward.

Two LOW nits (a couple of test blind spots already closed by sibling tests, and a missing description: field on 5 new secret declarations for consistency with reusable-triage.yml) are cosmetic and not blocking either.

Approving.

[fullsend-ai-review]

Review

Findings

Medium

• [protected-path] .github/workflows/ — This PR modifies 6 files under the .github/ protected path: reusable-code.yml, reusable-dispatch.yml, reusable-fix.yml, reusable-prioritize.yml, reusable-retro.yml, reusable-review.yml. The change is well-justified by issue Forward OTEL env vars to all managed workflow agents, not just triage #5528 (forwarding OTEL env vars to all managed workflow agents) and follows the established pattern from reusable-triage.yml. Human approval is always required for protected-path changes, regardless of context.

The code change itself is clean across all review dimensions. The three OTEL env vars are correctly threaded through all agent steps using the same vars.*/secrets.* pattern as the existing triage stage. Secret declarations use required: false consistently. Tests are expanded to cover all 14 workflow paths across both installation modes. Documentation accurately reflects the new all-agent forwarding behavior.

---

Labels: PR modifies CI workflow files under .github/workflows/, dispatch-related reusable workflows, scaffold templates, and infrastructure documentation

[fullsend-ai-retro]

🤖 Finished Retro · ✅ Success · Started 3:07 PM UTC · Completed 3:20 PM UTC
Commit: f303c40 · View workflow run →

[fullsend-ai-retro]

Retro: PR #5531 — Forward OTEL env vars to all managed workflow agents

Timeline

Time (UTC)	Event
12:49	Issue #5528 filed by rh-hemartin
12:51–12:55	Triage agent ran successfully, confirmed root cause, applied labels, recommended fix
13:04	rh-hemartin invoked /fs-code
13:05–13:17	Code agent ran (~12 min, $4.97), produced valid 12-file patch, but push rejected — GitHub App token lacked workflows permission
~13:17–14:09	rh-hemartin manually created PR with 13 files (same 12 + test updates)
14:10–14:28	Review agent ran (5 sub-agents + challenger, ~18 min), found code clean, flagged protected-path MEDIUM
14:25	Human reviewer waynesun09 approved with detailed 3-agent squad analysis
15:04	PR merged

Assessment

This workflow went well overall. The PR was a mechanical propagation of three OTEL env vars across 13 workflow and scaffold files — straightforward but security-sensitive (.github/workflows/ changes). The review agent correctly identified the code as clean across all dimensions and appropriately required human approval for the protected-path changes. The human reviewer confirmed the assessment and added minor non-blocking findings. Zero rework iterations.

The one pain point was the code agent failure: $4.97 and ~12 minutes wasted producing a valid patch that could never be pushed. The post-code script misclassified the GitHub permission rejection as a non-fast-forward error and retried with --force-with-lease, which also failed.

Evidence for existing open issues (no new proposals needed)

• Code agent should treat .github/workflows/ as unpushable when coder token lacks workflows permission #3627 and Triage should detect likely GHA workflow modifications and skip automatic code agent agents#325: This retro provides another concrete data point — the code agent spent $4.97 and ~12 minutes on issue Forward OTEL env vars to all managed workflow agents, not just triage #5528, producing correct changes to 12 files (6 reusable workflows, 5 scaffold templates, 1 doc), all of which were unpushable due to the missing workflows permission. The triage agent's recommendation correctly identified the fix needed but did not flag that it required workflow file modifications the code agent cannot perform.
• Post-script should detect GitHub App permission rejections and report the actual cause instead of retrying #3628 and Post-script push error handler should detect workflow-permission rejections before retrying agents#139: The post-code push error handler treated the permission rejection as a non-fast-forward error and retried with --force-with-lease. The actual error was refusing to allow a GitHub App to create or update workflow ... without workflows permission — a fundamentally different failure mode that should be detected before retrying.
• Add CI workflow correctness checks to review agent sub-agents agents#394: The human reviewer (waynesun09) caught two LOW nits the review agent missed — (1) missing description: field on 5 new secret declarations in the reusable workflows, inconsistent with the existing reusable-triage.yml pattern, and (2) test blind spots already covered by sibling tests. Both are CI workflow correctness checks that fall within the scope of agents#394.
• Review agent should check config file changes for internal consistency with existing patterns in the same file #4183: The missing description: field finding is also an internal consistency gap — new entries in sibling config files diverging from the established pattern in an existing file.

Autonomy readiness

The review agent performed well on this mechanical cross-file change. Its assessment (code clean, protected-path requires human approval) was correct. The human reviewer's additions were all non-blocking: one MEDIUM about doc scoping nuance (harness OTEL coverage claim across install modes, non-blocking since per-org mode is deprecated), and two cosmetic LOWs. The human approval was warranted by the protected-path nature of the changes — no autonomy expansion is indicated for this class of change.