Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
06cb1ee
feat(eval): add eval measurements and EM-001 trace_fitness scorer
ascerra Aug 10, 2026
16e82de
refactor(eval): keep measurements tool-agnostic and document ownership
ascerra Aug 10, 2026
2add5e9
fix(eval): address review feedback on PR #6036
ascerra Aug 11, 2026
71e411b
fix(eval): restore with: blocks and rename ledger extension
ascerra Aug 11, 2026
930db1a
fix(eval): address review feedback and rebase onto main
ascerra Aug 12, 2026
615dd69
fix(eval): move measure before upload and raise test coverage
ascerra Aug 13, 2026
c8438d8
fix(eval): address review findings on measurements path
ascerra Aug 17, 2026
4ff9499
chore: merge origin/main into feat/eval-measurements
ascerra Aug 17, 2026
3d5aefc
fix(eval): address second-round measurement review
ascerra Aug 17, 2026
61b9bae
fix(eval): keep partial eval-measure rows on persist error
ascerra Aug 18, 2026
5a21884
fix(eval): address third-round measurement review findings
ascerra Aug 18, 2026
d37d045
chore: merge origin/main into feat/eval-measurements
ascerra Aug 19, 2026
1b87426
Merge remote-tracking branch 'origin/main' into feat/eval-measurements
ascerra Aug 19, 2026
8029af1
fix(eval): address Wayne Aug-19 measurement review
ascerra Aug 19, 2026
e3731af
chore: merge origin/main into feat/eval-measurements
ascerra Aug 19, 2026
cffdfc7
chore: merge origin/main into feat/eval-measurements
ascerra Aug 19, 2026
8596989
fix(eval): tighten output exclude and evening review follow-ups
ascerra Aug 20, 2026
cc5d4e4
chore: merge origin/main into feat/eval-measurements
ascerra Aug 20, 2026
b7a6591
chore: merge origin/main into feat/eval-measurements
ascerra Aug 20, 2026
8f3564d
fix(eval)!: trust default-branch measurement manifests in CI
ascerra Aug 20, 2026
903c0a4
fix(eval): resolve trusted measure path from fullsend-dir layout
ascerra Aug 20, 2026
30564ef
chore: merge origin/main into feat/eval-measurements
ascerra Aug 20, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 46 additions & 0 deletions action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -409,6 +409,52 @@ runs:
"${STATUS_FLAGS[@]+"${STATUS_FLAGS[@]}"}" \
"${MINT_FLAGS[@]+"${MINT_FLAGS[@]}"}"

# Eval measurements (fail-open): score run-telemetry.jsonl with the agents
# measurement manifest. Same job as fullsend run; never fails the agent.
# Scores always land in eval-measurements.jsonl (tool-agnostic artifact).
- name: Eval measurements
Comment thread
ascerra marked this conversation as resolved.
Resolved
if: always() && inputs.agent != '__install_only__'
continue-on-error: true
shell: bash
env:
AGENT: ${{ inputs.agent }}
FULLSEND_DIR: ${{ inputs.fullsend-dir }}
GH_TOKEN: ${{ inputs.github_token }}
Comment thread
ascerra marked this conversation as resolved.
Resolved
run: |
set -euo pipefail
FULLSEND_DIR="${FULLSEND_DIR:-${GITHUB_WORKSPACE}}"
TELEMETRY=$(find "${GITHUB_WORKSPACE}/output" -name run-telemetry.jsonl -print -quit 2>/dev/null || true)
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
if [[ -z "${TELEMETRY}" ]]; then
echo "No run-telemetry.jsonl under output/; skipping eval measurements"
exit 0
fi

REGISTRY=""
LOCAL_REG="${FULLSEND_DIR}/eval/measurements/${AGENT}.yaml"
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
if [[ -f "${LOCAL_REG}" ]]; then
REGISTRY="${LOCAL_REG}"
else
# Same pin as agents-repo harness fallback (config.DefaultUpstreamRef=v0).
Comment thread
ascerra marked this conversation as resolved.
Resolved
URL="https://raw.githubusercontent.com/fullsend-ai/agents/v0/eval/measurements/${AGENT}.yaml"
TMP="$(mktemp)"
CURL_ARGS=( -fsSL -o "${TMP}" )
if [[ -n "${GH_TOKEN:-}" ]]; then
CURL_ARGS+=( -H "Authorization: Bearer ${GH_TOKEN}" )
Comment thread
ascerra marked this conversation as resolved.
Resolved
fi
if curl "${CURL_ARGS[@]}" "${URL}"; then
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
REGISTRY="${TMP}"
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Resolved
else
printf 'No eval measurement manifest for %q at %s or %s; skipping\n' "${AGENT}" "${LOCAL_REG}" "${URL}"
rm -f "${TMP}"
exit 0
fi
fi

fullsend eval-measure \
Comment thread
ascerra marked this conversation as resolved.
Resolved
--telemetry "${TELEMETRY}" \
--registry "${REGISTRY}" \
--out-dir "$(dirname "${TELEMETRY}")"

- name: Upload fullsend artifacts
if: always() && inputs.agent != '__install_only__'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
Expand Down
1 change: 1 addition & 0 deletions docs/.vitepress/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -259,6 +259,7 @@ export default defineConfig({
{ text: "Standalone Mint", link: "/guides/infrastructure/standalone-mint" },
{ text: "Private Repositories", link: "/guides/infrastructure/private-repositories" },
{ text: "Tracing Reference", link: "/guides/infrastructure/distributed-tracing" },
{ text: "Eval Measurements", link: "/guides/infrastructure/eval-measurements" },
{ text: "Advanced Setup", link: "/guides/infrastructure/advanced-setup" },
{
text: "Layered Config Reference",
Expand Down
8 changes: 8 additions & 0 deletions docs/ADRs/0050-distributed-tracing-instrumentation.md
Original file line number Diff line number Diff line change
Expand Up @@ -155,3 +155,11 @@ and agent spans in `run-telemetry.jsonl`, which became the sole Level 1
artifact. OTLP export also changed from post-hoc directory upload to live
span export via the OTel SDK's batch processor. The core decision (three-level
opt-in, OTel-native, W3C propagation) is unchanged.

**2026-08-10 — Eval measurements ([ADR 0087](0087-eval-measurements-online-trace-scoring.md)):**
online scoring of wild-run traces always writes `eval-measurements.jsonl`
beside telemetry (tool-agnostic). Distinct from functional eval fixtures
([ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md)).

> **Planned:** portable remote score export follows the same OTLP
> configuration as this ADR — no vendor score adapters in core.
114 changes: 114 additions & 0 deletions docs/ADRs/0087-eval-measurements-online-trace-scoring.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
---
title: "87. Eval measurements as online trace scoring with portable export"
status: Accepted
relates_to:
- operational-observability
- testing-agents
topics:
- observability
- evaluation
- opentelemetry
---

# 87. Eval measurements as online trace scoring with portable export

Date: 2026-08-10

## Status

Accepted

## Context

Agent runs already emit OpenTelemetry traces as `run-telemetry.jsonl`, with
optional live OTLP export when `OTEL_EXPORTER_OTLP_*` is set
([ADR 0050](0050-distributed-tracing-instrumentation.md)). Separately,
[ADR 0051](0051-agent-eval-harness-for-test-infrastructure.md) owns the
**functional** eval harness: curated fixtures / scenarios in
`fullsend-ai/agents` `eval/<agent>/` that gate agent PRs. Those fixtures do
not score wild production runs.

Operators also need an **online / trend** layer on wild traces (completeness
first; quality signals later). Fullsend must stay **backend-agnostic**: orgs
already choose Phoenix, MLflow, Jaeger, or another OTLP collector for traces.
Baking a single product’s Assessments/Quality API into the core CLI or managed
workflows would force a tool decision on every install.

Adjacent telemetry proposals (not competing with this score path):

- **Level 3 content capture** ([#5947](https://github.com/fullsend-ai/fullsend/pull/5947)
— proposed ADRs 0084/0085): richer span content enables later scorers;
first ship reads Level 1/2 metadata in `run-telemetry.jsonl`. Measure CLI is
host-side after sandbox exit.
- **Span status from run outcome** ([#5944](https://github.com/fullsend-ai/fullsend/pull/5944)):
OTLP Status (and `fullsend.transcript_error`) become the reliable
success/failure signal. EM-001 only checks that `exit_code` is **present**
(fitness). Outcome scorers must key on Status, not `exit_code == 0`.
- **Observer / lessons → fixtures** ([#2423](https://github.com/fullsend-ai/fullsend/pull/2423)):
narrative analysis and golden-set promotion. This ADR is same-job
deterministic scoring on traces.
- **Harness snapshot / forge join keys** ([#5524](https://github.com/fullsend-ai/fullsend/pull/5524)
— proposed ADR 0075): sibling artifact for harness fingerprint and
Comment thread
ascerra marked this conversation as resolved.
Resolved
Comment thread
ascerra marked this conversation as resolved.
Resolved
forge/CI pointers beside telemetry. Complementary join/identity layer;
primary run facts belong on the OTEL trace (Level 1), while measurements
stay a derived sibling file.

## Options

1. **Local JSONL only** — portable offline artifact; no remote scores from
fullsend itself.
2. **Backend-native APIs in core** (e.g. one vendor’s Assessments API) —
couples every managed workflow to that product’s auth and schema.
3. **Local JSONL + same OTLP path as agent traces for remote** — scores travel
with the endpoint/headers orgs already configure for ADR 0050; no second
vendor stack in core.

## Decision

Introduce **eval measurements**: deterministic scorers that read
`run-telemetry.jsonl` after `fullsend run` in the **same** managed job
(`fullsend eval-measure` in `action.yml`), **fail-open**. Functional eval
scenarios remain ADR 0051 / `eval/<agent>/`; measurements never block
delivery.

Scores always land in a tool-agnostic `eval-measurements.jsonl` (plus a
Comment thread
ascerra marked this conversation as resolved.
Resolved
small idempotency ledger) next to `run-telemetry.jsonl`. Remote score export
will use the same `OTEL_EXPORTER_OTLP_*` configuration as ADR 0050 — no
vendor-specific score adapters in core. `fullsend` owns the parser, scorers,
CLI, and GHA step; `fullsend-ai/agents` owns per-agent measurement manifests
(`eval/measurements/<agent>.yaml`) that declare which scorers to enable.
Stock-agent defaults resolve from `agents@v0` at runtime; local files are for
Comment thread
ascerra marked this conversation as resolved.
Resolved
override, opt-out, or custom agents only.

The first scorer is `trace_fitness` (catalog id `em-001`) — span-tree and
attribute fitness so later scorers can trust the trace.

### Versioning (per measurement, not platform “v1”)

There is no product-wide “eval measurements v1” switch. “First ship” just
means only one scorer is enabled yet. Each manifest entry carries:

| Field | Meaning |
|---|---|
| `id` | Stable catalog id (`em-001`). New measurement concept → new id. |
| `scorer` | Go dispatch name (`trace_fitness`). |
| `version` | Integer **contract** version of that measurement’s checks / pass rule. |

Scores and the idempotency ledger key on `id@version` (e.g. `em-001@1`).
Bump `version` when pass/fail semantics change so trends do not mix eras.
Add a check that does not change the pass definition → same version is fine.
Entirely new signal → new `em-NNN` (and usually a new `scorer` string).

## Consequences
Comment thread
ascerra marked this conversation as resolved.
Resolved

- Every measured run produces a reviewable, backend-agnostic score file beside
telemetry; missing manifests skip cleanly and measure failure never fails
the agent job.
- Core stays tool-agnostic: no product-specific score env vars in managed
workflows; remote scores follow OTEL when that path lands.
- Functional scenarios (gate) and eval measurements (trend) stay separate;
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Resolved
retro can recommend either a manifest scorer or a scenario fixture.
- Richer telemetry (Level 3 / Status fixes) expands what scorers *can* assert;
it does not replace this same-job path.
- Per-measurement versioning (`id@version`) lets pass/fail semantics evolve
without mixing trend eras.
4 changes: 4 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -287,11 +287,15 @@ Observability is a cross-cutting concern that touches every other component. Eac
- JSONL reasoning trace exposure: raw JSONL conversation transcripts are extracted from sandboxes and stored with owner-scoped access. Credential scanning acts as an invariant check on [ADR 0017](ADRs/0017-credential-isolation-for-sandboxed-agents.md)'s isolation model. Agents handling data from protected sources beyond the target repo can opt in to JSONL suppression via configuration ([ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md)).
- Event-driven stage dispatch remains traceable end-to-end in the GitHub Actions UI by using synchronous `workflow_call` dispatch (see [ADR 0041](ADRs/0041-synchronous-workflow-call-event-dispatch.md)).
- Distributed tracing: framework-native OpenTelemetry instrumentation with zero-configuration baseline. Every run produces `run-telemetry.jsonl` locally; optional live OTLP export to any compatible backend. W3C trace context propagation links multi-agent pipelines into unified traces. OTEL GenAI semantic conventions enable LLM-aware backends ([ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md)).
- Eval measurements: fail-open same-job scoring of wild-run traces into `eval-measurements.jsonl` beside telemetry; scores always land in a local, tool-agnostic JSONL artifact ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). See [Eval Measurements](guides/infrastructure/eval-measurements.md).

> **Planned:** portable remote score export via the same OTLP configuration as agent traces ([ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md)). Not yet implemented.

**Open questions:**

- What signals matter most — cost, latency, token usage, action logs, decision traces, or something else?
- ~~How do we balance detailed tracing (useful for debugging) with the volume of data agents will produce?~~ Decided in [ADR 0050](ADRs/0050-distributed-tracing-instrumentation.md): instrument all lifecycle steps comprehensively; volume is managed by backends not by suppressing data at the source.
- ~~How do we score wild agent traces for trends without a second export stack?~~ Decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md): eval measurements write local JSONL beside telemetry; portable remote export uses the same OTLP config as traces (planned); local eval-measurements.jsonl is always written.
Comment thread
ascerra marked this conversation as resolved.
Resolved
- What is the retention and access model for agent logs? Who can see what? (JSONL trace access model decided in [ADR 0021](ADRs/0021-jsonl-reasoning-trace-exposure.md); retention policy and broader log access remain open.)
- How does observability interact with the security requirement that "every action is logged, attributable, and reviewable"? (See [security-threat-model.md](problems/security-threat-model.md).)
- Is there a real-time monitoring requirement (agent is stuck, agent is behaving anomalously), or is observability primarily forensic?
Expand Down
8 changes: 4 additions & 4 deletions docs/glossary.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,13 +88,13 @@ See [autonomy-spectrum.md](problems/autonomy-spectrum.md) and [agent-architectur

### Eval Measurement

A score, judge, or metric applied to agent (or agent-chain) behavior — for example cost per run, whether the code agent later passes review, or whether a review agent recommends merge and a human still intervenes. Measurements are not the inputs under test; they are what you score. The same measurement can be applied to curated [eval scenarios](#eval-scenario) or to live ("wild") production traffic, at agent scope or across the platform chain. Prefer this term (or synonyms *eval score* / *eval judge*) over the bare word "evals," which is ambiguous with [eval scenarios](#eval-scenario).
See [testing-agents.md](problems/testing-agents.md) and [Observability](#observability).
A score, judge, or metric applied to agent (or agent-chain) behavior — for example cost per run, whether the code agent later passes review, or whether a review agent recommends merge and a human still intervenes. Measurements are not the inputs under test; they are what you score. Online / trend scoring of wild-run traces is decided in [ADR 0087](ADRs/0087-eval-measurements-online-trace-scoring.md) (`fullsend eval-measure`, `eval-measurements.jsonl`). The same measurement *idea* can also be applied to curated [eval scenarios](#eval-scenario), but those PR-gate fixtures are a separate path ([ADR 0051](ADRs/0051-agent-eval-harness-for-test-infrastructure.md)). Prefer this term (or synonyms *eval score* / *eval judge*) over the bare word "evals," which is ambiguous with [eval scenarios](#eval-scenario).
See [Eval Measurements](guides/infrastructure/eval-measurements.md), [testing-agents.md](problems/testing-agents.md), and [Observability](#observability).

### Eval Scenario

A fixed, reproducible test case — a concrete input with an expected outcome that you re-run when an agent changes. Example: triage is presented with an issue asking to add a cheeseburger to the README and is expected to reject and close it. Scenarios are maintained like tests: if intentional agent behavior changes, update the scenario expectations. They answer "did this change make the agent better or worse on known cases?" and can later grow by promoting interesting production cases from telemetry into the curated set. Distinct from [eval measurements](#eval-measurement) (the scores/judges applied to a scenario or to wild traffic). Prefer this term over the bare word "evals."
See [testing-agents.md](problems/testing-agents.md) (golden-set evaluation).
A fixed, reproducible test case — a concrete input with an expected outcome that you re-run when an agent changes. Example: triage is presented with an issue asking to add a cheeseburger to the README and is expected to reject and close it. Scenarios are maintained like tests: if intentional agent behavior changes, update the scenario expectations. They answer "did this change make the agent better or worse on known cases?" and can later grow by promoting interesting production cases from telemetry into the curated set. Distinct from [eval measurements](#eval-measurement) (online/trend scores on wild traces, or judges applied to a scenario). Prefer this term over the bare word "evals." Also called a *functional eval fixture* in agent CI.
See [ADR 0051](ADRs/0051-agent-eval-harness-for-test-infrastructure.md) and [testing-agents.md](problems/testing-agents.md) (golden-set evaluation).

### Evergreen

Expand Down
1 change: 1 addition & 0 deletions docs/guides/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ Advanced guides for platform operators who deploy and manage the GCP-side infras
- [Infrastructure reference](infrastructure/infrastructure-reference.md) — Token mint, WIF, and secrets deployment details
- [Enabling fullsend on private repositories](infrastructure/private-repositories.md) — Additional guardrails and configuration for private repos
- [Tracing reference](infrastructure/distributed-tracing.md) — Telemetry levels, environment variables, span hierarchy, and attributes
- [Eval measurements](infrastructure/eval-measurements.md) — Online trace scoring with `eval-measurements.jsonl` and measurement manifests

## User guides

Expand Down
4 changes: 4 additions & 0 deletions docs/guides/dev/cli-internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -109,6 +109,10 @@ fullsend
│ └── url # Validate URLs against SSRF attacks
├── post-review # Post PR review comments to GitHub
├── post-comment # Post issue/PR comments to GitHub
├── eval-measure # Score wild-run traces (eval measurements)
Comment thread
ascerra marked this conversation as resolved.
Resolved
│ ├── --telemetry <path> # Path to run-telemetry.jsonl (required)
│ ├── --registry <path> # Agents measurement manifest YAML (required)
│ └── --out-dir <path> # Output dir (default: telemetry directory)
└── reconcile-status # Finalize orphaned status comments
├── --repo <owner/repo> # Repository in owner/repo format
├── --number <int> # Issue/PR number
Expand Down
12 changes: 12 additions & 0 deletions docs/guides/infrastructure/distributed-tracing.md
Original file line number Diff line number Diff line change
Expand Up @@ -244,7 +244,19 @@ Any variable and secret names work here; the values reach the exporter
as-is. Consult your backend's documentation for the endpoint URL and
authentication mechanism.

## Eval measurements

After each managed agent run, `fullsend eval-measure` scores
`run-telemetry.jsonl` in the same job (fail-open). Scores always land in
`eval-measurements.jsonl` beside telemetry (tool-agnostic artifact). Portable
remote export will reuse the same `OTEL_EXPORTER_OTLP_*` configuration as
agent traces when implemented. Measurements read the Level 1/2 metadata
contract of `run-telemetry.jsonl` (not Level 3 content). See
[Eval Measurements](./eval-measurements.md) and
[ADR 0087](../../ADRs/0087-eval-measurements-online-trace-scoring.md).

## See also

- [How To Emit Traces](../user/how-to-emit-traces.md): step-by-step setup guide
- [Tracing Development Guide](../dev/tracing.md): implementation details for contributors
- [Eval Measurements](./eval-measurements.md): online scoring of wild-run traces
Loading
Loading