gen-ai: add judgment boundary attributes to evaluation result

[Nick-heo-eg]

Summary

This PR adds two minimal attributes to the existing gen_ai.evaluation.result event to enable standard telemetry recording of evaluation outcomes in GenAI pipelines:

• gen_ai.evaluation.outcome — the final outcome label assigned by the evaluator (e.g. pass, fail, allow, block)
• gen_ai.evaluation.multiple_outcomes — whether the evaluation assessed multiple outcome categories simultaneously

These attributes intentionally capture minimal evaluation metadata to remain vendor-neutral and avoid introducing lifecycle or decision-model semantics.

The design follows the direction discussed in open-telemetry/semantic-conventions-genai#72 and does not introduce new events or namespaces.

---

New Attributes

Attribute	Type	Stability	Description	Examples
gen_ai.evaluation.outcome	string	experimental	The evaluation outcome label assigned by the evaluator.	pass, fail, allow, block
gen_ai.evaluation.multiple_outcomes	boolean	experimental	Indicates whether the evaluation process assessed multiple outcome categories or labels.	true

Both attributes are added as Recommended to the gen_ai.evaluation.result event.

---

Motivation

GenAI evaluation pipelines (safety classifiers, quality scorers, guardrail systems) produce outcome labels as part of their standard output. Currently, the GenAI semantic conventions provide no standard attribute to record these outcome labels in telemetry.

Without a standardized attribute, each instrumentation library uses ad-hoc span attributes or event bodies, making cross-vendor analysis and alerting impossible.

These two attributes provide the minimal metadata needed to record evaluation outcomes in a vendor-neutral way, scoped to what is universally available across evaluation APIs and frameworks.

---

Relationship to Existing Evaluation Attributes

The gen_ai.evaluation namespace already defines the following attributes (stability: development):

Attribute	Purpose
gen_ai.evaluation.name	Name of the evaluation metric (e.g. Relevance, IntentResolution)
gen_ai.evaluation.score.value	Numeric score returned by the evaluator
gen_ai.evaluation.score.label	Human-readable interpretation of the score (evaluator-specific)
gen_ai.evaluation.explanation	Free-form explanation for the score

These are complementary, not overlapping:

• score.value / score.label capture how the evaluator scored the output — a numeric signal and its evaluator-specific label.
• outcome captures the final categorical decision — the action-oriented result of the evaluation (pass/fail/allow/block), which may derive from a score threshold, a classifier, or a multi-step pipeline.
• multiple_outcomes provides context when outcome is one of several categories evaluated simultaneously (e.g. a safety classifier that scores toxicity, bias, and PII at once).

In a typical LLM evaluation workflow:

evaluator runs → score.value=0.85, score.label="relevant"
                → outcome="pass"          ← threshold applied
                → multiple_outcomes=true  ← multiple categories were scored

The existing attributes describe evaluation measurement; the new attributes describe evaluation result selection.

---

Mapping to Existing APIs

API / Framework	Maps to gen_ai.evaluation.outcome	Maps to gen_ai.evaluation.multiple_outcomes
OpenAI Evals API (eval_run_result.result)	"pass" / "fail" per sample	true when multiple criteria are evaluated per sample
Anthropic model-graded eval (result field)	"pass" / "fail"	true for multi-metric evaluations
Azure AI Evaluation SDK (evaluator_result)	categorical label (e.g. "Very good", "Blocked")	true when composite evaluator runs
deepeval / ragas / RAGAS	per-metric pass/fail	true when test case checks multiple metrics
Custom safety classifiers	"allow" / "block"	true for multi-label classifiers

---

Example

Span attributes for a safety evaluation that assessed multiple categories:

{
  "name": "gen_ai.evaluation.result",
  "attributes": {
    "gen_ai.evaluation.name": "ContentSafety",
    "gen_ai.evaluation.score.value": 0.85,
    "gen_ai.evaluation.score.label": "pass",
    "gen_ai.evaluation.outcome": "pass",
    "gen_ai.evaluation.multiple_outcomes": true
  }
}

---

Related

• Issue: Proposal: GenAI semantic event for pre-execution judgment and negative proof semantic-conventions-genai#72
• GenAI events spec: docs/gen-ai/gen-ai-events.md
• GenAI registry: docs/registry/attributes/gen-ai.md

[Nick-heo-eg]

Note: This PR reintroduces the same changes as #3297, which was closed during repository cleanup.
The contents are unchanged; this PR exists to restore the proposal with a clean fork/PR state.

[Nick-heo-eg]

For additional background (non-normative, informational only), see:
https://github.com/Nick-heo-eg/decision-only-observability

This external reference does not propose new standards or semantic conventions,
and is intended only to clarify how existing OpenTelemetry concepts can be applied
to decision-only (non-executed) outcomes.

[github-actions]

This PR has been labeled as stale due to lack of activity. It will be automatically closed if there is no further activity over the next 7 days.

[Nick-heo-eg]

For clarity: the recent modeling adjustments only make existing evaluation outcomes queryable and consistent with attribute definitions, without introducing new lifecycle semantics or policy assumptions.

[Nick-heo-eg]

Revision Note

This revision narrows the proposal to two minimal, vendor-neutral evaluation metadata attributes.

It does not introduce lifecycle stages, decision models, or control semantics.
It only surfaces evaluation metadata already computed by implementations.

The goal is cross-implementation stability and improved queryability of evaluation results.

[Nick-heo-eg]

@lmolkova

Would this reduced scope align with the current GenAI semantic convention goals?

We’ve narrowed the PR to two minimal attributes focused strictly on execution outcome observability. It no longer introduces lifecycle stages, decision models, or policy semantics - only optional attributes to record execution outcomes.

Happy to adjust further if additional scope reduction would be helpful.

[github-actions]

This PR has been labeled as stale due to lack of activity. It will be automatically closed if there is no further activity over the next 7 days.

[Nick-heo-eg]

@lmolkova @trask

I pushed fixes after the previous CI failure
(docs regenerated with weaver + changelog added).

Could you please approve the workflow run again?

Thank you!

[trask]

hi @Nick-heo-eg!

can you update the PR description? currently it doesn't match the attributes added

can you also add links and explain how these two new attributes map to existing APIs? Thanks!

[Nick-heo-eg]

hi @Nick-heo-eg!

can you update the PR description? currently it doesn't match the attributes added

can you also add links and explain how these two new attributes map to existing APIs? Thanks!

Thanks for the review. I updated the PR description to match the current attributes and added links plus a mapping section explaining how the attributes relate to existing GenAI evaluation attributes and APIs.

[trask]

Thanks! can you also add links in the "Mapping to Existing APIs" tables wherever possible to make it easy for reviewers to verify?

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: claude / name: Claude (10d45b2, c016185, cdf6483)
• ✅ login: Nick-heo-eg / name: Nick Heo (02e92b7, 0f5f5f1, 10d45b2, 18cd219, 294379f, 2a0a9c3, 2e3e695, 3bd57d6, 4183440, 62563d3, c016185, cdf6483, e6eee35, f24f852, f2ea9d8)

[Nick-heo-eg]

/easycla

[Nick-heo-eg]

/easycla

[trask]

A couple of questions:

• How does gen_ai.evaluation.outcome differ from the existing score.label? They're the same type with the same example values (pass, fail) — when would an instrumentor set one but not the other?
• What's the concrete use case for gen_ai.evaluation.multiple_outcomes? It doesn't appear to map to a field in the listed APIs, and multiple gen_ai.evaluation.result events on the same span already signal that multiple evaluations happened.

[Nick-heo-eg]

A couple of questions:

• How does gen_ai.evaluation.outcome differ from the existing score.label? They're the same type with the same example values (pass, fail) — when would an instrumentor set one but not the other?

• What's the concrete use case for gen_ai.evaluation.multiple_outcomes? It doesn't appear to map to a field in the listed APIs, and multiple gen_ai.evaluation.result events on the same span already signal that multiple evaluations happened.

Thanks for the questions!

Regarding gen_ai.evaluation.outcome vs score.label:

score.label is the human-readable label associated with a numeric score.value. It represents the categorical interpretation of a score (for example, a score produced by an evaluation framework might map to labels such as "relevant" or "not_relevant"). In this sense, it is a companion attribute to score.value.

gen_ai.evaluation.outcome, on the other hand, represents the categorical result assigned by the evaluator independently of any numeric score. It is useful when an evaluation produces a direct categorical decision such as pass, fail, allow, or block, without necessarily producing a numeric score.

In practice, both attributes may be set on the same event when an evaluation framework produces both a numeric score (with score.value and score.label) and a separate categorical evaluation outcome. When only a categorical result exists without a numeric score, gen_ai.evaluation.outcome is the appropriate attribute.

Regarding gen_ai.evaluation.multiple_outcomes:

This attribute signals that the evaluation process assessed multiple outcome categories in a single evaluation run. For example, a safety classifier may simultaneously evaluate categories such as toxicity, bias, and PII exposure.

It allows telemetry consumers to understand that a single gen_ai.evaluation.result event represents a multi-category evaluation rather than a single-criterion result.

This is distinct from emitting multiple gen_ai.evaluation.result events on the same span, which would indicate multiple independent evaluation runs.

Also, this PR was closed due to a CI merge-base issue after a force push, so the same changes have been reopened in #3527 where CI can run with a fresh merge base.

[github-actions]

This PR contains changes to area(s) that do not have an active SIG/project and will be auto-closed:

• faas
• aws
• db
• gen-ai
• hardware

Such changes may be rejected or put on hold until a new SIG/project is established.

Please refer to the Semantic Convention Areas
document to see the current active SIGs and also to learn how to kick start a new one.