Skip to content
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
264 changes: 264 additions & 0 deletions .github/skills/update-otel-genai-conventions/SKILL.md

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
# Change Classification

Taxonomy for classifying gen-ai changes from semantic-conventions releases. Use this to assess each change's impact on dotnet/extensions.

## Classification Categories

### 🟢 No Action Required

| Type | Description | Example |
|------|-------------|---------|
| **N/A — No client exists** | Change affects a capability we don't implement (e.g. `retrieval`, `memory`) | `gen_ai.retrieval.*` attributes |
| **Already implemented** | Change was already implemented in a prior PR | A change that was part of an earlier draft spec we adopted |
| **Server-side only** | Change affects server/provider-side instrumentation, not client-side | Server span attributes |
| **Documentation only** | Clarification of existing semantics with no behavioral change | Rewording of attribute descriptions |

### 🟡 Minor Action Required

| Type | Description | Action |
|------|-------------|--------|
| **New constant (not emitted)** | New attribute defined but optional or not yet applicable | Add constant to `OpenTelemetryConsts.cs`, skip emission |
| **Version bump** | Convention version number changed | Update `v1.XX` in doc comments across all OpenTelemetry* files |
| **Stability promotion** | Attribute moved from experimental to stable | Usually no code change; note in audit table |

### 🔴 Code Change Required

| Type | Description | Action |
|------|-------------|--------|
| **New required attribute** | New attribute that should be emitted | Add constant, add emission code, add test assertion |
| **New metric** | New metric instrument defined | Add metric definition, emission, test |
| **Attribute rename** | Existing attribute renamed | Update constant value, verify backward compatibility |
| **New event** | New log/span event defined | Add event via `ILogger` + `[LoggerMessage]`, add test |
| **Behavioral change** | Change in how existing attributes are populated | Modify emission logic, update test expectations |
| **New operation name** | New `gen_ai.operation.name` value | Add detection logic, tests |
| **Schema change** | Change to JSON schema for serialized content (e.g. tool definitions) | Update serialization classes, `[JsonSerializable]` registration |

## Indicator Mapping

Use these indicators consistently in audit reports, implementation summaries, and PR descriptions:

| Indicator | Category | Meaning |
|---|---|---|
| 🟢 | No action required | No compensating code change is needed; explain why. |
| 🟡 | Minor action required | Small metadata, constant-only, stability, or version-reference update. |
| 🔴 | Code change required | Runtime behavior, emission logic, metrics, events, serialization, or tests must change. |

## Impact Assessment Heuristic

For each gen-ai change in a release:

1. **Does it affect a capability we instrument?** Check the [file inventory](file-inventory.md) for matching client types.
- No → classify as "N/A — No client exists"
2. **Is it already implemented?** Search `OpenTelemetryConsts.cs` for the attribute name.
- Yes → classify as "Already implemented"
3. **Is it client-side or server-side?** Check the semantic convention's `span_kind` or context.
- Server-side only → classify as "Server-side only"
4. **What kind of change is it?** Match to the categories above.
5. **How many files need modification?** Count affected files from the file inventory.
- 1–2 files → Low complexity
- 3–5 files → Medium complexity
- 6+ files → High complexity (likely involves shared code or cross-cutting concern)

## Audit Table Format

When presenting the analysis, use this table format:

```markdown
| Semantic Convention Change | Upstream PR | Classification | Action Required | Complexity |
|---|---|---|---|---|
| `gen_ai.new.attribute` | [#1234](link) | New required attribute | Add constant + emission + test | Low |
| `retrieval` operation | [#5678](link) | N/A — No client | None | — |
| Version reference | — | Version bump | Update doc comments | Low |
```

## PR Description Table Format

When preparing a PR description, adapt the audit table into a concise reviewer-facing table grouped or sorted by semantic-conventions version. Include every analyzed gen-ai change, not just changes that required code edits.

```markdown
| Version | Indicator | Semantic-conventions change | Classification | Compensating change / rationale |
|---|:---:|---|---|---|
| v1.XX | 🔴 | `gen_ai.new.attribute` added | New required attribute | Added constant, emission, and tests in `{files}`. |
| v1.XX | 🟡 | Version reference update | Version bump | Updated OpenTelemetry* doc comments to v1.XX. |
| v1.XX | 🟢 | Provider server span clarified | Server-side only | No client-side instrumentation change needed. |
```

The final column should either describe the compensating change made or explain why no code change was made, such as "already implemented", "no local source exists", "no client exists", "server-side only", or "documentation-only clarification".
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
# File Inventory

Files that are typically inspected and/or modified when updating OpenTelemetry gen-ai semantic conventions.

## Constants

| File | Purpose |
|------|---------|
| `src/Libraries/Microsoft.Extensions.AI/OpenTelemetryConsts.cs` | All OTel attribute and metric name constants. New attributes/metrics always get constants added here first. |

## Instrumentation Clients

These files contain the actual telemetry emission logic. Each wraps a different AI capability with OTel spans, metrics, and logs.

| File | Capability | Key Sections |
|------|-----------|--------------|
| `src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryChatClient.cs` | Chat completion | Activity creation, attribute emission, message serialization, streaming support, metrics |
| `src/Libraries/Microsoft.Extensions.AI/ChatCompletion/OpenTelemetryImageGenerator.cs` | Image generation | Activity creation, attribute emission |
| `src/Libraries/Microsoft.Extensions.AI/Embeddings/OpenTelemetryEmbeddingGenerator.cs` | Embeddings | Activity creation, attribute emission |
| `src/Libraries/Microsoft.Extensions.AI/SpeechToText/OpenTelemetrySpeechToTextClient.cs` | Speech-to-text | Activity creation, attribute emission |
| `src/Libraries/Microsoft.Extensions.AI/TextToSpeech/OpenTelemetryTextToSpeechClient.cs` | Text-to-speech | Activity creation, attribute emission |
| `src/Libraries/Microsoft.Extensions.AI/Realtime/OpenTelemetryRealtimeClientSession.cs` | Realtime sessions | Activity creation, attribute emission |
| `src/Libraries/Microsoft.Extensions.AI/Realtime/OpenTelemetryRealtimeClient.cs` | Realtime client wrapper | Delegates to session |
| `src/Libraries/Microsoft.Extensions.AI/Files/OpenTelemetryHostedFileClient.cs` | Hosted file management | Activity creation, attribute emission |

## Function Invocation / Tool Orchestration

These files handle `execute_tool`, `invoke_agent`, and `invoke_workflow` spans:

| File | Purpose |
|------|---------|
| `src/Libraries/Microsoft.Extensions.AI/ChatCompletion/FunctionInvokingChatClient.cs` | Chat-based tool orchestration |
| `src/Libraries/Microsoft.Extensions.AI/Realtime/FunctionInvokingRealtimeClientSession.cs` | Realtime tool orchestration |
| `src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationProcessor.cs` | Shared function invocation logic (used by both chat and realtime) |
| `src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationHelpers.cs` | Shared function invocation helpers |
| `src/Libraries/Microsoft.Extensions.AI/Common/FunctionInvocationLogger.cs` | Shared function invocation logging |

## Shared Code

| File | Purpose |
|------|---------|
| `src/Libraries/Microsoft.Extensions.AI/Common/OpenTelemetryLog.cs` | Shared `[LoggerMessage]` definitions for OTel events (e.g. exception recording) |
| `src/Libraries/Microsoft.Extensions.AI/TelemetryHelpers.cs` | Shared telemetry helper methods (at library root, not Common/) |

## Tests

| File | Tests For |
|------|----------|
| `test/Libraries/Microsoft.Extensions.AI.Tests/ChatCompletion/OpenTelemetryChatClientTests.cs` | Chat client telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/Image/OpenTelemetryImageGeneratorTests.cs` | Image generator telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/Embeddings/OpenTelemetryEmbeddingGeneratorTests.cs` | Embedding generator telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/SpeechToText/OpenTelemetrySpeechToTextClientTests.cs` | Speech-to-text telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/TextToSpeech/OpenTelemetryTextToSpeechClientTests.cs` | Text-to-speech telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/Realtime/OpenTelemetryRealtimeClientTests.cs` | Realtime session telemetry |
| `test/Libraries/Microsoft.Extensions.AI.Tests/Files/OpenTelemetryHostedFileClientTests.cs` | Hosted file client telemetry |

To discover any additional test files: `dir test\Libraries\Microsoft.Extensions.AI.Tests\ -Recurse -Filter "OpenTelemetry*Tests.cs"`

## Version References

The semantic conventions version is referenced in a doc comment in specific OpenTelemetry* instrumentation client files. When bumping the version, update all files that match the grep below — not all OpenTelemetry* files contain the version reference.

The reference looks like:

```csharp
/// This class provides an implementation of the Semantic Conventions for Generative AI systems v1.XX,
/// defined at <see href="https://opentelemetry.io/docs/specs/semconv/gen-ai/" />.
```

Find all occurrences with:
```
grep -rn "Semantic Conventions for Generative AI systems v" src/Libraries/Microsoft.Extensions.AI/
```
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
# Historical Releases

Mapping of OpenTelemetry semantic-conventions releases with gen-ai changes to dotnet/extensions PRs.

> **Note**: This file is a point-in-time reference and is not intended to be kept up to date with every new release. It provides context for how past convention updates were handled. For the latest release history, consult the [semantic-conventions releases page](https://github.com/open-telemetry/semantic-conventions/releases) and search the dotnet/extensions PR history.

## Release History

### Pre-v1.31 (Pre-release Era)

| Convention Version | dotnet/extensions PR | Description |
|-------------------|---------------------|-------------|
| Initial implementation | [#5532](https://github.com/dotnet/extensions/pull/5532) | Initial OpenTelemetry gen-ai instrumentation |
| v1.29 draft | [#5712](https://github.com/dotnet/extensions/pull/5712) | Align with v1.29 draft conventions |
| v1.30 | [#5815](https://github.com/dotnet/extensions/pull/5815) | Update to v1.30 conventions |

### v1.31–v1.40 (Stable Release Era)

| Convention Version | dotnet/extensions PR | Description |
|-------------------|---------------------|-------------|
| v1.31 | [#6073](https://github.com/dotnet/extensions/pull/6073) | Update to v1.31 conventions |
| v1.34 | [#6466](https://github.com/dotnet/extensions/pull/6466) | Update to v1.34 conventions |
| v1.35 | [#6557](https://github.com/dotnet/extensions/pull/6557) | Update to v1.35 conventions |
| v1.36 | [#6579](https://github.com/dotnet/extensions/pull/6579) | Bump version reference to v1.36 (CCA) |
| v1.37 | [#6767](https://github.com/dotnet/extensions/pull/6767) | Update to v1.37 conventions |
| v1.38 | [#6829](https://github.com/dotnet/extensions/pull/6829) | Update to v1.38 conventions |
| v1.38 update | [#6981](https://github.com/dotnet/extensions/pull/6981) | Additional v1.38 changes |
| v1.39 | [#7274](https://github.com/dotnet/extensions/pull/7274) | Bump version reference to v1.39 (CCA) |
| v1.40 | [#7322](https://github.com/dotnet/extensions/pull/7322) | Update to v1.40 conventions (CCA audit) |

### Feature-Specific PRs (v1.38–v1.40 era)

These PRs implemented specific gen-ai convention features rather than being tied to a single version bump:

| PR | Feature | Convention Source |
|----|---------|-------------------|
| [#7240](https://github.com/dotnet/extensions/pull/7240) | Server-side tool call attributes | v1.37+ |
| [#7241](https://github.com/dotnet/extensions/pull/7241) | Metric computation fix | Bug fix |
| [#7325](https://github.com/dotnet/extensions/pull/7325) | Streaming metrics (time_to_first_chunk, time_per_output_chunk) | v1.39 |
| [#7379](https://github.com/dotnet/extensions/pull/7379) | Exception event recording (gen_ai.client.operation.exception) | v1.40 |
| [#7382](https://github.com/dotnet/extensions/pull/7382) | invoke_workflow operation name | v1.40 |

## Authorship Pattern Evolution
Comment thread
jeffhandley marked this conversation as resolved.
Outdated

- **v1.29–v1.37**: Human-authored PRs by @stephentoub — terse PR descriptions, often just links
- **v1.36 onwards**: Copilot Coding Agent (CCA) introduced — detailed prompts in PR body
- **v1.39–v1.40**: Primarily CCA-authored with structured prompts or audit tables
- **v1.40 feature PRs**: Gold-standard CCA prompts (#7379, #7382) with Background → Required Changes → Tests structure

## Typical Change Patterns by Release

### Version-only releases (v1.36, v1.39)
- Only doc comment version bump needed
- Minimal code changes
- Quick turnaround

### Attribute addition releases (v1.31, v1.34, v1.35, v1.37, v1.38)
- New constants in `OpenTelemetryConsts.cs`
- New attribute emission in one or more OpenTelemetry* clients
- Test updates
- Version bump

### Behavioral change releases (v1.40 features)
- New code patterns (exception recording, streaming metrics)
- May require shared infrastructure (`Common/` classes)
- More extensive test changes
- Often split into multiple PRs per feature
Loading