Skip to content

feat(core): Align GenAI content telemetry fields #7667

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
doudouOUC merged 4 commits into from
Jul 24, 2026
Merged

feat(core): Align GenAI content telemetry fields #7667

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
496a37f
feat(core): align GenAI content telemetry fields
doudouOUC Jul 24, 2026
5230bd8
fix(telemetry): address content observability review
doudouOUC Jul 24, 2026
cde4418
docs(telemetry): clarify optional tool parameters
doudouOUC Jul 24, 2026
04bb830
docs(core): clarify telemetry compatibility behavior
doudouOUC Jul 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
95 changes: 72 additions & 23 deletions docs/design/gen-ai-arms-field-alignment.md
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion docs/design/telemetry-llm-request-timing-design.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -542,7 +542,9 @@ Rollback path: revert the single PR (or each of 4a/4b/4c independently). All new

- **After Phase 3 (#4410, in review)**: not a hard dependency. Phase 4 attributes attach to `qwen-code.llm_request` spans regardless of whether they're under a `qwen-code.subagent` (Phase 3) or `qwen-code.interaction` (Phase 1) parent. Recommend Phase 3 land first so per-attempt aggregation under subagent subtrees works naturally.
- **Independent of #4384** (`traceparent` + `X-Qwen-Code-Session-Id` outbound propagation). They touch the HTTP layer; Phase 4 touches the stream/retry/metric layer.
- **Independent of `clearDetailedSpanState` chat-compression follow-up** (#4097 follow-up). Different surface.
- **Independent of GenAI content capture**. Chat compression no longer resets
process-global sensitive-attribute hash state because that state was removed;
request timing remains a separate surface.

## Open questions

Expand Down
95 changes: 50 additions & 45 deletions docs/developers/development/telemetry.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -66,21 +66,21 @@ observability framework — Qwen Code's observability system provides:
All telemetry behavior is controlled through your `.qwen/settings.json` file.
These settings can be overridden by environment variables or CLI flags.

| Setting | Environment Variable | CLI Flag | Description | Values | Default |
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------- |
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
| `sensitiveSpanAttributeMaxLength` | `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | - | Maximum JavaScript string length for each sensitive native span attribute content payload. Set lower if your backend rejects large attributes. | `1..104857600` | `1048576` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |
| Setting | Environment Variable | CLI Flag | Description | Values | Default |
| --------------------------------- | ---------------------------------------------------- | -------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | ----------------- | ----------------------- |
| `enabled` | `QWEN_TELEMETRY_ENABLED` | `--telemetry` / `--no-telemetry` | Enable or disable telemetry | `true`/`false` | `false` |
| `target` | `QWEN_TELEMETRY_TARGET` | `--telemetry-target <local\|gcp>` _(deprecated)_ | Informational destination label; does not control exporter routing — set `otlpEndpoint` or `outfile` to configure where data is sent | `"gcp"`/`"local"` | `"local"` |
| `otlpEndpoint` | `QWEN_TELEMETRY_OTLP_ENDPOINT` | `--telemetry-otlp-endpoint <URL>` | OTLP collector endpoint | URL string | `http://localhost:4317` |
| `otlpProtocol` | `QWEN_TELEMETRY_OTLP_PROTOCOL` | `--telemetry-otlp-protocol <grpc\|http>` | OTLP transport protocol | `"grpc"`/`"http"` | `"grpc"` |
| `otlpTracesEndpoint` | `QWEN_TELEMETRY_OTLP_TRACES_ENDPOINT` | - | Per-signal endpoint override for traces (HTTP only) | URL string | - |
| `otlpLogsEndpoint` | `QWEN_TELEMETRY_OTLP_LOGS_ENDPOINT` | - | Per-signal endpoint override for logs (HTTP only) | URL string | - |
| `otlpMetricsEndpoint` | `QWEN_TELEMETRY_OTLP_METRICS_ENDPOINT` | - | Per-signal endpoint override for metrics (HTTP only) | URL string | - |
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include standard GenAI messages, instructions, tool definitions, tool arguments, and successful tool results as native span attributes | `true`/`false` | `false` |
| `sensitiveSpanAttributeMaxLength` | `QWEN_TELEMETRY_SENSITIVE_SPAN_ATTRIBUTE_MAX_LENGTH` | - | Maximum compact JSON string length for each sensitive native span attribute. Set lower if your backend rejects large attributes. | `1..104857600` | `1048576` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |

**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
Expand All @@ -91,31 +91,37 @@ other value will disable it.
must be a positive integer when set. Invalid values fail telemetry configuration
resolution instead of silently falling back.

`gen_ai.tool.description` is non-sensitive static registry metadata and is
emitted independently of `includeSensitiveSpanAttributes`. This includes
descriptions supplied by MCP servers and other workspace tool providers. The
value is limited to 4096 UTF-16 code units and never includes dynamic invocation
details.

**Sensitive span attributes:** When `includeSensitiveSpanAttributes` is enabled,
two things happen:

1. **Native span attributes (`qwen-code.interaction`, `api.generateContent*`,
`tool.<name>`)** carry verbatim conversation content:
- User prompts (`new_context`)
- System prompts (`system_prompt` — full text once per session, deduped by
SHA-256 hash; subsequent spans only carry `system_prompt_hash` +
`system_prompt_preview` + `system_prompt_length`)
- Tool schemas (emitted as `tool_schema` events, also hash-deduped)
- Tool inputs (`tool_input`) and tool results (`tool_result`)
- Model output (`response.model_output`)

Each content payload is truncated at `sensitiveSpanAttributeMaxLength`
JavaScript string units. The default is 1 MiB (`1048576`), raised from the
previous 60 KiB default; set `61440` to preserve the old cap. The limit
must be between `1` and `104857600` (100 MiB). For labeled attributes, fixed
labels such as `[USER PROMPT]`, `[TOOL INPUT: ...]`, and
`[TOOL RESULT: ...]` count against the cap; the truncation marker also counts
against it. The limit is measured as JavaScript string length rather than
UTF-8 bytes. Non-ASCII content can therefore occupy more bytes after OTLP
export. For most payload types, truncation adds both `*_truncated` and
`*_original_length`. System prompts also set `system_prompt_truncated` when
truncated, but use the always-present `system_prompt_length` for the original
length.
1. **Native span attributes** carry standard OpenTelemetry GenAI JSON:
- LLM input messages (`gen_ai.input.messages`)
- System instructions (`gen_ai.system_instructions`)
- Tool definitions (`gen_ai.tool.definitions`)
- LLM output messages (`gen_ai.output.messages`)
- Final executed tool arguments (`gen_ai.tool.call.arguments`)
- Successful tool results (`gen_ai.tool.call.result`)
- Interaction spans continue to use `new_context` because they are not GenAI
inference spans.

LLM values come from provider-final SDK request objects and raw provider
responses, not the original logical configuration. Tool values come from
the final invocation parameters and successful model-facing result. Each
standard GenAI value is compact JSON and must be complete and schema-valid.
A value that is invalid, cyclic, or longer than
`sensitiveSpanAttributeMaxLength` is omitted as a whole; JSON is never
truncated and no preview, hash, or truncation metadata is emitted. The
interaction-specific `new_context` attribute retains its existing
truncation behavior. The default maximum is 1 MiB (`1048576`) per attribute
and the accepted range is `1..104857600` (100 MiB). The limit is measured as
JavaScript string length rather than UTF-8 bytes. Non-ASCII content can
therefore occupy more bytes after OTLP export.

2. **Log-to-span bridge spans** (used when HTTP traces are exported without a
logs endpoint) keep their existing `prompt`, `function_args`, and
Expand All @@ -127,14 +133,13 @@ secrets in env vars or arguments), and model responses to the configured OTLP
backend. Treat the backend as a privileged data sink. The flag defaults to
`false`.

**Cost / payload size:** A heavy turn at the default limit (1 MiB system prompt
plus 10 tool calls, each up to 1 MiB input + 1 MiB result, plus 1 MiB model
output) can produce up to ~22 MiB of attribute payload before OTLP compression,
plus up to 1 MiB per emitted tool schema in workspaces with large tool
definitions. This is Qwen Code's application-side cap, not a guarantee that
every collector or backend accepts a single attribute that large. If spans are
rejected or dropped, lower `sensitiveSpanAttributeMaxLength` (for example, to
`61440`) and monitor exporter throughput.
**Cost / payload size:** At the default limit, one LLM span can carry at most
about 4 MiB across input, output, system instructions, and tool definitions;
one Tool span can carry about 2 MiB across arguments and result. This is Qwen
Code's application-side cap, not a guarantee that every collector or backend
accepts a single attribute that large. If spans are rejected or dropped, lower
`sensitiveSpanAttributeMaxLength` (for example, to `61440`) and monitor exporter
throughput.

This setting does not disable sensitive data in OTel logs or other telemetry
sinks; non-internal API response telemetry can populate `response_text`, so
Expand Down
Loading
Loading