Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
762 changes: 762 additions & 0 deletions docs/design/telemetry-resource-attributes-design.md

Large diffs are not rendered by default.

137 changes: 137 additions & 0 deletions docs/developers/development/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ These settings can be overridden by environment variables or CLI flags.
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |

**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
Expand Down Expand Up @@ -125,6 +127,141 @@ The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
For detailed information about all configuration options, see the
[Configuration Guide](./cli/configuration.md).

### Resource attributes

Resource attributes are static key-value pairs attached to every span, log,
and metric exported via OTLP. Use them to slice telemetry by team, environment,
deployment region, or any other dimension your backend cares about.

Two sources, merged in priority order (lowest → highest):

1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
key conflict)

`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
`service.name` from any other source (per the OpenTelemetry spec).

#### Examples

**Slice all telemetry by team / environment:**

```bash
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
```

**Route to a per-tenant collector via `service.name`:**

```bash
export OTEL_SERVICE_NAME=qwen-code-ci
```

**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**

```json
{
"telemetry": {
"resourceAttributes": {
"deployment.environment": "production",
"service.namespace": "engineering-tooling"
}
}
}
```

```bash
# Add a one-off tag without touching settings:
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
```

#### Reserved keys

Some keys are runtime-controlled and cannot be overridden:

- `service.version` — always set to the running CLI version. Setting it from
any source is silently dropped with a warning.
- `session.id` — runtime-injected per session. User-provided values from
either env or settings are dropped with a warning. The reason is that
Resource attributes auto-attach to every metric data point; allowing user
override would bypass [Cardinality controls](#cardinality-controls) below.
Spans and logs always carry `session.id`.

`service.name` is **not** reserved; it follows the precedence chain above.

#### Format

`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
be encoded as `%20`, **commas as `%2C`** (unencoded commas split the value at
the wrong boundary and the second half is dropped as malformed). Malformed
pairs are skipped with a warning rather than failing telemetry startup.

#### Troubleshooting: when a user-provided attribute appears not to take effect

Reserved keys (`service.version`, `session.id`), malformed pairs, non-string
settings values, and invalid percent-encoding are all silently dropped with a
warning logged via the OpenTelemetry diagnostics channel. That channel routes
to the debug log file (`~/.qwen/log/otel-*.log`), **not** the console, so the
behavior can look like silent failure.

If a custom resource attribute isn't appearing on exported telemetry:

1. Check `~/.qwen/log/otel-*.log` for lines matching `cannot override` (reserved
key dropped), `Skipping malformed` (bad env var pair), or `must be a string`
(non-string settings value).
2. Verify the env var is set in the qwen-code process's environment (not just
your shell) and that values are percent-encoded.
3. Confirm `telemetry.enabled` is `true` — telemetry init only runs if enabled.

### Cardinality controls

Metrics are aggregated by attribute set at the backend — every distinct
combination of attribute values produces a new time series. Attaching a
high-cardinality field like `session.id` to a metric causes time-series fan-out
proportional to the number of sessions, which quickly exhausts metric backend
storage.

To prevent this, Qwen Code keeps high-cardinality attributes off metric data
points by default. Spans and logs are per-event and unaffected, so they
continue to carry `session.id` for trace and log correlation.

#### `telemetry.metrics.includeSessionId` (default: `false`)

Setting this to `true` (via settings or
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
every metric data point.

⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
fleet will blow up metric storage. Recommended only for short-term debugging.
For long-term session correlation, query trace or log backends instead.

#### Migration from earlier versions

Prior to this release, `session.id` was attached to metrics by default. If
your Prometheus queries / Grafana dashboards / alert rules reference
`session_id` on a metric, you have two options:

**Option A** — restore the previous behavior for short-term debugging:

```bash
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
```

or:

```json
{
"telemetry": {
"metrics": { "includeSessionId": true }
}
}
```

**Option B (recommended)** — move session-level analysis off metrics. Spans
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
cardinality pressure.

## Aliyun Telemetry

### Manual OTLP Export
Expand Down
20 changes: 20 additions & 0 deletions packages/cli/src/config/settingsSchema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1012,6 +1012,26 @@ const SETTINGS_SCHEMA = {
type: 'boolean',
default: false,
},
resourceAttributes: {
description:
'Static resource attributes attached to every span/log/metric the SDK exports (OTLP or file outfile — they share the same Resource). Merged with the OTEL_RESOURCE_ATTRIBUTES env var; settings win on key conflict. Reserved keys (service.version, session.id) are dropped with a warning.',
type: 'object',
additionalProperties: { type: 'string' },
default: {},
},
Comment thread
doudouOUC marked this conversation as resolved.
metrics: {
description: 'Per-signal cardinality controls for exported metrics.',
type: 'object',
additionalProperties: false,
properties: {
includeSessionId: {
description:
'Include session.id on every metric data point. WARNING: each CLI session creates a new value, causing unbounded metric time-series fan-out at the backend. Only enable for short-term debugging — spans and logs still carry session.id.',
type: 'boolean',
default: false,
},
},
},
},
additionalProperties: true,
},
Expand Down
46 changes: 46 additions & 0 deletions packages/core/src/config/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,37 @@ export interface TelemetrySettings {
logPrompts?: boolean;
includeSensitiveSpanAttributes?: boolean;
outfile?: string;
/**
* Static resource attributes attached to every span/log/metric the SDK
* exports (OTLP or file outfile — they share the same Resource).
* Merged with `OTEL_RESOURCE_ATTRIBUTES`; settings win on key conflict.
* Reserved keys (`service.version`, `session.id`) are dropped with a
* `diag.warn`.
*/
resourceAttributes?: Record<string, string>;
/** Per-signal cardinality controls. */
metrics?: TelemetryMetricsSettings;
Comment thread
doudouOUC marked this conversation as resolved.
/**
* Human-readable diagnostics produced while resolving
* `resourceAttributes` (drops, coercions, reserved-key strips).
* Populated by `resolveTelemetrySettings()`; the SDK emits a one-time
* console summary at startup when this is non-empty so users notice
* silent drops without scanning the OTel debug log.
*
* Not a user-settable field — operators should leave it unset.
*/
resourceAttributeWarnings?: string[];
}

export interface TelemetryMetricsSettings {
/**
* Include `session.id` on every metric data point. Default: false.
*
* WARNING: each CLI session creates a new value, causing unbounded
* metric time-series fan-out at the backend. Only enable for
* short-term debugging — spans and logs still carry session.id.
*/
includeSessionId?: boolean;
}

export interface OutputSettings {
Expand Down Expand Up @@ -987,6 +1018,9 @@ export class Config {
includeSensitiveSpanAttributes:
params.telemetry?.includeSensitiveSpanAttributes ?? false,
outfile: params.telemetry?.outfile,
resourceAttributes: params.telemetry?.resourceAttributes,
metrics: params.telemetry?.metrics,
resourceAttributeWarnings: params.telemetry?.resourceAttributeWarnings,
};
this.gitCoAuthor = {
...normalizeGitCoAuthor(params.gitCoAuthor),
Expand Down Expand Up @@ -2807,6 +2841,18 @@ export class Config {
return this.telemetrySettings.target ?? DEFAULT_TELEMETRY_TARGET;
}

getTelemetryResourceAttributes(): Record<string, string> {
return this.telemetrySettings.resourceAttributes ?? {};
}

getTelemetryMetricsIncludeSessionId(): boolean {
return this.telemetrySettings.metrics?.includeSessionId ?? false;
}

getTelemetryResourceAttributeWarnings(): readonly string[] {
return this.telemetrySettings.resourceAttributeWarnings ?? [];
}

getTelemetryOutfile(): string | undefined {
return this.telemetrySettings.outfile;
}
Expand Down
Loading
Loading