Skip to content
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
762 changes: 762 additions & 0 deletions docs/design/telemetry-resource-attributes-design.md

Large diffs are not rendered by default.

119 changes: 119 additions & 0 deletions docs/developers/development/telemetry.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,8 @@ These settings can be overridden by environment variables or CLI flags.
| `outfile` | `QWEN_TELEMETRY_OUTFILE` | `--telemetry-outfile <path>` | Save telemetry to file (overrides OTLP export) | file path | - |
| `logPrompts` | `QWEN_TELEMETRY_LOG_PROMPTS` | `--telemetry-log-prompts` / `--no-telemetry-log-prompts` | Include prompts in telemetry logs | `true`/`false` | `true` |
| `includeSensitiveSpanAttributes` | `QWEN_TELEMETRY_INCLUDE_SENSITIVE_SPAN_ATTRIBUTES` | - | Include user prompts, system prompts, tool I/O, and model output as native span attributes (in addition to log-to-span bridge spans) | `true`/`false` | `false` |
| `resourceAttributes` | `OTEL_RESOURCE_ATTRIBUTES` (+ `OTEL_SERVICE_NAME`) | - | Static resource attributes attached to every exported span / log / metric. See [Resource attributes](#resource-attributes) below. | `key=value,…` | `{}` |
| `metrics.includeSessionId` | `QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID` | - | Include `session.id` on metric data points. **Disabled by default** to protect metric backends from time-series fan-out. | `true`/`false` | `false` |

**Note on boolean environment variables:** For the boolean settings (`enabled`,
`logPrompts`, `includeSensitiveSpanAttributes`), setting the
Expand Down Expand Up @@ -125,6 +127,123 @@ The `QWEN_TELEMETRY_OTLP_*` variants take precedence over the `OTEL_*` variants.
For detailed information about all configuration options, see the
[Configuration Guide](./cli/configuration.md).

### Resource attributes

Resource attributes are static key-value pairs attached to every span, log,
and metric exported via OTLP. Use them to slice telemetry by team, environment,
deployment region, or any other dimension your backend cares about.

Two sources, merged in priority order (lowest → highest):

1. The standard `OTEL_RESOURCE_ATTRIBUTES` env var
2. `telemetry.resourceAttributes` in `.qwen/settings.json` (overrides env on
key conflict)

`OTEL_SERVICE_NAME` is a separate escape hatch — when set, it overrides
`service.name` from any other source (per the OpenTelemetry spec).

#### Examples

**Slice all telemetry by team / environment:**

```bash
export OTEL_RESOURCE_ATTRIBUTES="team=platform,env=prod,cost_center=eng-123"
```

**Route to a per-tenant collector via `service.name`:**

```bash
export OTEL_SERVICE_NAME=qwen-code-ci
```

**Fleet baseline (`~/.qwen/settings.json`) + per-host override:**

```json
{
"telemetry": {
"resourceAttributes": {
"deployment.environment": "production",
"service.namespace": "engineering-tooling"
}
}
}
```

```bash
# Add a one-off tag without touching settings:
export OTEL_RESOURCE_ATTRIBUTES="debug_run=true"
```

#### Reserved keys

Some keys are runtime-controlled and cannot be overridden:

- `service.version` — always set to the running CLI version. Setting it from
any source is silently dropped with a warning.
- `session.id` — runtime-injected per session. User-provided values from
either env or settings are dropped with a warning. The reason is that
Resource attributes auto-attach to every metric data point; allowing user
override would bypass [Cardinality controls](#cardinality-controls) below.
Spans and logs always carry `session.id`.

`service.name` is **not** reserved; it follows the precedence chain above.

#### Format

`OTEL_RESOURCE_ATTRIBUTES` follows the OpenTelemetry spec:
`key1=value1,key2=value2` with values percent-encoded. Spaces in values must
be encoded as `%20`. Malformed pairs are skipped with a warning rather than
failing telemetry startup.

### Cardinality controls

Metrics are aggregated by attribute set at the backend — every distinct
combination of attribute values produces a new time series. Attaching a
high-cardinality field like `session.id` to a metric causes time-series fan-out
proportional to the number of sessions, which quickly exhausts metric backend
storage.

To prevent this, Qwen Code keeps high-cardinality attributes off metric data
points by default. Spans and logs are per-event and unaffected, so they
continue to carry `session.id` for trace and log correlation.

#### `telemetry.metrics.includeSessionId` (default: `false`)

Setting this to `true` (via settings or
`QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true`) re-attaches `session.id` to
every metric data point.

⚠️ **Warning:** each CLI session creates a new value. Leaving this on for a
fleet will blow up metric storage. Recommended only for short-term debugging.
For long-term session correlation, query trace or log backends instead.

#### Migration from earlier versions

Prior to this release, `session.id` was attached to metrics by default. If
your Prometheus queries / Grafana dashboards / alert rules reference
`session_id` on a metric, you have two options:

**Option A** — restore the previous behavior for short-term debugging:

```bash
export QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID=true
```

or:

```json
{
"telemetry": {
"metrics": { "includeSessionId": true }
}
}
```

**Option B (recommended)** — move session-level analysis off metrics. Spans
and logs still carry `session.id`, and trace / log backends (Jaeger, Tempo,
Loki, Aliyun SLS / ARMS Tracing) handle per-session slicing natively without
cardinality pressure.

## Aliyun Telemetry

### Manual OTLP Export
Expand Down
20 changes: 20 additions & 0 deletions packages/cli/src/config/settingsSchema.ts
Original file line number Diff line number Diff line change
Expand Up @@ -1012,6 +1012,26 @@ const SETTINGS_SCHEMA = {
type: 'boolean',
default: false,
},
resourceAttributes: {
description:
'Static resource attributes attached to every span/log/metric the SDK exports (OTLP or file outfile — they share the same Resource). Merged with the OTEL_RESOURCE_ATTRIBUTES env var; settings win on key conflict. Reserved keys (service.version, session.id) are dropped with a warning.',
type: 'object',
additionalProperties: { type: 'string' },
default: {},
},
Comment thread
doudouOUC marked this conversation as resolved.
Resolved
metrics: {
description: 'Per-signal cardinality controls for exported metrics.',
type: 'object',
additionalProperties: false,
properties: {
includeSessionId: {
description:
'Include session.id on every metric data point. WARNING: each CLI session creates a new value, causing unbounded metric time-series fan-out at the backend. Only enable for short-term debugging — spans and logs still carry session.id.',
type: 'boolean',
default: false,
},
},
},
},
additionalProperties: true,
},
Expand Down
31 changes: 31 additions & 0 deletions packages/core/src/config/config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -304,6 +304,27 @@ export interface TelemetrySettings {
logPrompts?: boolean;
includeSensitiveSpanAttributes?: boolean;
outfile?: string;
/**
* Static resource attributes attached to every span/log/metric the SDK
* exports (OTLP or file outfile — they share the same Resource).
* Merged with `OTEL_RESOURCE_ATTRIBUTES`; settings win on key conflict.
* Reserved keys (`service.version`, `session.id`) are dropped with a
* `diag.warn`.
*/
resourceAttributes?: Record<string, string>;
/** Per-signal cardinality controls. */
metrics?: TelemetryMetricsSettings;
Comment thread
doudouOUC marked this conversation as resolved.
Resolved
}

export interface TelemetryMetricsSettings {
/**
* Include `session.id` on every metric data point. Default: false.
*
* WARNING: each CLI session creates a new value, causing unbounded
* metric time-series fan-out at the backend. Only enable for
* short-term debugging — spans and logs still carry session.id.
*/
includeSessionId?: boolean;
}

export interface OutputSettings {
Expand Down Expand Up @@ -987,6 +1008,8 @@ export class Config {
includeSensitiveSpanAttributes:
params.telemetry?.includeSensitiveSpanAttributes ?? false,
outfile: params.telemetry?.outfile,
resourceAttributes: params.telemetry?.resourceAttributes,
metrics: params.telemetry?.metrics,
};
this.gitCoAuthor = {
...normalizeGitCoAuthor(params.gitCoAuthor),
Expand Down Expand Up @@ -2807,6 +2830,14 @@ export class Config {
return this.telemetrySettings.target ?? DEFAULT_TELEMETRY_TARGET;
}

getTelemetryResourceAttributes(): Record<string, string> {
return this.telemetrySettings.resourceAttributes ?? {};
}

getTelemetryMetricsIncludeSessionId(): boolean {
return this.telemetrySettings.metrics?.includeSessionId ?? false;
}

getTelemetryOutfile(): string | undefined {
return this.telemetrySettings.outfile;
}
Expand Down
149 changes: 149 additions & 0 deletions packages/core/src/telemetry/config.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,8 @@ describe('telemetry/config helpers', () => {
otlpTracesEndpoint: undefined,
otlpLogsEndpoint: undefined,
otlpMetricsEndpoint: undefined,
resourceAttributes: undefined,
metrics: { includeSessionId: false },
});
});

Expand Down Expand Up @@ -113,6 +115,8 @@ describe('telemetry/config helpers', () => {
logPrompts: true,
includeSensitiveSpanAttributes: true,
outfile: 'env.log',
resourceAttributes: undefined,
metrics: { includeSessionId: false },
});

const resolvedArgv = await resolveTelemetrySettings({
Expand All @@ -131,6 +135,8 @@ describe('telemetry/config helpers', () => {
logPrompts: false,
includeSensitiveSpanAttributes: true,
outfile: 'argv.log',
resourceAttributes: undefined,
metrics: { includeSessionId: false },
});
});

Expand Down Expand Up @@ -233,4 +239,147 @@ describe('telemetry/config helpers', () => {
);
});
});

describe('resolveTelemetrySettings — resource attributes', () => {
it('returns undefined resourceAttributes when nothing set', async () => {
const resolved = await resolveTelemetrySettings({});
expect(resolved.resourceAttributes).toBeUndefined();
});

it('parses OTEL_RESOURCE_ATTRIBUTES from env', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'team=platform,env=prod' },
});
expect(resolved.resourceAttributes).toEqual({
team: 'platform',
env: 'prod',
});
});

it('merges settings on top of env (settings wins)', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'team=x,env=prod' },
settings: { resourceAttributes: { team: 'y' } },
});
expect(resolved.resourceAttributes).toEqual({
team: 'y',
env: 'prod',
});
});

it('reads service.name from OTEL_SERVICE_NAME alone', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_SERVICE_NAME: 'A' },
});
expect(resolved.resourceAttributes).toEqual({ 'service.name': 'A' });
});

it('reads service.name from OTEL_RESOURCE_ATTRIBUTES alone', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'service.name=B' },
});
expect(resolved.resourceAttributes).toEqual({ 'service.name': 'B' });
});

it('drops user-provided session.id from env with warning', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'session.id=spoofed,team=x' },
});
expect(resolved.resourceAttributes).toEqual({ team: 'x' });
});

it('drops user-provided session.id from settings with warning', async () => {
const resolved = await resolveTelemetrySettings({
settings: {
resourceAttributes: { 'session.id': 'spoofed', team: 'x' },
},
});
expect(resolved.resourceAttributes).toEqual({ team: 'x' });
});

it('drops non-string settings values', async () => {
const resolved = await resolveTelemetrySettings({
settings: {
resourceAttributes: {
team: 'platform',
// @ts-expect-error — runtime defensive path against bad JSON.
count: 42,
},
},
});
expect(resolved.resourceAttributes).toEqual({ team: 'platform' });
});

it('OTEL_SERVICE_NAME wins over OTEL_RESOURCE_ATTRIBUTES.service.name', async () => {
const resolved = await resolveTelemetrySettings({
env: {
OTEL_SERVICE_NAME: 'A',
OTEL_RESOURCE_ATTRIBUTES: 'service.name=B',
},
});
expect(resolved.resourceAttributes?.['service.name']).toBe('A');
});

it('OTEL_SERVICE_NAME wins over settings.resourceAttributes.service.name', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_SERVICE_NAME: 'A' },
settings: { resourceAttributes: { 'service.name': 'C' } },
});
expect(resolved.resourceAttributes?.['service.name']).toBe('A');
});

it('settings.service.name wins over env.OTEL_RESOURCE_ATTRIBUTES.service.name when no OTEL_SERVICE_NAME', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'service.name=B' },
settings: { resourceAttributes: { 'service.name': 'C' } },
});
expect(resolved.resourceAttributes?.['service.name']).toBe('C');
});

it('strips service.version from env source', async () => {
const resolved = await resolveTelemetrySettings({
env: { OTEL_RESOURCE_ATTRIBUTES: 'service.version=fake,team=x' },
});
expect(resolved.resourceAttributes).toEqual({ team: 'x' });
});

it('strips service.version from settings source', async () => {
const resolved = await resolveTelemetrySettings({
settings: {
resourceAttributes: { 'service.version': 'fake', team: 'x' },
},
});
expect(resolved.resourceAttributes).toEqual({ team: 'x' });
});
});

describe('resolveTelemetrySettings — metrics.includeSessionId', () => {
it('defaults to false', async () => {
const resolved = await resolveTelemetrySettings({});
expect(resolved.metrics?.includeSessionId).toBe(false);
});

it('reads from settings', async () => {
const resolved = await resolveTelemetrySettings({
settings: { metrics: { includeSessionId: true } },
});
expect(resolved.metrics?.includeSessionId).toBe(true);
});

it('reads from env (override settings)', async () => {
const resolved = await resolveTelemetrySettings({
env: { QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID: 'true' },
settings: { metrics: { includeSessionId: false } },
});
expect(resolved.metrics?.includeSessionId).toBe(true);
});

it('explicit env=false overrides settings=true', async () => {
const resolved = await resolveTelemetrySettings({
env: { QWEN_TELEMETRY_METRICS_INCLUDE_SESSION_ID: 'false' },
settings: { metrics: { includeSessionId: true } },
});
expect(resolved.metrics?.includeSessionId).toBe(false);
});
});
});
Loading
Loading