[henrypark133]

Summary

Arm stock SGLang async OTLP tracing in the promoted W4AFP8 + HiCache long-context compose file. Both replicas start at SGLANG_TRACE_LEVEL=0. An operator-only, one-shot Compose service calls /set_trace_level on a selected replica from inside the CVM network; no guest shell or publicly exposed management endpoint is needed. The existing collector already accepts OTLP gRPC on port 4317 and forwards traces to the telemetry gateway. This PR does not deploy the file or turn on request traces at startup.

The generator, generated compose file, and exact production validator move together. The config variant identifies the armed state.

Production capture after review and rollout

1. First confirm the pinned image starts with tracing enabled and that the async exporter is healthy. Check for missing OTel or pyzmq dependencies and exporter errors before a capture.
2. On one engine replica only, use the glm53-trace-control service described below to set level 3. Capture a short window of representative 128K+ requests, then run the service with level 0.
3. Watch TTFT, decode inter-token latency, GPU utilization, collector queue/drops, and async trace exporter warnings. Roll back the trace level immediately if latency or drops rise. Requests already admitted while level 3 was active may continue emitting spans after the level is reset.
4. Correlate SGLang rid/request metadata with proxy request IDs where possible. Verify that the traces actually arrive in the backend before using them for conclusions. chunked_prefill spans can include scheduler waiting between chunks; they are not pure GPU compute time.

Level 3 creates a decode_loop span on each decode step plus prefill and other request-stage spans. A 2,000-token output can therefore generate roughly 2,000 decode spans. Level 0 avoids request spans but still initializes tracing/exporter machinery; its startup and performance impact require a production check.

CVM control procedure

Use the same immutable tag, compose file, project, and complete gpu-manager env map as the deployed engines. Add GLM53_TRACE_REPLICA=1 and GLM53_TRACE_LEVEL=3 to that env map. Call compose-manager POST /compose/up with services: ["glm53-trace-control"], force_recreate: false, and dry_run: true. Inspect the raw plan: only the control job may be created/recreated/started, with no removals or engine changes. Apply the identical payload with dry_run: false, then read its result through scoped POST /compose/logs for glm53-trace-control (tail: 15). Expect trace_control_applied replica=1 level=3 and no curl error. Repeat with GLM53_TRACE_LEVEL=0 to stop new requests from tracing; that env change recreates the stopped one-shot control job. A restart of the SGLang engine also resets it to startup level 0.

The job uses the already-pinned curlimages/curl image, runc, no GPU, read-only filesystem, dropped capabilities, no Docker socket or mounted secrets, and no published port. It accepts only replica 1/2 and level 0/3. Do not invoke this file before the traced long-context file is itself deployed: compose-manager's up uses --remove-orphans, and a different live file/tag could evict serving containers.

Validation

• Generated target matches generator (--check).
• Production compose contract validator passes for all five GLM-5.3 files.
• Eleven focused generator/validator tests pass, including trace argv/environment and control-job drift checks.
• git diff --check passes.
• Full long-context test module has two DCGM validator cases blocked locally by system Ruby 2.6 (Array#filter_map requires a newer Ruby). Other eleven cases pass.
• The control script passed a local shell syntax and input-validation check with a stubbed HTTP client. No Docker daemon/Compose plugin is available locally, so image startup, compose-manager execution, trace delivery, and performance impact remain runtime gates.

[github-actions]

✅ OpenCodeReview: Review failed: 0 finding(s); 3 of 3 selected item(s) failed.

[PierreLeGuen]

Reviewed with a focus on privacy, since these spans leave the CVM.

What the spans contain. I read the tracing code at SGLang fc91d24, the base of the pinned image. At level 3, each request produces:

• its SGLang request ID
• prompt, cached and completion token counts
• sampling params (max_tokens, temperature, top_p, top_k, n)
• the model name and the finish type
• per-stage timings, plus one span per decode step with batch size

There's no prompt text, output text or token IDs (span attributes). The only paths that put error messages into spans are in the disaggregation code, which this file doesn't use. There are no customer identifiers either: SGLang only reads traceparent/tracestate from request headers, and the proxy doesn't send those.

One change before merging: filter inside the CVM. The traces pipeline in otelcol_app_config exports spans unfiltered to telemetry.infra.near.ai. That's fine today, but only because this SGLang build happens to emit no content. A future SGLang upgrade could add prompt or completion attributes, and we wouldn't notice. Please add an allowlist to the in-CVM traces pipeline that keeps only the attributes above, on both span and span-event attributes. For example, a transform processor with keep_keys. Then extend the validator to pin it.

Two things to agree on, not blocking:

1. The trace level can change without a config change. The file starts both engines at level 0, but SGLang's /set_trace_level has no auth, unlike /freeze_gc next to it. So anything on the CVM network, or an operator going through compose-manager, can turn on level 3 without a new file or release.
   • It isn't reachable from outside today. The proxy only serves an explicit route list and returns 404 for everything else, and the soak relay allows six paths and only starts under the verification profile.
   • Since external verifiers review this file, let's announce each capture, time-box it, and record it.
2. This is the first per-request record exported from the CVM. Step 4 of the runbook, correlating SGLang request IDs with proxy request IDs, is what would tie spans to customers. Let's keep captures short, only do that join when we need it, and make sure our data-handling docs cover trace metadata.

Already handled: SGLang defaults to level 3 when SGLANG_TRACE_LEVEL is unset. The file sets 0 and the validator rejects anything else, so please keep that check.

Non-privacy notes:

• Deploying this reverts the r2 A/B. This file puts both gpu02 replicas on chunk 8192 / pdi 1. The r2 A/B running there now (chunk 16384, indexer split, pdi 2) would be undone if this is deployed as-is, so coordinate the rollout.
• Span volume. Level 3 emits one span per decode step. The in-CVM collector (512 MiB memory limiter, 128-batch sending queue) will likely drop spans during a long-context burst, so check for drops before trusting a capture.

[henrypark133]

@PierreLeGuen, thanks for the review. All of it is addressed in 91b88b5 (merge of main) and 043ddb8 (review fixes).

Blocking: filter inside the CVM. Fixed.

• New processor. transform/sglang_trace_allowlist (collector v0.120.0, OTTL) runs keep_keys on span attributes and span-event attributes.
• What survives. Only the keys SGLang fc91d24 emits on this path: request id, token counts, sampling params, model, finish reasons, per-stage latencies, rank/thread labels, decode count, accepted-draft counts, and the schedule event's bid/batch_size/forward_mode.
• What is dropped. Everything else, including abort/error text, matched stop strings and host_id. Span status descriptions are cleared.
• Fail closed. error_mode: propagate drops a batch rather than exporting it unfiltered.
• Validator. It pins the processor exactly, requires a single traces pipeline that goes through it, and has contract tests for adding or removing a key and for bypassing the processor.
• Checked in the pinned collector image. A crafted span with extra attributes came out with only allowlisted keys.
• Known remainder, documented. Span names still carry an 8-char rid prefix and an 8-char machine-id prefix.

Deploying reverts the r2 A/B. Fixed by merging main through the generator (never hand-merging the YAML). The file keeps #307's replica configs: both replicas on 8ff1a487, chunk 8192, indexer split on, pdi 1 on r1 and pdi 2 on r2.

Span volume and drops. Addressed.

• Traces now use their own otlphttp/gateway_traces exporter: same endpoint and auth, in-memory queue of 32, no file storage, retries capped at 60 s.
• So a level-3 burst can only drop traces, never logs or metrics.
• The validator rejects sharing that exporter with the logs or metrics pipelines.
• The runbook says to check collector drop metrics before trusting a capture.

Found in testing and fixed. After the merge, r2 has its own environment: list. A YAML merge key replaces a list rather than extending it, so the trace variables have to be in r2's list too. An unset SGLANG_TRACE_LEVEL defaults to 3 with synchronous export.

• Both replicas now set SGLANG_TRACE_ASYNC=1 and SGLANG_TRACE_LEVEL=0 explicitly.
• The validator checks each traced replica's effective environment.
• Mutation tests cover removing either variable from r1 or r2, and setting r2 to level 3.

Non-blocking items, now in the runbook. They're in a new "Request tracing" section of docs/glm53-w4afp8-long-context-rollout.md and the file header.

• /set_trace_level has no auth and is reachable only on the CVM network. Every capture is announced, time-boxed and recorded.
• The rid ↔ proxy-request join is done only when needed, and data-handling docs must cover trace metadata.
• Level 2 already emits spec_draft/spec_verify spans on every decode step because EAGLE is on.
• OTel scheduler spans are lost with --tokenizer-worker-num > 1 ([Bug] OTel tracing loses Scheduler spans when using --tokenizer-worker-num>1 sgl-project/sglang#38210). Revisit this if tokenizer workers are enabled.
• The control job still accepts only levels 0 and 3.

r2 image dependencies. Likely fine, but not confirmed by an import test.

• The registry metadata traces 8ff1a487 back to lmsysorg/sglang@eb090e39, built with pip install ".[all]".
• At that commit, all includes the tracing extra, and pyzmq is a core dependency.
• Our recipes install SGLang with --no-deps.

Checks. These CI steps were run locally inside ruby:3.3-alpine, as CI runs them, and pass:

• validate_glm53_prod_config.rb
• the long-context, base and canary generators (--check) and their tests (19/19 for long-context)
• the hicache canary test
• the OTel label and DCGM contracts
• the embedded-collector validate

GitHub CI on 043ddb8 is pending.