Repository navigation
Rollout doc: recreate otelcol-contrib when a canary changes its labels - #303
Open
pranavraja99 wants to merge 1 commit into
Open
pranavraja99 wants to merge 1 commit into
pranavraja99 wants to merge 1 commit into
Conversation
A replica-scoped compose/up leaves otelcol-contrib on the previous tag, and config_variant and engine_image live in the otelcol_app_config block that only the collector reads. The engine then runs the new config while reporting itself as the old one, both replicas carry identical labels, and the A/B silently compares a replica against itself. Hit exactly this on the 2026-09-25 gpu02 canary: r2 ran c16384/pdi2 for about 15 minutes while reporting c8192/pdi1/fde25985aea3. The dashboard looked fine, which is what makes it worth writing down. Fixed by recreating the collector alone. Adds the step, a PromQL check that the two arms report different labels, a note that the stale series lingers for one staleness window, and the gpu02 startup record including the first in-CVM confirmation of the HCC-safe allocator. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
✅ OpenCodeReview: Review skipped: no items were selected. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Docs only. No prod file, script or image changes.
The problem
The canary procedure says to
compose/downthencompose/upscoped to["model-sg-glm53-w4afp8-tp4-r2"]. That is right for the engines — butconfig_variantandengine_imagelive in theotelcol_app_configblock, which only otelcol-contrib reads. A replica-scoped deploy leaves the collector on the previous tag.The result is silent: the engine runs the new config while reporting itself as the old one. Both replicas then carry identical
config_variant, and the A/B compares a replica against itself while the dashboard looks entirely normal.This happened on the 2026-09-25 gpu02 canary. r2 ran
c16384/pdi2for ~15 minutes while reportingc8192/pdi1/fde25985aea3. It was caught by checking the labels rather than trusting that they were wired, and fixed by recreating the collector alone (telemetry-only; no engine touched).What this adds
otelcol-contribrecreate step, scoped to canaries that change either label, with why it is needed.quant=w4afp8, KV pool 3,525,632 tokens, DeepGEMM warmup 32,768 shapes in 3 s from the retained kernel-cache volume, old r2 draining its 2 in-flight requests before exit, andAllocated 57.07 GiB CUDA-owned pinned host memory … SGLANG_HICACHE_CUDA_HOST_MEMORY=1— the first confirmation that the HCC-safe allocator works inside a CVM, which gpu31 could not test with CC off.Validation
prepare_glm53_w4afp8_long_context.py --checkandvalidate_glm53_prod_config.rbboth pass; neither is affected by a docs change.🤖 Generated with Claude Code