docs: Document SMG_CACHE_TRACE: per-dispatch cache-aware routing evidence logs and the x-smg-cache-trace header

[github-actions]

What this PR does

How do I capture evidence of why a request landed on a worker under cache-aware or least-load routing — what do SMG_CACHE_TRACE=1, SMG_CACHE_TRACE_HEADER=1, SMG_CACHE_TRACE_SAMPLE and SMG_CACHE_TRACE_MAX_BYTES emit, on which log target, and for which worker types?

Why we need it

Source change: smg-project/smg@1b23934

Retained pending concern, re-verified and extended. Commit 64 (#2866, 1b239342, 2026-10-07) added the opt-in routing-evidence trace (primary; its supplied diff read); commit 55 (#2872) moved candidate observation to before selection; commit 18 (#2925, 432db139, 2026-10-08, diff read) added sampling and a byte cap. All verified on current main in model_gateway/src/observability/cache_trace.rs: enabled() reads SMG_CACHE_TRACE == "1" (:53-56), header_enabled() reads SMG_CACHE_TRACE_HEADER == "1" (:58-61); dispatch() (:262-297) logs an INFO "Cache routing dispatch" event on the smg::cache_trace target with one JSON evidence object per dispatch attempt carrying schema 1, router_epoch, dispatch_timestamp_ns, root_id (the x-request-id), dispatch_id, attempt, engine_ids, mode, up to 16 selections, truncated, unattributed_gates and unattributed_prediction. Each selection (:229-260) records policy and origin, the chosen worker, the cache-aware prediction, up to 32 candidates observed before selection (url, load, healthy, overloaded, registry_revision; candidates_complete; observation_started/finished_ns; load_observation_phase "before_selection"), up to 64 scores (cache_aware entries at policies/cache_aware.rs:2145-2267, least_load expected_wait entries at policies/least_load.rs:492-525) and gates capped at 32 (:196-207); PolicyRegistry::select calls begin_selection (policies/registry.rs:289). Failed pipeline runs log "Cache routing failure" (:320-331). New since the pending plan (commit 18): SMG_CACHE_TRACE_SAMPLE=N logs one dispatch line in every N (default 1; parse_sample :106-111 ignores 0/unparsable; the first dispatch is always logged; the response header is never sampled, doc comment :86-87) and SMG_CACHE_TRACE_MAX_BYTES=B compacts an over-cap evidence line — compact() :137-159 drops each selection's candidates/scores/gates lists, records their counts under elided, removes top-level gates, marks capped:true, keeping the decision record — and also caps the failure line (:327); unset or unparsable means no cap. Scope: evidence is flushed only on the gRPC pipeline — run() wraps run_inner in cache_trace::scope and records failures (routers/grpc/pipeline.rs:600-613), and the request-execution stage records dispatch (routers/grpc/common/stages/request_execution.rs:360-369) — so it covers requests served by gRPC- and ZMQ-connected workers; HTTP-relayed requests emit nothing. With SMG_CACHE_TRACE_HEADER=1 the response additionally carries x-smg-cache-trace (pipeline.rs ~733 and :2254-2255): ASCII JSON of at most 2048 bytes with the join fields only (gateway_header :299-318 strips candidates, scores and gates), omitted with a "Cache trace header omitted" log line when over the limit. The TokenSpeed Python servicer joins from the worker side under the same env var (grpc_servicer/smg_grpc_servicer/tokenspeed/servicer.py:229, cache_request_mapping with parent rid and child ids). Missing documentation: a repo-wide grep for SMG_CACHE_TRACE, cache_trace, smg::cache_trace and x-smg-cache-trace finds nothing in src/lib/content; getting-started/logging.md's Other Structured Logs table (:150-154) and Routing Decision Logs section (:158-182) cover only the smg::policies DEBUG lines from #2210; concepts/routing/cache-aware.md's Monitoring > Debug Logs (:340-342) names only those; reference/api/openai.md's Response Headers table (:786-794) lacks the opt-in x-smg-cache-trace row. Version framing: #2866/#2872 postdate the v1.11.0 bump (commit 181, #2670) and precede the v1.12.0 bump (commit 40, #2888); the sample/cap knobs (#2925) postdate the v1.12.0 bump; the docs name no release past v1.11.0 and this repo holds no tag evidence, so present everything as current-main behavior newer than v1.11.0 without naming a release. One independent concern: one commit introduced one opt-in routing-evidence surface (env toggles, log events, header), documented where routing observability already lives; commits 55 and 18 refined that same surface. Dedup: no open or closed PR mentions cache trace (closed-unmerged #108, #86, #72 were different observability concerns and are not recreated); no open PR touches logging.md, cache-aware.md or openai.md; my other selected items touch metrics.md/monitoring.md and rl-control-plane.md, so no file overlaps. Edit is well under 1000 lines.

Scope: observability / cache-aware-routing-trace-smg-cache-trace. Other concerns are deferred.

How to test

• Passed the documentation path and size guard (under 1000 added plus deleted lines; no file-count limit).
• Passed an independent accuracy and single-concern review.
• Passed git diff --check and pnpm check and the production pnpm build.

Checklist

• Tests added/updated (if applicable)
• Docs updated (if applicable)
• Commit includes a DCO sign-off

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: f4e6d882-fa9f-440b-857e-9071e4fd4f71

---

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}

[github-actions]

Documentation maintenance: ready. Unsuccessful content rounds: 0/3. Operational attempts: 0/3.

Head: e2abcaf73e4e1bfdb4c2223f44eb73c855404450; reviewed main: e17c545940df0ab77f19d09285b49aa9e4773d7c.
Reviewed SMG source: 2fae47becb202cd6d36e28a9e4359a2fd53901d4.

All ~50 changed lines serve the single routing-evidence-trace concern across exactly the three allowlisted pages. Attribution verified from the supplied source_patch (commit 1b239342 added SMG_CACHE_TRACE/SMG_CACHE_TRACE_HEADER, the smg::cache_trace dispatch/failure events, the 2048-byte ASCII gateway header, and the TokenSpeed cache_request_mapping line). Every documented claim verifies against the checkout: exact-"1" OnceLock env switches read once per process (cache_trace.rs:72-159); sample default 1 with first-dispatch-always, 0/unparsable=every, failure lines and header never sampled (:107-180 plus tests); byte-cap compaction dropping per-selection candidates/scores/gates into elided counts with capped:true, also applied to failure lines (:183-219, :395); dispatch schema with router_epoch, mode labels (context.rs:427-438), root_id = the x-request-id middleware ID (request_id.rs:115-125, helpers.rs:129-133), engine_ids <=32 with {id}-{index} PD fan-out (request_execution.rs:162-202); selections <=16, candidates <=32 observed before selection (registry.rs:289, cache_trace.rs:37-60,319), backend epochs withdrawn on NotReady/Failed (worker.rs:1761-1767, 2218-2233); predictions event_index_overlap/approximate_tree/approximate_hash_index and policy_affinity/expected_wait scores and eligibility/spill/queue-cap gates at the cited cache_aware.rs/least_load.rs sites, with the queue cap wired only for least_load (factory.rs:27-42) and power_of_two sharing the scorer; gRPC-pipeline scope covering grpc:// and ipc:// (ZMQ) workers while HTTP routers never open a capture scope (pipeline.rs:615-628, routers/factory.rs:62-63); header attached at routed_response (pipeline.rs:790-793) with request-opt-in and DEBUG-vs-INFO omission (:349-358). Placement: schema documented once in logging.md with cross-references, new behavior in its own sections/paragraphs, release notes as short standalone current-main statements below the tables naming no version (no tag evidence exists), and no PR/commit/code references in the added prose; the one edit to retained text is the required in-place qualification of the DEBUG routing-logs generalization. Related pages hold no contradictions: no other content page mentions the trace, the DEBUG-log descriptions elsewhere make no exclusivity claims, and configuration.md's env-var reference is explicitly v1.11.0-scoped so it correctly omits the unreleased variables. Limitations: Read/Glob/Grep only, so the docs build was not re-run and header attachment was verified at the shared routed_response chokepoint rather than per endpoint. The PR has no review threads, so none are addressed.

Workflow evidence

Human review threads and CODEOWNER approval remain under repository policy.