Skip to content

observability: publish the generated forward-pass-metrics endpoint to the bags - #33241

Closed
ch-wan wants to merge 1 commit into
cheng/gc-wb-3-control-planefrom
cheng/gc-wb-4-fpm-endpoint
Closed

ch-wan wants to merge 1 commit into
cheng/gc-wb-3-control-planefrom
cheng/gc-wb-4-fpm-endpoint

Conversation

@ch-wan

@ch-wan ch-wan commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator

What

When --forward-pass-metrics-ipc-name is left unset, the reporter generates an
endpoint and has to hand it to external consumers; the documented contract is that
they read it back from the server config (server_arguments.mdx points the Dynamo
planner at server_args.forward_pass_metrics_ipc_name).

That readback is the scheduler's get_internal_state, which already reports
get_context().resolved_server_args_dict() — the pristine config with post-publish
overrides overlaid. So the write moves to get_context().override(...) and the read
next to it to get_observability(). The endpoint still appears in /server_info's
internal_states, and ServerArgs stops being used as a message bus between the
reporter and its consumers.

Test changes

The test's server_args stand-in was a SimpleNamespace carrying a hand-rolled
override; it now publishes a real config, so the reporter goes through the same
accessors as production, and the endpoint is asserted through the resolved config.

Validation

test/registered/unit/observability and the config ratchets pass; full registered
CPU battery shows no new failures against the base commit.

Writer ratchet 19 → 18.


CI States

Latest PR Test (Base): ❌ Run #30771345753
Latest PR Test (Extra): 🚫 Run #30784016246

@gemini-code-assist

Copy link
Copy Markdown
Contributor

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@ch-wan

ch-wan commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Part of the writer-ratchet stack: #33238#33239#33240#33241. Base is the previous PR's branch; review/merge in order.

self.assertEqual(scheduler._fpm_publisher.dp_rank, 2)
self.assertTrue(scheduler._fpm_publisher.endpoint.startswith("ipc://"))
self.assertIsNotNone(scheduler.server_args.forward_pass_metrics_ipc_name)
self.assertIsNotNone(

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] The assertion that the generated endpoint appears in get_context().resolved_server_args_dict()["forward_pass_metrics_ipc_name"] is necessary for the external-consumer contract, but it is weak as a regression guard for how the value was published. resolved_server_args_dict is vars(server_args) overlaid with _overrides_log. A regression that wrote only via server_args.override(...) (instance mutation, no bag write) would still make the key non-None through the vars path, so this assert alone would stay green. The writer ratchet is the real fence against reintroducing the ServerArgs.override call site; the unit test does not independently pin “bags written, seed pristine.”

Suggestion: Strengthen the case that must go red if the write path regresses off bags, e.g. assert both of:

@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 595924b to a489a5f Compare August 2, 2026 08:56
@ch-wan

ch-wan commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Re-review (Round 2 — a489a5f994)

Rebased on the updated control-plane tip. Production path and tests still look right:

  • write via get_context().override
  • read via get_observability().forward_pass_metrics_ipc_name
  • test pins bag leaf + resolved dict + pristine seed None

Optional non-blocking nit (unchanged): _init_fpm still gates enable_forward_pass_metrics / worker_id from server_args while the IPC name uses bags — fine after publish; can unify in a later bag-read sweep.

No open correctness issues.

@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from a489a5f to 03e797b Compare August 2, 2026 09:04
@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 03e797b to d959e89 Compare August 2, 2026 09:15
@github-actions github-actions Bot added the hicache Hierarchical Caching for SGLang label Aug 2, 2026
@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from d959e89 to 8df9807 Compare August 2, 2026 09:18

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8df9807256

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread python/sglang/srt/managers/scheduler_components/metrics_reporter.py
@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 8df9807 to 7ef5e92 Compare August 2, 2026 09:36
@ch-wan

ch-wan commented Aug 2, 2026

Copy link
Copy Markdown
Collaborator Author

Round 5 — the observation is correct but it is not a regression from this PR, so I am leaving it.

@chatgpt-codex-connector is right that with pp_size > 1 the generated endpoint never reaches the rank that answers get_internal_state. But that is unchanged by this PR: on main the write is self.scheduler.server_args.override(...), which mutates that process's ServerArgs — equally process-local — and the _init_fpm gate (attn_tp_rank == 0 and pp_rank == pp_size - 1) is untouched here. So the last PP stage recorded it and rank 0 reported None before this change too; the PR moves where the value is stored inside one process, not which process stores it.

Fixing it properly means either propagating the endpoint from the last stage to the rank that serves the readback, or deriving it deterministically instead of from NamedTemporaryFile so every rank can compute the same name. Both are behaviour changes to the FPM feature that want their own PR and a PP deployment to test against — I would rather not smuggle either into a config-plumbing change.

@chatgpt-codex-connector

Copy link
Copy Markdown

Outcome

  • No code change is needed. The PP behavior identified by the reviewer predates this PR: FPM initialization remains restricted to the last pipeline stage, where the generated endpoint is stored in that process’s runtime context. python/sglang/srt/managers/scheduler_components/metrics_reporter.pyL198-L227
  • The tokenizer-facing IPC channels are initialized only for pipeline rank 0, confirming that a multi-process propagation or deterministic endpoint-generation change would be separate behavioral work. python/sglang/srt/managers/scheduler.pyL680-L704
  • The working tree was already clean, so I did not create an empty commit or a pull request.

Checks

  • git status --short --branch
  • git diff --stat 2112497ae5d7c005492acbb28d19119654ec4604...HEAD
  • nl -ba python/sglang/srt/managers/scheduler_components/metrics_reporter.py | sed -n '190,240p'
  • nl -ba python/sglang/srt/managers/scheduler.py | sed -n '670,710p'

View task →

@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 7ef5e92 to 76bbbe2 Compare August 2, 2026 18:22
@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 76bbbe2 to 626e102 Compare August 2, 2026 18:34
@ch-wan
ch-wan force-pushed the cheng/gc-wb-4-fpm-endpoint branch from 626e102 to 78cfa10 Compare August 2, 2026 22:43
@ch-wan ch-wan added the ready-to-merge The PR is ready to merge after the CI is green. label Aug 2, 2026
… the bags

When --forward-pass-metrics-ipc-name is left unset the reporter generates an
endpoint and has to hand it to external consumers (the documented contract is
that they read it back from the server config). That readback is the scheduler's
get_internal_state, which already reports get_context().resolved_server_args_dict(),
so the write moves to get_context().override and the read alongside it to
get_observability() — the endpoint still shows up in /server_info's
internal_states, and the ServerArgs instance stops being a message bus.

The test's server_args stand-in (a SimpleNamespace with a hand-rolled override)
becomes a real published config, so the reporter exercises the same accessors as
production.

Writer ratchet 19 -> 18.
@ch-wan

ch-wan commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

Closing unmerged and reopening against main directly.

GitHub classifies a chained-base series as a stack, and in that mode it refuses base retargeting (Cannot change the base branch because the pull request is part of a stack), the classic merge API (must be merged using the asynchronous merge REST API), and the plain REST merge (403). The async endpoint accepts the request but honours branch protection, and it has no bypass parameter — so this series could not be merged in order.

The replacement PR carries the identical commit; the review history, the six rounds of comment triage and the validation notes stay here for reference. Link posted below.

@ch-wan

ch-wan commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

Reopened as #33337 (base main, identical commit). Review context stays here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hicache Hierarchical Caching for SGLang ready-to-merge The PR is ready to merge after the CI is green. run-ci run-ci-extra

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant