test(deploy): add SGLang JSON logging coverage

[knarangN]

Overview:

Add deployment coverage for SGLang structured JSONL logging through the agg_logging profile.

Details:

• add agg_logging to the post-merge and scheduled-nightly SGLang deployment matrices
• use the nightly image_tag input required by the shared deploy workflow
• verify the DYN_LOGGING_JSONL runtime configuration
• capture the frontend log baseline immediately before inference and inspect only newly emitted lines
• report validation failures explicitly with pytest.fail
• keep configuration and output checks in focused helpers
• use the unique sglang-agg-logging resource name in both API manifest versions
• integrate the Kubernetes nightly deployment infrastructure merged in ci: add Kubernetes deploy test to nightly #12090

Where should reviewer start?

1. tests/deploy/test_dgd.py for the deployment test and post-inference validation
2. tests/deploy/dgd_utils.py for structured-log validation helpers
3. tests/deploy/test_dgd_utils.py for focused helper coverage
4. .github/workflows/nightly-ci.yml and .github/workflows/post-merge-ci.yml

Related Issues

• No linked GitHub issue
• Internal tracking: OPS-7603
• Nightly dependency: ci: add Kubernetes deploy test to nightly #12090 (merged)

Validation

• Current exact head: 122b028e55a6c6e439bf4d6f97241f5ebc267842.
• The current head is conflict-free, and all seven required merge checks pass: DCO, copyright-checks, codeowners, pre-merge-status-check, backend-status-check, dynamo-status-check, and deploy-status-check.
• Exact-head trusted PR CI passed in workflow run 32909279218.
• An exact-head non-release Nightly was dispatched in workflow run 32913487029 with release=false, run_tests=true, and profiles ["agg","agg_logging"].
• The PR-specific sglang Nightly Deploy Test / agg_logging job 98017495851 passed. Its JUnit artifact reports one test, zero failures, and zero errors for tests.deploy.test_dgd::test_deployment[sglang-agg_logging].
• Runtime evidence confirms DYN_LOGGING_JSONL=1, a successful HTTP 200 inference response, and a newly emitted structured JSON log record containing the required time, level, target, and message fields. The validator found target=dynamo_llm::http::service::metrics and message=request received.
• Current review state is approved, and all eight review threads are resolved.
• The aggregate Nightly is still running and has independent failures outside the PR-specific target: the vLLM EFA operator image pull received 401 Unauthorized and entered ImagePullBackOff; separately, tests.serve.test_sglang::test_sglang_deployment[disaggregated_router-4] reported dynamo_component_requests_total count 0 instead of at least 6. Neither failing test path is changed by this PR, and neither failure exercises the successful agg_logging deployment test.
• Evidence boundary: the passing exact-head target job validates the SGLang JSONL logging deployment path; it does not claim that independent Nightly lanes passed.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[knarangN]

/ok to test a866864

[coderabbitai]

Walkthrough

The SGLang deployment workflows now run the agg_logging profile. Deployment tests validate its JSONL configuration and inspect frontend logs for a structured Dynamo record containing required fields.

Changes

SGLang logging validation

Layer / File(s)	Summary
Enable agg_logging deployment profile .github/workflows/post-merge-ci.yml, .github/workflows/pr.yaml	Adds agg_logging to the SGLang deployment profiles used by post-merge and pull-request workflows.
Validate structured JSONL output tests/deploy/test_deploy.py	Adds JSONL parsing helpers and verifies logging configuration plus emitted frontend records for agg_logging.

Estimated code review effort: 2 (Simple) | ~10 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly summarizes the primary change: adding SGLang JSON logging deployment coverage.
Description check	✅ Passed	The description includes the required overview, details, reviewer starting points, related-issue status, and validation evidence. It accurately describes the deployment coverage and its verification.

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tests/deploy/test_deploy.py`:
- Around line 264-276: Update the agg_logging validation around
frontend_pod.logs and find_structured_json_log so it captures the frontend log
baseline before the inference request, then searches only log output emitted
afterward. Keep the existing structured-record assertion and logger details, but
ensure startup or health-check records cannot satisfy the inference-time logging
check.
- Around line 190-195: Replace the assert in the agg_logging branch of the
deployment test with an explicit pytest.fail(...) or raised exception when
logging_config["jsonl_enabled"] is false. Also update the related structured-log
check below to use explicit failure handling, ensuring both runtime validations
execute under python -O.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fe02a3a5-6bcb-4e49-965d-7bf321e362cf

📥 Commits

Reviewing files that changed from the base of the PR and between d880803 and a866864.

📒 Files selected for processing (3)

• .github/workflows/post-merge-ci.yml
• .github/workflows/pr.yaml
• tests/deploy/test_deploy.py

[datadog-official]

⚠️ Warnings

🚦 2 Pipeline jobs failed

Pre Merge | pre-commit     

🔧 Fix in code. This looks caused by changes in this PR.

Undefined name 'BaseConfigModifier' in components/src/dynamo/profiler/tests/unit/test_profiler_protocol.py:861:24

Pre Merge | pre-merge-status-check     

📋 Copy prompt for your agent

CI on my pull request is failing. Help me find and fix the root cause of each failing job below — they were flagged as caused by changes in this PR, so focus on the diff. For each job, explain the failure and propose a fix.

Branch: tests/ops-7603-agg-logging

Pre Merge | pre-commit
Commit: c7b7e27cd70d086a66d3f89c994498f19bddb523
Error (code / build):
Undefined name 'BaseConfigModifier' in components/src/dynamo/profiler/tests/unit/test_profiler_protocol.py:861:24
CI job: https://github.com/ai-dynamo/dynamo/actions/runs/30846701442/job/91796654256

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: c7b7e27 | Docs | Datadog PR Page | Give us feedback!}

[knarangN]

/ok to test be5e9f1

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

[knarangN]

/ok to test 1f855e5

[datadog-official]

⚠️ Warnings

⚠️ Your PR has warnings. Tests failed — unable to determine if these are from required jobs.

🧪 13 Tests failed in 1 job

Nightly CI Pipeline | test — ❌ 13 tests failed

View in GitHub Actions

❌ test_router_decisions_trtllm_disagg[nats] from test_router_e2e_with_trtllm.py  

Timeout waiting for workers. Found 0 instance(s), expected 2

❌ test_trtllm_indexers_sync[nats-tcp] from test_router_e2e_with_trtllm.py  

Timeout (>150.0s) from pytest-timeout.

❌ test_trtllm_kv_router_basic[tcp] from test_router_e2e_with_trtllm.py  

Timeout waiting for workers. Found 0 instance(s), expected 2

❌ test_sglang_deployment[diffusion_t2v_wan_1_3b-4] from test_sglang.py  

Timeout (>420.0s) from pytest-timeout.

❌ test_sglang_deployment[disaggregated_router-4] from test_sglang.py  

dynamo_component_requests_total has count 0 which is less than required 6

↳ and 8 more — View all

ℹ️ Info

No other issues found (see more)

❄️ No new flaky tests detected

🔄 Datadog retried 1 test - 1 passed on retry

Useful? React with 👍 / 👎

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 122b028 | Docs | View more details | Give us feedback!}

[knarangN]

@ai-dynamo/dynamo-operator-codeowners @ai-dynamo/dynamo-backend-sglang-codeowners — review requested for PR #12085.

Current exact head 1f855e5 is conflict-free. All seven required merge checks pass, all five review threads are resolved, and the exact-head sglang Nightly Deploy Test / agg_logging job passed:
https://github.com/ai-dynamo/dynamo/actions/runs/32416962238/job/96584009321

The aggregate Nightly workflow is red in other lanes, but the PR-specific target job passed. Please review and approve if the change is acceptable.

[knarangN]

/ok to test 122b028