Skip to content

ci(nightly): open the Slack thread at nightly start, like dynamo - #353

Merged
pvijayakrish merged 2 commits into
mainfrom
pvijayakrish/nightly-slack-thread
Sep 30, 2026
Merged

pvijayakrish merged 2 commits into
mainfrom
pvijayakrish/nightly-slack-thread

Conversation

@pvijayakrish

Copy link
Copy Markdown
Contributor
  • slack-thread-start posts dynamo's root message as soon as the run is cleared to reach the GitLab trigger, instead of after the build.
  • slack-staging-reply posts "Staging complete" with the wheel and crate versions.
  • trigger-gitlab-security replies "GitLab release pipeline triggered" and forwards the root ts, so GitLab replies land in the same thread.
  • slack-thread-stopped closes the thread when a later gate stops the run.

Why and what changed

Review map

  • Risk level:
  • Start with:
  • Public or serialized contract changed:
  • Compatibility or rollback concern:

Evidence

  • Tests:
  • Fast CI:
  • Full CI:
  • CodeRabbit reviewed commit:
  • Codex reviewed commit:
  • Negative or boundary cases:
  • Expected-value derivation (for exact numerical assertions):
  • Before/after output, trace, benchmark, or golden diff:

Modeling or data provenance

Tracking

  • Closes:
  • Related PRs or issues:

- slack-thread-start posts dynamo's root message as soon as the run is
  cleared to reach the GitLab trigger, instead of after the build.
- slack-staging-reply posts "Staging complete" with the wheel and crate
  versions.
- trigger-gitlab-security replies "GitLab release pipeline triggered"
  and forwards the root ts, so GitLab replies land in the same thread.
- slack-thread-stopped closes the thread when a later gate stops the run.

Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
@pvijayakrish
pvijayakrish requested review from a team as code owners September 30, 2026 05:14
@coderabbitai

coderabbitai Bot commented Sep 30, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository: ai-dynamo/aisimulate/.coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 6c575fed-1c63-4bcd-9b27-552e84409b48

📥 Commits

Reviewing files that changed from the base of the PR and between e66bd87 and d355254.

📒 Files selected for processing (2)
  • .github/workflows/nightly-ci.yml
  • tests/test_ci_workflow_contracts.py
🔗 Linked repositories identified

CodeRabbit considers these linked repositories for cross-repo context during reviews:

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

📜 Recent review details
⏰ Context from checks skipped due to timeout. (22)
  • GitHub Check: Prediction Regression / Collect prediction snapshot (new)
  • GitHub Check: Prediction Regression / Collect prediction snapshot (old)
  • GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_aarch64)
  • GitHub Check: Platform Wheels / Build wheels (manylinux_2_28_x86_64)
  • GitHub Check: Collector Data / Perf data sanity (informational)
  • GitHub Check: Collector Data / Check collector data
  • GitHub Check: Platform Wheels / Build wheels (macosx_arm64)
  • GitHub Check: Public API Rust (amd64)
  • GitHub Check: Application Test Wheel (arm64)
  • GitHub Check: Release Artifact Contract (arm64)
  • GitHub Check: Release Artifact Contract (amd64)
  • GitHub Check: Public API Rust (arm64)
  • GitHub Check: Application Test Wheel (amd64)
  • GitHub Check: Python Dependency Licenses
  • GitHub Check: Python 3.11 compatibility
  • GitHub Check: Engine Golden Regression
  • GitHub Check: Rust feature modes
  • GitHub Check: Cargo Deny (workspace, arm64)
  • GitHub Check: Python 3.13 compatibility
  • GitHub Check: Rust (arm64)
  • GitHub Check: Cargo Deny (workspace, amd64)
  • GitHub Check: Rust (amd64)
🧰 Additional context used
📓 Path-based instructions (5)
Only root workflows are active.

⚙️ CodeRabbit configuration file

Files:

  • .github/workflows/nightly-ci.yml
Require coverage of the changed behavior and its negative or boundary cases.

⚙️ CodeRabbit configuration file

Files:

  • tests/test_ci_workflow_contracts.py
Read REVIEW.md before commenting.

⚙️ CodeRabbit configuration file

Files:

  • tests/test_ci_workflow_contracts.py
Before making any change under: `python/aisimulate/src/aisimulate/generator/**` MUST read: `python/aisimulate/.claude/rules/generator-development.md` Before making any change under `python/aisimulate/collector/**` MUST read: `python/aisimul...

📄 CodeRabbit inference engine (AGENTS.md)

Files:

  • tests/test_ci_workflow_contracts.py
Source excerpt: Only workflows under the repository-root `.github/workflows/` run for this repository.

📄 CodeRabbit inference engine (REVIEW.md)

Files:

  • tests/test_ci_workflow_contracts.py
🪛 ast-grep (0.45.3)
tests/test_ci_workflow_contracts.py

[error] 2829-2850: Command coming from incoming request
Context: subprocess.run(
["bash", "-e", "-c", step["run"]],
cwd=tmp_path,
capture_output=True,
text=True,
env={
**os.environ,
"PATH": f"{fake_bin}:{os.environ['PATH']}",
"CAPTURE_REQUEST": str(capture),
"CURL_EXIT_CODE": "22" if failure == "http-error" else "0",
"GITLAB_TRIGGER_TOKEN": "" if failure == "missing-token" else "fixture-token",
"GITLAB_PIPELINE_URL": endpoint,
"WHEEL_VERSION": "0.12.0.dev202609170000001234",
"TOOLING_SHA": "b" * 40,
"GH_RUN_ID": "123456",
"GH_SHA": "a" * 40,
"SLACK_THREAD_TS": "1234567890.123456",
"SLACK_CHANNEL_ID": "fixture-channel",
"RUNNER_TEMP": str(tmp_path),
"GITHUB_OUTPUT": str(tmp_path / "output"),
},
)
Note: [CWE-78] Improper Neutralization of Special Elements used in an OS Command ('OS Command Injection').

(subprocess-from-request)

🔀 Multi-repo context ai-dynamo/dynamo, ai-dynamo/aiconfigurator

Linked repositories findings

ai-dynamo/dynamo

  • nightly-ci.yml:157-223 creates a thread_ts output only for eligible release-mode nightlies. Slack posting is best-effort (continue-on-error), and downstream jobs must tolerate an empty timestamp. [::ai-dynamo/dynamo::]

  • nightly-ci.yml:1383-1422 includes the Slack-start job in needs, but uses !cancelled() and explicit build-result gates so a failed Slack notification does not suppress the release. [::ai-dynamo/dynamo::]

  • release.yml:27-31, 1024-1028, 1123-1151 defines the cross-workflow contract: slack_thread_ts is optional, staging replies use always(), and the GitLab trigger forwards the timestamp and channel to GitLab. [::ai-dynamo/dynamo::]

  • release.yml:1243-1249 posts the GitLab reply only when a non-empty timestamp exists and treats Slack failure as non-blocking. [::ai-dynamo/dynamo::]

  • The reference workflow has no dedicated stopped-thread job; the PR’s stopped notice is additional behavior rather than an existing Dynamo-compatible contract. [::ai-dynamo/dynamo::]

ai-dynamo/aiconfigurator

  • The checked-out repository contains no nightly-ci.yml, Slack-thread, or GitLab-trigger contract matching these changes; no downstream compatibility references were found. [::ai-dynamo/aiconfigurator::]
🔇 Additional comments (2)
.github/workflows/nightly-ci.yml (1)

210-275: LGTM!

Also applies to: 948-1023, 1032-1032, 1055-1060, 1072-1076, 1092-1183, 1194-1194

tests/test_ci_workflow_contracts.py (1)

2242-2244: LGTM!

Also applies to: 2277-2299, 2848-2849, 2859-2859


📝 Summary

Risk: Medium. The three areas needing the most human attention are workflow job conditions and dependencies, Slack credentials and best-effort posting, and GitLab trigger response handling.

  • The workflow now posts a Slack root message after nightly eligibility and approval checks, before the build completes. The GitLab trigger waits for this job and receives its thread-ts output.
  • Staging results, including available wheel and crate versions, are posted as thread replies. The GitLab trigger response is saved to gitlab-trigger.json; its pipeline ID and URL, plus the nightly tag, are posted in the thread. A stopped reply is sent when the GitLab trigger does not succeed.
  • Public workflow contract: slack-thread-start exports thread-ts for downstream replies. The added contract tests check job conditions and dependencies, timestamp forwarding, and trigger-response file handling.
  • Technical quality evidence: source and test changes support the described behavior. Slack posting steps use continue-on-error, so notification failures do not fail their containing jobs.
  • Merge readiness: test execution results and other validation evidence were not supplied. Current review severity counts are unavailable; the bot-generated summaries are not approval.

Walkthrough

The nightly workflow now posts Slack thread updates for eligible runs. It reports staging artifact results and GitLab pipeline details, or posts a stopped notice when the GitLab trigger does not succeed.

Changes

Nightly Slack notifications

Layer / File(s) Summary
Start the thread and report staging
.github/workflows/nightly-ci.yml, tests/test_ci_workflow_contracts.py
The workflow posts a nightly start message and exports its Slack timestamp. The staging reply uses that timestamp to report artifact results. The retry-condition test includes the start job.
Report GitLab trigger outcomes
.github/workflows/nightly-ci.yml, tests/test_ci_workflow_contracts.py
The GitLab trigger uses the Slack timestamp, saves its response to a temporary JSON file, and exposes the nightly tag. Slack replies report pipeline details or a stopped notice. Contract tests check job dependencies, timestamp wiring, and the response file output.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 9bc7c

The nightly notifications preserve release triggering when Slack posting fails and close existing threads when later gates stop the run. No merge-blocking issue remains; live Slack and GitLab posting has not been validated.

🚥 Pre-merge checks | ✅ 4 | ❌ 3 | ❓ 1

❌ Failed checks (3 warnings, 1 inconclusive)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description states the intended Slack-thread behavior, but the required template fields remain empty. Review risk, starting points, contract impact, compatibility concerns, test results, reviewed … Complete the template. Add the problem and intended consumer under Why and what changed, fill the Review map, provide exact validation commands and results under Evidence, write N/A or provenance details under Modeling or data provenance, a…
Cross-Layer Contract ⚠️ Warning The PR adds new cross-layer CI contracts without complete consumer tests or documentation. .github/workflows/nightly-ci.yml exports slack-thread-start.outputs.thread-ts, forwards it to the GitLab … Add executable workflow-contract tests with a fake curl/Slack API. Cover root-message success and missing ts, staging success/failure version formatting, GitLab response parsing and nightly-tag output, Slack API failures, and the stoppe…
Modeling And Data Evidence ⚠️ Warning The PR changes nightly selection and gating logic. It adds the slack-thread-start eligibility predicate, changes job dependencies, and gates Slack replies on workflow results in `.github/workflows/n… Add reproducible evidence for the changed nightly selection behavior. Include the source or provenance for the expected gate rules, plus an explained before/after comparison or independent oracle. A machine-readable scenario matrix should c…
Review Evidence ❓ Inconclusive The PR description does not provide commands, results, negative or boundary validation, or distinctions between local, fake-runner, parity, and production evidence. It also does not cite an inactive n… Update the PR description with exact validation commands and results, negative or boundary cases for the Slack and workflow gates, and clear labels for local, fake-runner, parity, and production evidence. State any reusable workflow's activ…
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title precisely states the behavioral change: opening the Slack thread at nightly start. The extra phrase does not make the title vague or unrelated.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Compatibility Boundaries ✅ Passed PASS. The pull request changes only nightly workflow orchestration and workflow contract tests. It does not change Python, Rust, binding, schema, or public API sources. The unchanged release builder s…
Full details: Description check

Explanation

The description states the intended Slack-thread behavior, but the required template fields remain empty. Review risk, starting points, contract impact, compatibility concerns, test results, reviewed commits, boundary cases, and tracking information are missing.

Resolution

Complete the template. Add the problem and intended consumer under Why and what changed, fill the Review map, provide exact validation commands and results under Evidence, write N/A or provenance details under Modeling or data provenance, and complete Closes and Related PRs or issues.

Full details: Cross-Layer Contract

Explanation

The PR adds new cross-layer CI contracts without complete consumer tests or documentation. .github/workflows/nightly-ci.yml exports slack-thread-start.outputs.thread-ts, forwards it to the GitLab trigger, writes gitlab-trigger.json, and adds Slack start, staging, GitLab-reply, and stopped-reply paths. tests/test_ci_workflow_contracts.py checks YAML wiring and multipart arguments only. It does not execute any new Slack run block or the GitLab reply parser, and its fake curl does not create gitlab-trigger.json. docs/ci.md is unchanged and documents only the request fields, not the new Slack thread lifecycle or the GitLab response fields (id, web_url).

Resolution

Add executable workflow-contract tests with a fake curl/Slack API. Cover root-message success and missing ts, staging success/failure version formatting, GitLab response parsing and nightly-tag output, Slack API failures, and the stopped-thread condition. Assert the complete payload fields and forwarded thread-ts. Update docs/ci.md to describe the new Slack jobs, their gates, best-effort behavior, thread timestamp propagation, and the GitLab response contract.

Full details: Modeling And Data Evidence

Explanation

The PR changes nightly selection and gating logic. It adds the slack-thread-start eligibility predicate, changes job dependencies, and gates Slack replies on workflow results in .github/workflows/nightly-ci.yml. The added tests are binary contract assertions only. The PR description supplies no provenance, explained before/after result, independent oracle, golden diff, or machine-readable anomaly-checked summary. This fails the required evidence standard for selection logic.

Resolution

Add reproducible evidence for the changed nightly selection behavior. Include the source or provenance for the expected gate rules, plus an explained before/after comparison or independent oracle. A machine-readable scenario matrix should cover schedule, manual approval, retry attempt, branch, build guard, trigger flag, skipped/failing dependencies, and Slack failure cases, with anomaly checks and recorded results.

Full details: Review Evidence

Explanation

The PR description does not provide commands, results, negative or boundary validation, or distinctions between local, fake-runner, parity, and production evidence. It also does not cite an inactive nested workflow as hosted CI evidence. The available description therefore does not establish the pass criteria or the stated fail condition.

Resolution

Update the PR description with exact validation commands and results, negative or boundary cases for the Slack and workflow gates, and clear labels for local, fake-runner, parity, and production evidence. State any reusable workflow's activation mode and hosted CI result explicitly.


Comment @coderabbitai help to get the list of available commands.

@Harrilee Harrilee left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The gating looks right: slack-thread-start uses the same approval/attempt/GITLAB_SECURITY_TRIGGER_ENABLED gate as trigger-gitlab-security. The trigger job keeps !cancelled() and never checks the Slack job's result, so a Slack outage still can't block the GitLab trigger. The follow-up jobs skip cleanly when thread-ts is empty, and the crate version derivation (X.Y.Z.devN → X.Y.Z-dev.N) matches the versioning contract in the header. The GitLab response now goes to a file and is never echoed, so the internal pipeline URL stays out of the public log. I ran the nightly/gitlab contract tests locally on d355254: 45 passed.

A few non-blocking nits:

  1. Slack post boilerplate is repeated 4 times. The same curl chat.postMessage + .ok check block appears in slack-thread-start, slack-staging-reply, the GitLab reply step, and slack-thread-stopped. Consider pulling it into a small script or composite action so later fixes (retry flags, error handling) only have to happen in one place.
  2. Extra automated-release deployments. Every new Slack job binds environment: automated-release just to read the Slack secrets, so each nightly now creates about 4 extra deployment records for that environment. That's fine if intended. Otherwise, moving the two Slack secrets to repo level (like SLACK_NOTIFY_NIGHTLY_WEBHOOK_URL) would avoid it.
  3. Silent Slack failures. Every Slack step uses continue-on-error: true, so notify-slack never flags a broken Slack thread; it only shows up as an ::error:: annotation. That's acceptable for best-effort, but worth knowing.
  4. slack-thread-stopped wording. On a cancelled run, or when the GitLab trigger itself fails, it still says "Nightly stopped before the GitLab pipeline". Including needs.trigger-gitlab-security.result in the text would make the thread easier to triage.
  5. Internal GitLab URL in Slack. The trigger reply now posts the internal GitLab pipeline URL to Slack. Please confirm that SLACK_RELEASE_CHANNEL_ID is an internal-only channel.
  6. PR description. The template sections (risk level, Evidence, Tracking) are still empty. Please fill in at least the risk level and the test evidence.

@pvijayakrish
pvijayakrish enabled auto-merge (squash) September 30, 2026 16:32
@pvijayakrish
pvijayakrish merged commit 91ea673 into main Sep 30, 2026
68 checks passed
@pvijayakrish
pvijayakrish deleted the pvijayakrish/nightly-slack-thread branch September 30, 2026 16:47
tianhaox added a commit to tianhaox/aisimulate that referenced this pull request Oct 1, 2026
Brings in the FPM decoupling / self-service onboarding stack (ai-dynamo#238, ai-dynamo#248,
ai-dynamo#347), the output adapters (ai-dynamo#334) and the CI changes (ai-dynamo#349, ai-dynamo#351, ai-dynamo#353,
ai-dynamo#354, ai-dynamo#330, ai-dynamo#319).

Conflict resolutions:

- ENGINE_SPEC_SCHEMA_VERSION: upstream claimed 25 for the FPM decoupling
  selector; decode CP is renumbered to 26 (positional dcp_size tails on the
  attention / MLA / DSA ops). Stale-payload loops reject 20..25; the 25
  payload keeps the selector like the decoupling branch's 21.
- cp_size: upstream added a CP1-only `cp_size` to compile_engine,
  estimate_kv_cache / estimate_num_gpu_blocks, EngineBuildRequest and the
  legacy Rust compile path ("this SDK entry point does not support context
  parallelism"). This branch supports prefill CP at exactly those entry
  points, so the duplicate parameters / struct field are folded into ours,
  the CP1 gates become positive-integer validation, and the FPM profile cell
  selection receives the real cp_size (a profile without that cell fails
  loud with "no matching FPM deployment profile"). Upstream's tests are
  adjusted accordingly.
- FpmCompileConfig / AicTimingConfig parallel-shape checks combine
  upstream's `fpm_profile.is_none()` exemption with the `* cp_size` fold;
  aic_capacity_kwargs gains cp_size / dcp_size; fpm best_available keeps
  upstream's registered-architecture check ahead of the DCP mode gate.
- capacity.py worker resolution carries aic_cp_size / aic_dcp_size next to
  aic_fpm_profile / worker_type.
- ParallelismPresetConfig.prefill_context becomes Optional (None = 1), like
  decode_context, so default parallelism dumps carry no CP keys; the new
  onboarding topology tests (ai-dynamo#248) compare those dumps against the
  six-key request parallelism. Consumers read it through compiler._prefill_cp.

Signed-off-by: Tianhao Xu <tianhaox@nvidia.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
carlory pushed a commit to carlory/aisimulate that referenced this pull request Oct 30, 2026
…dynamo#353)

- slack-thread-start posts dynamo's root message as soon as the run is
  cleared to reach the GitLab trigger, instead of after the build.
- slack-staging-reply posts "Staging complete" with the wheel and crate
  versions.
- trigger-gitlab-security replies "GitLab release pipeline triggered"
  and forwards the root ts, so GitLab replies land in the same thread.
- slack-thread-stopped closes the thread when a later gate stops the run.

Signed-off-by: pvijayakrish <pvijayakrish@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants