ci(gfx1250): email a test report after every nightly by archana-ramalingam · Pull Request #13620 · ROCm/rocm-libraries

archana-ramalingam · 2026-10-14T10:09:38Z

JIRA ID : N/A

Motivation

Email a test report after every gfx1250-strict nightly.

Technical Details

Report step: the test job's last step before cleanup runs the new .github/scripts/blas_nightly_report.py (standard library only) and emails through a Power Automate flow (secret GFX1250_BLAS_NIGHTLY_WEBHOOK_URL).
- Sends on the nightly however the tests ended, and on a manual run with send_report; never on PRs. A step, so PRs show no skipped check. No runner, no email; a failed send doesn't fail the job.
- Step results come from the workflow (steps.<id>.outcome, job.status); per-test results from the test logs.
- Email: banner, run metadata, per-suite counts, and a per-test comparison with the latest nightly in 7 days (new / existing / fixed / skipped or removed). Test lists at the end under Failures, tagged new / timeout / crashed. Lists read hipBLASLt: / (failing), one test per line; the comparison's age counts from this run's start, not from when the email is sent.
- JSON for the flow: outcome (passed / setup_failed / tests_failed / tests_incomplete / cancelled), failed_step, steps, per-test status (failing / timeout / crashed).
Workflow: send_report input, actions: read (previous nightly's results), .github/scripts in the sparse checkout, step ids.

Test Plan

Real nightly and PR artifacts, plus crafted logs: alarm-timeout shard, crash mid-test, pytest timeout, no GPU, cancelled job, no previous nightly.
Manual run of this branch with send_report=true.
Failed send: posted to an unused local port, so nothing is emailed.
--steps as the workflow builds it, including steps that never started.
A log whose first pytest session dies before its summary: the later session keeps its own results (the old parser reported them under the first suite).

Test Result

Email delivered (HTTP 202) from https://github.com/ROCm/rocm-libraries/actions/runs/38626703691: ~68k hipBLASLt, 111 TensileLite common, 264 rocisa passed.
Every crafted case gets the right outcome, failed_step and per-test status, e.g. "5 failing tests (2 timeout, 1 crashed)".
Failed send logs only Report not sent (ConnectionRefusedError), no URL.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests.

Risk level

Low: CI-only. The step runs only on the nightly or a manual run with send_report.

🤖 Generated with Claude Code

Add a report job and .github/scripts/gfx1250_nightly_report.py. After every nightly (or a manual run with send_report), it builds an HTML report and POSTs it to the Power Automate flow in the GFX1250_BLAS_NIGHTLY_WEBHOOK_URL secret, which emails it. GFX1250_BLAS_SYSTEM_LABEL (optional) fills the System row. Without the webhook secret the job skips with a notice. - Contents: status banner and subject (✅/❌), run metadata, per-suite counts with failing tests, and a per-test comparison with the most recent nightly that has results (within 7 days): new, existing, fixed (fail -> pass) and skipped or removed. It warns when that nightly isn't last night's. - Runs on a GitHub-hosted runner from the test job's artifact and outputs, so a report goes out even if the test job failed, timed out or never started, and the secret never reaches the self-hosted runner. Follow-ups from the #12983 review and the first per-PR runs: - Gate: add cmake/ (in the sparse checkout; rocisa and stinkytofu include cmake/modules/default_amdclang.cmake). - One-GPU check: if the runner's .env doesn't set ROCR_VISIBLE_DEVICES, use N from the runner name's -gpuN suffix, with a warning. Drop HIP_VISIBLE_DEVICES, which is never forwarded into the container. - SDK pin 10.2.0a20261007 -> 10.2.0a20261013, whose compiler (amd-llvm c7ceccf9a5) matches per-PR CI's again. The pin moves to workflow-level env for both jobs. - upload-artifact v4.6.2 -> v7.0.2 (Node 20 is deprecated); runners report 2.337.0, above v7's 2.327.1 minimum. Co-Authored-By: Claude <noreply@anthropic.com>

therock-pr-bot · 2026-10-14T10:16:50Z

❌ PR Check — Action Required

Check	Status	Details
📝 PR Description	❌ Fail	Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword. Expected: include a `JIRA ID` / `ISSUE ID` line (separator `:` or `-`, or omitted; value may be a JIRA key, a number with/without `#`, or a link), OR a closing keyword + issue reference. Accepted examples: • `JIRA ID : TESTAUTO-6039` • `JIRA ID - #330` • `JIRA ID #330` • `JIRA ID` (on separate line) `ROCM-25757` • `ISSUE ID : TESTUTO-3334` • `ISSUE ID #3334` • `ISSUE ID - TESTAUTO-3433` • `ISSUE ID` (on separate line) `AIRUNTIME-2352` • `ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234` • `Closes #10` • `Fixes octo-org/octo-repo#100` • `Resolves: #123` • `#123` • `https://github.com/<org_name>/<repo_name>/issues/123` Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files	✅ Pass	—
🧪 Unit Test	✅ Pass	—
🔎 pre-commit	⏳ Pending	⏳ Still running…
🚫 Draft PR	🔜 To Be Enabled	—
🚩 Feature Flag	🔜 To Be Enabled	—
📊 Code Coverage	🔜 To Be Enabled	—

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

therock-pr-bot · 2026-10-14T10:16:55Z

Pre-commit check failed

⛔ pre-commit failed

Please run locally:

python -m pip install pre-commit
pre-commit install
pre-commit run --all-files --show-diff-on-failure

This repo uses .pre-commit-config.yaml.

therock-pr-bot · 2026-10-14T10:22:23Z

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

A skipped "Nightly report" job showed on every gated PR. Move it to gfx1250_nightly_report.yml, which starts when a develop run of the test workflow finishes (workflow_run) and acts only on nightly runs, so PRs get no report check. It keeps the same guarantees: GitHub-hosted (the webhook secret never reaches the gfx1250 runner), and a report even when the test job failed, timed out or never started. - It reads the run's results by run ID and its runner and step results from the jobs API; the script takes REPORT_* overrides for the reported run's ID, commit, branch and event. - The test job records its SDK version in the artifact (test_results/rocm_sdk_version.txt). - Nightly runs only: no manual trigger (the send_report input is gone), so manual runs of the test workflow don't email. - The trigger matches the test workflow's name; a comment there says to rename both together. - Drop the test job's outputs and the step IDs only the report used. Co-Authored-By: Claude <noreply@anthropic.com>

Replace gfx1250_nightly_report.yml with a report job in gfx1250_hipblaslt_tensilelite_test.yml that calls a new reusable workflow, nightly_test_report.yml, after the test job. It runs for the nightly, however the test job ended, on a GitHub-hosted runner, and reads the artifact from the same run. A manual run with send_report (no tests needed) sends it too, to try the webhook; a follow-up commit removes that once tested. - nightly_test_report.yml inputs: title, test_job, artifact, suites, and optional meta and strip; secrets webhook_url (skips with a notice when unset) and system_label. - The script, renamed to nightly_test_report.py, takes everything project-specific from options. Suites are gtest log globs or the Nth pytest session of a log; --meta LABEL=@path reads a value from the results. It finds the previous nightly from the run's own workflow. - The report warns when the test job or the artifact isn't in the run, rather than silently dropping rows. Steps are listed by their names, so a renamed step can't vanish from the email. - It now fails when a listed suite didn't run or the test job was skipped; before, one suite with results was enough for PASSED. - Email layout: one heading per suite (no TensileLite group), and Architecture and ROCm SDK now follow Commit ID. Co-Authored-By: Claude <noreply@anthropic.com>

rocm is a source-only package, so pip builds it and stores the built wheel in the cache, which is mounted read-only outside the nightly. With the new SDK pin nothing was cached yet, and every PR and manual run failed at "Read-only file system: '/pip-cache/wheels/...'" (develop dispatch run 38544746130). Install rocm itself with --no-cache-dir and --no-deps, then the extras, whose multi-GB wheels still come from the cache. Co-Authored-By: Claude <noreply@anthropic.com>

Remove the send_report input now that a manual run has confirmed the email end to end: manual runs of the test workflow no longer email. The report job runs only for the nightly; re-running its "Nightly report" job re-sends a night's report. Co-Authored-By: Claude <noreply@anthropic.com>

The report job showed as a skipped check on every PR. The report is now the test job's last step before cleanup, calling the script directly; the reusable workflow is removed. It runs on the nightly, and on a manual run with send_report. continue-on-error: a send failure doesn't fail the tests. With --job-status the script reads this job's finished steps from the API (the container has no gh or jq); the webhook URL comes from WEBHOOK_URL, so it stays off the command line, and a failed send prints only the HTTP status. Nothing is sent when the job never gets a runner. The payload tells failures apart for the flow: outcome (passed / setup_failed / tests_failed / tests_incomplete / cancelled), failed_step, steps, and per-test status timeout or crashed. A gtest shard that stops in a test (the client's alarm timeout, or a crash) used to drop that test and the rest of the shard silently; that test is now listed. All test lists sit at the end of the email under Failures, tagged new / timeout / crashed. Cleanup: no global state (the timeout/crash tags are passed in), the unused --warn and --meta @path are gone (the SDK version comes from the job's environment), files are closed. Co-Authored-By: Claude <noreply@anthropic.com>

geomin12 · 2026-10-15T17:08:09Z

pls make this file name specific to blas, this is too generally sounding

Done: renamed to .github/scripts/blas_nightly_report.py.

geomin12 · 2026-10-15T17:09:36Z

seems mostly AI generated, and it's mostly html created. does it work?

Yes, it works.

Run 38626703691, a manual run of this branch, sent the email through the flow with nightly comparison.

I ran the same script locally on last night's develop nightly, 38675731057 + compared it with 10-14's.
Reported 68,040 hipBLASLt, 111 TensileLite common and 264 rocisa passes.
It also flagged 8 hipBLASLt tests as removed, which #13214 did.
Setup flow (temporarily) to send you that email.

geomin12 · 2026-10-15T17:10:38Z

+        sha_full[:6],
+    )
+    kind = "Nightly" if env.get("GITHUB_EVENT_NAME") == "schedule" else "Manual"
+    run_url = f"{server}/{repo}/actions/runs/{env.get('GITHUB_RUN_ID', '')}"


not sure why we are scraping it. why can't we just use the data from previous workflow step?

Step results: agreed, done. They come from steps.<id>.outcome and job.status; no more API calls.

Per-test results: steps only give pass/fail per step, so per-test lists need the test output. The logs are already scrubbed and uploaded, and last night's comparison reads them too. JUnit XML would need its own scrubbing of HW sensitive info, and a crashed shard writes none.

geomin12 · 2026-10-15T17:11:25Z

a lot of these changes are not relevant to email. let's keep those out and add it as separate PR

Agreed: moved to #13771

…il only - Rename the script to .github/scripts/blas_nightly_report.py. - Step results come from the workflow (steps.<id>.outcome, job.status) through --steps; the script no longer reads them from the GitHub API. A step that never started counts as skipped. - Keep this PR to the email report. Co-Authored-By: Claude <noreply@anthropic.com>

…line The "latest nightly with results was N nights back" warning measured from when the report was sent, so a nightly that waited hours for a runner (or a report sent later) claimed 2 nights for last night's run. It now measures from this run's start and ignores later runs. The warning and the test lists put one item per line (<br>); some mail views ran the <ul> items together. Lists read "<suite>:" then "(failing)" etc., and the warning drops the "nightlies with no results" line. Co-Authored-By: Claude <noreply@anthropic.com>

The TensileLite log holds two pytest sessions (rocisa, then the common tests), picked by position. Sessions were split at their summary lines, so a session that died before its summary merged into the next one and its results were reported under the wrong suite. Sessions now start at pytest's "test session starts" line: an unfinished session keeps its place and is reported as not run. Co-Authored-By: Claude <noreply@anthropic.com>

geomin12

lgtm for temporary notification and efforts

this workflow and work is definitely pretty hacky. ideally, we can make this dynamic so when we get new architectures, we should not have to create these "one off" workflow to test

just to keep in mind. after this lands, we should re-work it. TheRock CI integrates quite well with this to be honest

geomin12 · 2026-10-16T22:18:58Z

let's make this more generic in a future PR. same for email feature, i feel lots of identical features are in this repo and we should condense and re-use components when we can

Posting for clarification:
We have this standalone workflow to address gfx1250-strict testing gap, necessary as TheRock wasn’t building strict artifacts and this was P1 to be enabled asap per-PR.
Will follow up on integrating to TheRock infra once this is available.
Emailing everyday isn’t already a feature, we only send emails on failures, hence the need for custom script.
@amd-shiraz

amd-shiraz · 2026-10-20T23:17:59Z


+      # Before the hand-back, which also covers the previous results it downloads.
+      - name: Email the test report
+        if: ${{ always() && (github.event_name == 'schedule' || inputs.send_report) }}


if we somehow manually or re-trigger with run_hipblaslt=false (or run_tensilelite=false) but leave send_report=true, the skipped tests will show up as setup_failed in the report. This will cause a X FAILED email and ok=false in the webhook even though the run was intentionally partial.

…ite timeout; cleanup (#13771) JIRA ID : N/A ## Motivation Follow-ups from the #12983 and #13464 reviews, split out of #13620, plus a fix for hipBLASLt tests running ~7x slower since per-PR runs started. ## Technical Details - **hipBLASLt test speed:** `OMP_NUM_THREADS` and `OPENBLAS_NUM_THREADS` = host CPUs / (4 runners x 4 shards); `BLIS_NUM_THREADS` = the same, at most 8 (hipBLASLt's own BLIS default, which setting `OMP_NUM_THREADS` turns off). By default the CPU reference's OpenMP and OpenBLAS pools each start one thread per host CPU, so 4 runners x 4 shards oversubscribe the host 16x per pool: since per-PR runs started (10-12 to 10-15), the step took ~80 min when other runners were testing vs ~15 min alone, up to 110 min of its 120-min timeout. - **Gate:** add `cmake/` (rocisa and stinkytofu include `cmake/modules/default_amdclang.cmake`). - **One-GPU check:** runs first in its step, under `set -euo pipefail`. `ROCR_VISIBLE_DEVICES` comes from the runner's `.env`; if it is unset, N comes from the runner name's `-gpuN` (the pool registers runner N as `...-gpuN` with `ROCR_VISIBLE_DEVICES=N`), with a warning. It must be a single GPU index (`^[0-9]+$`), or the job fails before the build. Drop `HIP_VISIBLE_DEVICES` (never forwarded into the container). - SDK pinned to `10.2.0a20261013` (compiler `c7ceccf9a5`). Per-PR CI has since moved to compiler `bdc35841c9` (#13835, 10-19), so the workflow's compiler check now warns; re-pinning to `10.2.0a20261018` or later, which carries `bdc35841c9`, is a follow-up. - **upload-artifact:** v4.6.2 -> v7.0.2 (Node 20 deprecated). v7 needs Actions Runner >= 2.327.1; the gfx1250 runner is 2.337.0. - **TensileLite per-test timeout:** 600 -> 1200 s, this workflow only (FFM and TheRock keep their own). `decouple_pgr_tdm_fuse_gfx1250.yaml` has timed out at 600 s in most runs since #12825 grew its sweep 4.5x; before, it passed 41 of 41. - **Header:** the workflow is marked temporary, to be removed once TheRock's multi-arch builds produce gfx1250-strict artifacts (ROCm/TheRock#8350) and gfx1250-strict runners are available in TheRock CI. ## Test Plan - OpenMP cap: 4 jobs x 4 shards of a `*pre_checkin*` slice on a 128-core gfx942 host, default vs capped. - One-GPU check: 7 values under `bash -euo pipefail`. ## Test Result - [x] Run https://github.com/ROCm/rocm-libraries/actions/runs/38994215757: all steps green with no warnings on SDK 10.2.0a20261013 - hipBLASLt 68,925 tests, 0 failed, in 3.9 min (OpenMP/OpenBLAS capped, BLIS at most 8) while an uncapped run on the same host took 52 min - TensileLite 113 passed, including decouple_pgr_tdm_fuse_gfx1250.yaml under the 1200 s timeout. ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/TheRock/blob/main/GOVERNANCE.md#pull-requests. ## Risk level Low: CI-only, gfx1250 workflow. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude <noreply@anthropic.com>

assistant-librarian · 2026-10-22T19:21:15Z

TheRock Submodule Bump Activity

Newest first

2026-10-22 — Included in TheRock via ROCm/TheRock#9426.

github-actions Bot added github actions project: none Does not target any component labels Oct 14, 2026

archana-ramalingam marked this pull request as ready for review October 14, 2026 10:12

archana-ramalingam requested a review from a team as a code owner October 14, 2026 10:12

assistant-librarian Bot added the organization: ROCm label Oct 14, 2026

style: black-format gfx1250_nightly_report.py

1b342a9

archana-ramalingam changed the title ~~ci(gfx1250): email a report after every nightly; review follow-ups~~ ci(gfx1250): email a report after every nightly; review follow-ups Oct 14, 2026

archana-ramalingam marked this pull request as draft October 14, 2026 10:18

archana-ramalingam marked this pull request as ready for review October 14, 2026 10:18

archana-ramalingam and others added 3 commits October 14, 2026 10:55

archana-ramalingam requested a review from geomin12 October 15, 2026 00:19

archana-ramalingam and others added 4 commits October 15, 2026 04:40

Merge branch 'develop' into users/aramalin/gfx1250-phase2

f56e698

small refactor

9c1707d

geomin12 reviewed Oct 15, 2026

View reviewed changes

archana-ramalingam mentioned this pull request Oct 15, 2026

ci(gfx1250): cap OpenMP threads in hipBLASLt tests; increase tensilelite timeout; cleanup #13771

Merged

2 tasks done

archana-ramalingam changed the title ~~ci(gfx1250): email a report after every nightly; review follow-ups~~ ci(gfx1250): email a test report after every nightly Oct 15, 2026

archana-ramalingam and others added 2 commits October 15, 2026 14:08

Merge branch 'develop' into users/aramalin/gfx1250-phase2

34f5d4c

archana-ramalingam requested a review from geomin12 October 15, 2026 22:42

geomin12 approved these changes Oct 16, 2026

View reviewed changes

tony-davis merged commit 84e72fa into develop Oct 20, 2026
305 of 318 checks passed

tony-davis deleted the users/aramalin/gfx1250-phase2 branch October 20, 2026 21:51

amd-shiraz reviewed Oct 20, 2026

View reviewed changes

therock-pr-bot Bot added the Not ready to Review label Oct 21, 2026

Conversation

archana-ramalingam commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Risk level

Uh oh!

therock-pr-bot Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ PR Check — Action Required

Uh oh!

therock-pr-bot Bot commented Oct 14, 2026

Pre-commit check failed

Uh oh!

therock-pr-bot Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

archana-ramalingam Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

archana-ramalingam Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

geomin12 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

assistant-librarian Bot commented Oct 22, 2026

TheRock Submodule Bump Activity

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

archana-ramalingam commented Oct 14, 2026 •

edited

Loading

therock-pr-bot Bot commented Oct 14, 2026 •

edited

Loading

therock-pr-bot Bot commented Oct 14, 2026 •

edited

Loading

archana-ramalingam Oct 15, 2026 •

edited

Loading

archana-ramalingam Oct 15, 2026 •

edited

Loading