Skip to content

ci(hipblaslt): [TheRock] race detection using rocjitsu - #8929

Closed
newling wants to merge 7 commits into
ROCm:developfrom
newling:tensilelite-rj-race-ci-2
Closed

newling wants to merge 7 commits into
ROCm:developfrom
newling:tensilelite-rj-race-ci-2

Conversation

@newling

@newling newling commented Jun 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Superseded by #9363

Add a first rocjitsu race-detection smoke lane for hipBLASLt changes in TheRock Linux CI.

This adds a rocjitsu-race-smoke-linux job to therock-ci-linux.yml. The job runs only for the existing gfx950 Linux test target and only when the selected test projects include hipblaslt.

The smoke test:

  • fetches the same-run TheRock BLAS test artifacts with --blas --tests
  • checks out ROCm/rocm-systems sparsely for emulation/rocjitsu
  • builds rocjitsu locally as a temporary bridge
  • runs a tiny hipblaslt-bench gfx950 GEMM under RJ_RACE=1
  • fails if rocjitsu writes any RACE records

Background

This follows up on the earlier proof-of-concept in #7951, which explored running race detection for hipBLASLt/TensileLite changes in rocm-libraries.

The main differences from that PoC are:

  • This is attached to the existing TheRock Linux CI flow instead of being a mostly standalone workflow.
  • It consumes the hipBLASLt test binary from the same-run TheRock BLAS test artifacts, so it exercises the PR-built hipBLASLt artifacts.
  • It is restricted to the existing gfx950 test runner path instead of running on free GitHub-hosted runners.
  • The test payload is intentionally smaller: one explicit hipblaslt-bench smoke command to validate the CI wiring and race-report behavior before broadening coverage.
  • rocjitsu is still built locally in this job, but only as a temporary bridge until rocjitsu is available as a complete runnable TheRock artifact.

Follow-up

Longer term, this should move toward a fully TheRock-native component that fetches rocjitsu via TheRock artifacts, for example with --blas --tests --rocjitsu, rather than building rocjitsu inside this job.

Testing

Local lightweight checks only:

  • bash -n .github/scripts/run_rocjitsu_hipblaslt_race_smoke.sh
  • git diff --check -- .github/scripts/run_rocjitsu_hipblaslt_race_smoke.sh .github/workflows/therock-ci-linux.yml
  • Parsed .github/workflows/therock-ci-linux.yml with PyYAML

JIRA ID : AIHPBLAS-3605
https://amd-hub.atlassian.net/browse/AIHPBLAS-3605

@github-actions github-actions Bot added github actions project: none Does not target any component labels Jun 29, 2026
@therock-pr-bot

therock-pr-bot Bot commented Jun 29, 2026 •

Copy link
Copy Markdown

✅ All Checks Passed — Ready for Review

Check Status Details
🌿 Branch Name ✅ Pass —
📝 PR Title/Description ✅ Pass —
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —
🤖 therock-pr-bot ✅ Pass —

🎉 All checks passed! This PR is ready for review.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

@therock-pr-bot

Copy link
Copy Markdown

🎉 All checks passed! This PR is ready for review.

@newling
newling force-pushed the tensilelite-rj-race-ci-2 branch from 594d914 to 1f5ad38 Compare June 29, 2026 22:19
@newling
newling force-pushed the tensilelite-rj-race-ci-2 branch 4 times, most recently from a94625c to d36cfec Compare June 30, 2026 21:10
@talumbau
talumbau self-requested a review July 6, 2026 19:25
@newling
newling force-pushed the tensilelite-rj-race-ci-2 branch 9 times, most recently from dcf56cb to 9152373 Compare July 10, 2026 17:05
@newling newling changed the title ci(hipblaslt): add rocjitsu race smoke for gfx950 ci(hipblaslt): [TheRock] race detection using rocjitsu Jul 10, 2026
@newling
newling force-pushed the tensilelite-rj-race-ci-2 branch from 9152373 to 0e9ebbb Compare July 10, 2026 18:06
@newling

newling commented Jul 16, 2026

Copy link
Copy Markdown
Contributor Author

Superseded by #9363

newling added a commit that referenced this pull request Jul 23, 2026
## Motivation

Adds minimal rocjitsu race detection for GEMMs on gfx1151, gfx950, and
gfx942/gfx94X, run through `hipblaslt-bench` and `tensilelite-client`.

This is the third step in the evolution of race detection for hipBLASLt:

1. #7951 did not use TheRock.
It used a free GitHub runner, checked out rocm-libraries and
rocm-systems, built them, and ran rocjitsu with hipBLASLt.
2. #8929 used TheRock, but
only ran `hipblaslt-bench` on gfx950.
3. This PR uses TheRock, adds gfx1151 and gfx942/gfx94X coverage, adds a
reduced `tensilelite-client` path, and runs rocjitsu on a CPU build
runner instead of consuming a GPU test runner.

JIRA ID : AIHPBLAS-3605
https://amd-hub.atlassian.net/browse/AIHPBLAS-3605

## High Level Overview

For hipBLASLt TheRock Linux rows on supported GPU families, this adds a
post-build rocjitsu race-check job.

The flow is:

1. The normal TheRock build produces the ROCm runtime, hipBLASLt, and
TensileLite test artifacts.
2. The race-check job fetches those artifacts from the same TheRock run.
3. The job builds rocjitsu from a pinned rocm-systems revision.
4. The job runs small `hipblaslt-bench` and `tensilelite-client`
workloads under rocjitsu race detection.
5. The script reports a failure if a workload fails or rocjitsu reports
a race.

The check is initially advisory. Failures remain visible in the job logs
and uploaded reports, but do not fail the required TheRock CI summary
while the job's reliability is evaluated.

The goal is to test GPU-targeted hipBLASLt and TensileLite artifacts
under CPU-side emulation without requiring GPU runner capacity.

### Workspace Layout

The job uses three repository checkouts in one workspace:

- `ROCm/TheRock` is checked out at the workspace root. Its actions and
scripts fetch the artifacts produced by the current TheRock run.
- The current `ROCm/rocm-libraries` revision is checked out under
`rocm-libraries/`. A sparse checkout provides the race-check driver
script from `.github/scripts`.
- `ROCm/rocm-systems` is checked out under `rocm-systems/` at a pinned
revision. A sparse checkout provides the rocjitsu sources and
machine-readable ISA data needed to build rocjitsu.

These checkouts provide the workflow logic and test tooling. The ROCm
runtime, compiler, hipBLASLt, TensileLite, and test payload being
exercised still come from the artifacts produced by the associated
TheRock build.

## End-to-end CI Proof

A recent CI run has successful rocjitsu race-check jobs for all three
target rows:

- gfx1151:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89005639878
- gfx950-dcgpu:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89010903066
- gfx94X-dcgpu:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89015610352

## Current Limitations

- rocjitsu is still built from a pinned rocm-systems source checkout.
This should move to a packaged TheRock component artifact.
- The check is initially advisory while its reliability is evaluated.
- The TensileLite YAMLs are embedded in the script. They should move to
checked-in test data or be replaced by existing product-owned workloads.

## Next Steps And Ongoing Work

- Evaluate the job's reliability and decide when it should become
blocking.
- Move rocjitsu consumption to a complete TheRock runnable artifact.
- Move the inline TensileLite YAMLs into checked-in test data or reuse
existing workloads.
- Continue the TheRock-side work toward a cleaner component path:
  - ROCm/TheRock#6517 makes rocjitsu available as a runnable artifact.
  - ROCm/TheRock#6525 adds a component-level rocjitsu smoke test.
- ROCm/TheRock#6575 separates artifact selection from runner placement
so GPU-targeted artifacts can be tested on CPU runners.
bghimireamd pushed a commit that referenced this pull request Jul 28, 2026
## Motivation

Adds minimal rocjitsu race detection for GEMMs on gfx1151, gfx950, and
gfx942/gfx94X, run through `hipblaslt-bench` and `tensilelite-client`.

This is the third step in the evolution of race detection for hipBLASLt:

1. #7951 did not use TheRock.
It used a free GitHub runner, checked out rocm-libraries and
rocm-systems, built them, and ran rocjitsu with hipBLASLt.
2. #8929 used TheRock, but
only ran `hipblaslt-bench` on gfx950.
3. This PR uses TheRock, adds gfx1151 and gfx942/gfx94X coverage, adds a
reduced `tensilelite-client` path, and runs rocjitsu on a CPU build
runner instead of consuming a GPU test runner.

JIRA ID : AIHPBLAS-3605
https://amd-hub.atlassian.net/browse/AIHPBLAS-3605

## High Level Overview

For hipBLASLt TheRock Linux rows on supported GPU families, this adds a
post-build rocjitsu race-check job.

The flow is:

1. The normal TheRock build produces the ROCm runtime, hipBLASLt, and
TensileLite test artifacts.
2. The race-check job fetches those artifacts from the same TheRock run.
3. The job builds rocjitsu from a pinned rocm-systems revision.
4. The job runs small `hipblaslt-bench` and `tensilelite-client`
workloads under rocjitsu race detection.
5. The script reports a failure if a workload fails or rocjitsu reports
a race.

The check is initially advisory. Failures remain visible in the job logs
and uploaded reports, but do not fail the required TheRock CI summary
while the job's reliability is evaluated.

The goal is to test GPU-targeted hipBLASLt and TensileLite artifacts
under CPU-side emulation without requiring GPU runner capacity.

### Workspace Layout

The job uses three repository checkouts in one workspace:

- `ROCm/TheRock` is checked out at the workspace root. Its actions and
scripts fetch the artifacts produced by the current TheRock run.
- The current `ROCm/rocm-libraries` revision is checked out under
`rocm-libraries/`. A sparse checkout provides the race-check driver
script from `.github/scripts`.
- `ROCm/rocm-systems` is checked out under `rocm-systems/` at a pinned
revision. A sparse checkout provides the rocjitsu sources and
machine-readable ISA data needed to build rocjitsu.

These checkouts provide the workflow logic and test tooling. The ROCm
runtime, compiler, hipBLASLt, TensileLite, and test payload being
exercised still come from the artifacts produced by the associated
TheRock build.

## End-to-end CI Proof

A recent CI run has successful rocjitsu race-check jobs for all three
target rows:

- gfx1151:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89005639878
- gfx950-dcgpu:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89010903066
- gfx94X-dcgpu:
https://github.com/ROCm/rocm-libraries/actions/runs/29942116838/job/89015610352

## Current Limitations

- rocjitsu is still built from a pinned rocm-systems source checkout.
This should move to a packaged TheRock component artifact.
- The check is initially advisory while its reliability is evaluated.
- The TensileLite YAMLs are embedded in the script. They should move to
checked-in test data or be replaced by existing product-owned workloads.

## Next Steps And Ongoing Work

- Evaluate the job's reliability and decide when it should become
blocking.
- Move rocjitsu consumption to a complete TheRock runnable artifact.
- Move the inline TensileLite YAMLs into checked-in test data or reuse
existing workloads.
- Continue the TheRock-side work toward a cleaner component path:
  - ROCm/TheRock#6517 makes rocjitsu available as a runnable artifact.
  - ROCm/TheRock#6525 adds a component-level rocjitsu smoke test.
- ROCm/TheRock#6575 separates artifact selection from runner placement
so GPU-targeted artifacts can be tested on CPU runners.
@newling newling closed this Aug 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant