Skip to content

ci: raise setup test environment timeout from 15 to 25 minutes - #7483

Closed
lumachad wants to merge 1 commit into
mainfrom
users/lumachad/raise_setup_environment_timeout
Closed

lumachad wants to merge 1 commit into
mainfrom
users/lumachad/raise_setup_environment_timeout

Conversation

@lumachad

@lumachad lumachad commented Aug 19, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Raises the Run setup test environment workflow step timeout from 15 to 25 minutes in test_component.yml
  • Updates both timeout-minutes and the matching STEP_TIMEOUT_MINUTES env var (kept in sync per the existing comment)

The step was observed timing out at 15 minutes (e.g. https://github.com/ROCm/TheRock/actions/runs/32165005832/job/96055002747?pr=7449).

Test plan

  • Confirm the setup test environment step completes within 25 minutes in CI

🤖 Generated with Claude Code

@skip-pr-bot

The Run setup test environment step was timing out at 15 minutes.
Raising both timeout-minutes and the matching STEP_TIMEOUT_MINUTES
env var to 25 minutes to give the step enough headroom to complete.

Co-Authored-By: Claude <noreply@anthropic.com>
@lumachad lumachad self-assigned this Aug 19, 2026
@lumachad
lumachad requested a review from geomin12 August 19, 2026 11:59
@lumachad
lumachad marked this pull request as ready for review August 19, 2026 11:59
@lumachad
lumachad requested a review from ndhillon07 August 19, 2026 11:59
@therock-pr-bot

therock-pr-bot Bot commented Aug 19, 2026 •

Copy link
Copy Markdown

❌ PR Check — Action Required

Check Status Details
📝 PR Description ❌ Fail Error: PR description must reference a JIRA ID, ISSUE ID, or a GitHub closing keyword.
Expected: include a JIRA ID / ISSUE ID line (separator : or -, or omitted; value may be a JIRA key, a number with/without #, or a link), OR a closing keyword + issue reference. Accepted examples:
• JIRA ID : TESTAUTO-6039
• JIRA ID - #330
• JIRA ID #330
• JIRA ID (on separate line)
ROCM-25757
• ISSUE ID : TESTUTO-3334
• ISSUE ID #3334
• ISSUE ID - TESTAUTO-3433
• ISSUE ID (on separate line)
AIRUNTIME-2352
• ISSUE ID : https://github.com/<org_name>/<repo_name>/issues/1234
• Closes #10
• Fixes octo-org/octo-repo#100
• Resolves: #123
• #123
• https://github.com/<org_name>/<repo_name>/issues/123
Current: no valid JIRA/ISSUE/closing-keyword reference found
⛔ Forbidden Files ✅ Pass —
🧪 Unit Test ✅ Pass PR does not contain code files — Unit Test auto-passed
🔎 pre-commit ✅ Pass —
🚫 Draft PR 🔜 To Be Enabled —
🚩 Feature Flag 🔜 To Be Enabled —
📊 Code Coverage 🔜 To Be Enabled —

⚠️ 1 policy check(s) failed. Please address the issues above before this PR can be Reviewed.

🚫 Please fix the failed policies

  • ❌ PR Description

The Not ready to Review label was added to this PR. Once all policies pass, the label is removed automatically.

📖 Need help? See the Policy FAQ for details on every check and how to fix failures.

🙋 Wish to Override Policy?

@therock-pr-bot therock-pr-bot Bot added the Not ready to Review PR has unresolved policy failures — reviews blocked label Aug 19, 2026
@therock-pr-bot

Copy link
Copy Markdown

🚫 Please fix the failed policies before requesting reviews.

The following policy checks failed:

  • ❌ PR Description

The Not ready to Review label has been added to this PR.
Once all policies pass, the label will be removed automatically.

@lumachad

Copy link
Copy Markdown
Contributor Author

I think we're approaching the timeout limit for the setup environment if we need to pull a LOT of artifacts, like the math-libs.

@therock-pr-bot

Copy link
Copy Markdown

✅ Author chose to skip pr bot run hence removing label (@skip-pr-bot found in the PR description).

@therock-pr-bot therock-pr-bot Bot removed the Not ready to Review PR has unresolved policy failures — reviews blocked label Aug 19, 2026
Comment on lines 132 to +136
- name: Run setup test environment workflow
timeout-minutes: 15
timeout-minutes: 25
env:
RETRY_THIS_STEP: "true"
STEP_TIMEOUT_MINUTES: "15" # should match timeout-minutes specified above at this step
STEP_TIMEOUT_MINUTES: "25" # should match timeout-minutes specified above at this step

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please no :(, or at least not for an extended prior outside of ongoing infrastructure incidents. The execution time expectations per https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md are:

  • < 5 minutes for 'quick' (sanity check tests)
  • < 30 minutes for 'standard' (tests for key functionality)

If we spend this much of our budget on setup that leaves no time for testing.

We should work to optimize the setup steps:

If runners are unable to download artifacts within budgets they are not healthy enough to be used on CI

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's fair. I can't tell whether these are infra issues we're seeing or whether we grew the artifacts/setup enough that we're taking too long now to run things on time.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The https://github.com/ROCm/TheRock/actions/runs/32165005832/job/96055002747?pr=7449 run you linked spent 6+ minutes downloading the 64MB hip-python python package but did not actually use it. I commented about that over at #6730 (comment)

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm dropping this for now. And the scope of the optimization required here unfortunately is beyond what I have cycles to work with.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, and sorry if I sounded too dismissive. If we just increase the timeouts that can help with test failures, but it puts increasing pressure on runner queues that lead to negative feedback loops (longer job time -> longer queues).

@lumachad
lumachad marked this pull request as draft August 19, 2026 15:51
@lumachad lumachad added the ci:skip Skip all CI builds/tests for this PR label Aug 19, 2026
@lumachad lumachad closed this Aug 19, 2026
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Aug 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:skip Skip all CI builds/tests for this PR

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants