Repository navigation
Conversation
The Run setup test environment step was timing out at 15 minutes. Raising both timeout-minutes and the matching STEP_TIMEOUT_MINUTES env var to 25 minutes to give the step enough headroom to complete. Co-Authored-By: Claude <noreply@anthropic.com>
❌ PR Check — Action Required
📖 Need help? See the Policy FAQ for details on every check and how to fix failures. |
|
🚫 Please fix the failed policies before requesting reviews. The following policy checks failed:
The |
|
I think we're approaching the timeout limit for the setup environment if we need to pull a LOT of artifacts, like the math-libs. |
|
✅ Author chose to skip pr bot run hence removing label ( |
| - name: Run setup test environment workflow | ||
| timeout-minutes: 15 | ||
| timeout-minutes: 25 | ||
| env: | ||
| RETRY_THIS_STEP: "true" | ||
| STEP_TIMEOUT_MINUTES: "15" # should match timeout-minutes specified above at this step | ||
| STEP_TIMEOUT_MINUTES: "25" # should match timeout-minutes specified above at this step |
There was a problem hiding this comment.
Please no :(, or at least not for an extended prior outside of ongoing infrastructure incidents. The execution time expectations per https://github.com/ROCm/TheRock/blob/main/docs/development/test_filtering.md are:
- < 5 minutes for 'quick' (sanity check tests)
- < 30 minutes for 'standard' (tests for key functionality)
If we spend this much of our budget on setup that leaves no time for testing.
We should work to optimize the setup steps:
- [ci] Investigate using a cache for artifacts on test machines #3674
- Improve network speed on runners
- Reduce the amount of data needed
- experiment with compression
- trim the list of packages in https://github.com/ROCm/TheRock/blob/main/requirements-test.txt
If runners are unable to download artifacts within budgets they are not healthy enough to be used on CI
There was a problem hiding this comment.
That's fair. I can't tell whether these are infra issues we're seeing or whether we grew the artifacts/setup enough that we're taking too long now to run things on time.
There was a problem hiding this comment.
The https://github.com/ROCm/TheRock/actions/runs/32165005832/job/96055002747?pr=7449 run you linked spent 6+ minutes downloading the 64MB hip-python python package but did not actually use it. I commented about that over at #6730 (comment)
There was a problem hiding this comment.
I'm dropping this for now. And the scope of the optimization required here unfortunately is beyond what I have cycles to work with.
There was a problem hiding this comment.
Thanks, and sorry if I sounded too dismissive. If we just increase the timeouts that can help with test failures, but it puts increasing pressure on runner queues that lead to negative feedback loops (longer job time -> longer queues).
Summary
Run setup test environment workflowstep timeout from 15 to 25 minutes intest_component.ymltimeout-minutesand the matchingSTEP_TIMEOUT_MINUTESenv var (kept in sync per the existing comment)The step was observed timing out at 15 minutes (e.g. https://github.com/ROCm/TheRock/actions/runs/32165005832/job/96055002747?pr=7449).
Test plan
🤖 Generated with Claude Code
@skip-pr-bot