[None][infra] Retry SLURM agent online timeouts - #17586
Conversation
|
/bot run |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
WalkthroughThe Jenkins SLURM startup flow now stops on all nonzero queue results, polls for 10 minutes, propagates interruptions during log retrieval, and raises typed transient failures after startup timeouts. ChangesSLURM agent startup flow
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to This localized change shortens SLURM agent connection timeouts and routes transient failures through the existing retry behavior; no actionable merge-blocking risk remains after normal checks and review. Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant JenkinsPipeline
participant SLURMQueue
participant SLURMAgent
JenkinsPipeline->>SLURMQueue: wait for queue result
SLURMQueue-->>JenkinsPipeline: success or nonzero status
JenkinsPipeline->>SLURMAgent: poll agent availability for up to 10 minutes
SLURMAgent-->>JenkinsPipeline: online or unavailable
JenkinsPipeline->>SLURMAgent: retrieve setup-log tail
SLURMAgent-->>JenkinsPipeline: setup-log tail or retrieval failure
JenkinsPipeline-->>JenkinsPipeline: raise transient SLURM InfraFailure
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@jenkins/L0_Test.groovy`:
- Around line 1307-1321: Update echoRemoteLogTail to catch InterruptedException
before its broad Exception handler and rethrow it unchanged, allowing
FlowInterruptedException to propagate. Leave the existing timeout-block
interruption handling unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 0b7787ac-72a6-4273-bd5e-463dd098e8b7
📒 Files selected for processing (1)
jenkins/L0_Test.groovy
|
PR_Github #65721 [ run ] triggered by Bot. Commit: |
|
PR_Github #65721 [ run ] completed with state |
|
/bot run |
|
PR_Github #65727 [ run ] triggered by Bot. Commit: |
|
PR_Github #65727 [ run ] completed with state
|
|
/bot run |
|
PR_Github #65873 [ run ] triggered by Bot. Commit: |
|
PR_Github #65873 [ run ] completed with state
|
|
/bot run |
|
PR_Github #65912 [ run ] triggered by Bot. Commit: |
|
PR_Github #65912 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66049 [ run ] triggered by Bot. Commit: |
|
/bot run |
|
PR_Github #66060 [ run ] triggered by Bot. Commit: |
|
PR_Github #66049 [ run ] completed with state |
|
/bot run |
|
PR_Github #66063 [ run ] triggered by Bot. Commit: |
|
PR_Github/17586-bc6114f #66060 was force-killed by a newer pipeline run. |
|
PR_Github #66063 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66096 [ run ] triggered by Bot. Commit: |
|
PR_Github #66096 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66154 [ run ] triggered by Bot. Commit: |
|
PR_Github #66154 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66179 [ run ] triggered by Bot. Commit: |
|
PR_Github #66179 [ run ] completed with state
|
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
Signed-off-by: Maxim Gluhovskoi <mgluhovskoi@nvidia.com>
bc6114f to
e1a271b
Compare
|
/bot run |
|
PR_Github #66281 [ run ] triggered by Bot. Commit: |
|
PR_Github #66281 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66310 [ run ] triggered by Bot. Commit: |
|
PR_Github #66310 [ run ] completed with state
|
|
/bot run |
|
PR_Github #66344 [ run ] triggered by Bot. Commit: |
|
PR_Github #66344 [ run ] completed with state |
3 participants
Description
Retry SLURM Jenkins-agent connection timeouts as infrastructure failures.
After a SLURM allocation reaches
RUNNING, the pipeline currently waits up to one hour for its Jenkins agent and then raises an untyped pipeline error. That error bypasses the existing SLURM infrastructure-retry path.This change:
InfraFailurewith a dedicated typed marker.The existing retry loop then makes at most one fresh SLURM attempt, subject to the existing per-stage retry cap and remaining-CI-budget check. The separate SLURM queue wait is unchanged.
Rationale
In the sampled 500-build dataset, 94.9% of successful agent connections completed within five minutes. Waiting an additional 50 minutes before retrying provides little recovery value and can consume most of the pipeline budget.
Test Coverage
jenkins/L0_Test.groovy.20 × 30 seconds = 600 seconds.TRANSIENT, scoped toSLURM, and reaches the existing typed-exception classification and one-retry path.InfraFailureinstances through unchanged.git diff --check.Dev Engineer Review
InfraFailurefor retry classification.QA Engineer Review
No test changes.