Skip to content

ci: treat docker container-removal conflict as flaky #5120

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ko3n1g merged 1 commit into from
Jun 2, 2026
Merged

ci: treat docker container-removal conflict as flaky #5120

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions tests/test_utils/python_scripts/launch_nemo_run_workload.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,9 @@ def is_flaky_failure(concat_allranks_logs: str) -> bool:
or "free(): corrupted unsorted chunks" in concat_allranks_logs
or "Segfault encountered" in concat_allranks_logs
or "The following metrics failed" in concat_allranks_logs
or "removal of container" in concat_allranks_logs
or "is already in progress" in concat_allranks_logs

@greptile-apps greptile-apps Bot Jun 2, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Overly broad flaky-detection substring

"is already in progress" is a generic phrase that could appear in unrelated training or NCCL log lines (e.g. an NCCL collective saying an operation is already in progress, or any distributed-sync message). If a genuinely broken job emits this phrase, it will be silently retried up to three times before actually failing, delaying failure detection and consuming CI resources.

"removal of container" already captures the same 409-Conflict Docker error message more precisely; "is already in progress" adds little discriminative value on top of it. Removing this entry, or replacing it with a tighter composite like "removal of container" in ... and "is already in progress" in ..., eliminates the false-positive risk without reducing coverage of the target error.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

or "Error deleting container" in concat_allranks_logs
)


Expand Down
Loading