Fix Google Batch hang when internal error during scheduling #5567
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: jorgee <[email protected]>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
Signed-off-by: jorgee <[email protected]>
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy
Outdated
Show resolved
Hide resolved
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy
Outdated
Show resolved
Hide resolved
…atchTaskHandlerTest.groovy Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
…atchTaskHandlerTest.groovy Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy
Outdated
Show resolved
Hide resolved
…atchTaskHandlerTest.groovy [ci skip] Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
Signed-off-by: jorgee <[email protected]>
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
pditommaso
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
…atchTaskHandler.groovy Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
…atchTaskHandler.groovy Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
…atchTaskHandler.groovy [ci skip] Co-authored-by: Paolo Di Tommaso <[email protected]> Signed-off-by: Jorge Ejarque <[email protected]>
Signed-off-by: Jorge Ejarque <[email protected]>
pditommaso
approved these changes
Dec 4, 2024
Yes, initially the job might not have any tasks, but we need to check the task state now to support job arrays, so when there are no tasks we just wait for tasks to be created. Checking the job status for failure makes sense |
|
@bentsherman approve if you think it's good |
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/client/BatchClient.groovy
Outdated
Show resolved
Hide resolved
Signed-off-by: Ben Sherman <[email protected]>
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/test/nextflow/cloud/google/batch/GoogleBatchTaskHandlerTest.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/client/BatchClient.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Outdated
Show resolved
Hide resolved
bentsherman
reviewed
Dec 4, 2024
plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
Show resolved
Hide resolved
Signed-off-by: Ben Sherman <[email protected]>
|
It would be nice to test against a real error case, but not sure if it can be easily replicated. Aside from that, the changes look good to me |
bentsherman
approved these changes
Dec 4, 2024
|
Good point, Esha reported the issue she has a test case |
|
Tested internally |
pditommaso
deleted the
5550-google-batch-run-hangs-when-a-job-fail-to-start
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When there is an internal error in Google Batch the job during scheduling. Jobs go from queued to failed. This situation is happening when request a VM type which is not available in the selected region. In this situation, there are no tasks in the batch job and NF is blocked in an infinite loop caused by the following part of code.
nextflow/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy
In this PR, this part is modified including a job status check to avoid to continue checking when job is failed.
I still have pending to decide what to do if job do not have tasks and job status is null. I think we should abort but @bentsherman do you remember why you returned PENDING when there are no tasks in the job? Is the some initial state where GCP is creating the tasks and we should consider PENDING?