Improve Google Batch 5000x error class handling

[pditommaso]

This PR improve the handling of Google Batch 5000x error class in such a way that

1. All 50001 .. 50006 error codes managed, see here
2. When this error condition arises is converted into ProcessException including the source error description
3. The usual Nextflow errorRetry strategies can be applied to handle the error condition

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	cb63ec9
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/6690fe184f75e4000875c90e
😎 Deploy Preview	https://deploy-preview-5141--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[bentsherman]

See also:

nextflow/plugins/nf-google/src/main/nextflow/cloud/google/batch/GoogleBatchTaskHandler.groovy

Lines 262 to 277 in cb63ec9

	// retry on spot reclaim
	if( executor.config.maxSpotAttempts ) {
	// Note: Google Batch uses the special exit status 50001 to signal
	// the execution was terminated due a spot reclaim. When this happens
	// The policy re-execute the jobs automatically up to `maxSpotAttempts` times
	taskSpec
	.setMaxRetryCount( executor.config.maxSpotAttempts )
	.addLifecyclePolicies(
	LifecyclePolicy.newBuilder()
	.setActionCondition(
	LifecyclePolicy.ActionCondition.newBuilder()
	.addExitCodes(50001)
	)
	.setAction(LifecyclePolicy.Action.RETRY_TASK)
	)
	}

[VasLem]

Just a user opinion, I would prefer these errors not to change the task attempt number, as in a dynamic configurations this would lead to unnecessary increase of resources, or unnecessary termination/ignoration of the process.

[pditommaso]

This change makes it possible to recover errors from 50002 to 50006, similarly to 50001. Currently, they are causing the termination of the pipeline execution.

@bentsherman Regarding the Batch automatic recovery, I didn't change it on purpose because the semantic settings are specific for spot interruption since it's named maxSpotAttempts.

The this failures can be recovered with the usual error strategy.

[bentsherman]

From meeting discussion, we suggested using a new config setting e.g. google.batch.autoRetryExitCodes to provide a list of exit codes to retry natively (without increasing the task attempt)

[pditommaso]

Merging this. I'll open a separate PR for the retry logic at this comment