Fix race condition when resuming an aborted run

[bentsherman]

Close #5595

When a Nextflow run is aborted for any reason, Nextflow will try to kill all jobs and save them to the cache as failed tasks, so that on a resumed run, the new jobs will use new task directories.

However, if Nextflow is unable to complete this cleanup for any reason, such as an abrupt crash, the task cache might not be updated. On a resumed run, the new task will be re-executed in the same directory as before:

nextflow/modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy

Lines 806 to 826 in b23e42c

	Path resumeDir = null
	boolean exists = false
	try {
	final entry = session.cache.getTaskEntry(hash, this)
	resumeDir = entry ? FileHelper.asPath(entry.trace.getWorkDir()) : null
	if( resumeDir )
	exists = resumeDir.exists()
	log.trace "[${safeTaskName(task)}] Cacheable folder=${resumeDir?.toUriString()} -- exists=$exists; try=$tries; shouldTryCache=$shouldTryCache; entry=$entry"
	final cached = shouldTryCache && exists && entry.trace.isCompleted() && checkCachedOutput(task.clone(), resumeDir, hash, entry)
	if( cached )
	break
	}
	catch (Throwable t) {
	log.warn1("[${safeTaskName(task)}] Unable to resume cached task -- See log file for details", causedBy: t)
	}
	if( exists ) {
	tries++
	continue
	}

What I think happens here is:

1. the cache entry is missing
2. Nextflow doesn't check whether the task dir exists
3. Nextflow proceeds to re-use the task dir with a new task

This can cause a race condition because if the begin and exit files are still present, Nextflow could submit a job, detect the begin/exit codes from the previous job, mark the job as completed (as long as the required outputs are still present), and launch downstream tasks on truncated data.

I think this can only happen on grid executors because the GridTaskHandler checks the begin/exit files before polling the scheduler. The cloud executors poll their API first, and the local executor doesn't check these files at all.

It should be possible to replicate this issue on a grid executor by cancelling a run with -disable-jobs-cancellation and allowing the jobs to complete before resuming the run. Which by the way is an unsafe property of that CLI option.

I'm trying to modify the local executor to simulate the issue but haven't been able to replicate yet

[netlify]

✅ Deploy Preview for nextflow-docs-staging ready!

Name	Link
🔨 Latest commit	544be43
🔍 Latest deploy log	https://app.netlify.com/sites/nextflow-docs-staging/deploys/675a0eb51e62630007303948
😎 Deploy Preview	https://deploy-preview-5600--nextflow-docs-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.