Failure with awsbatch when running processes that are using executor: local

[rmvgavan]

Description of the bug

Although we are running in awsbatch, some of the processes (e.g., MULTIQC_TSV_FAIL_TRIMMED, MULTIQC_TSV_FAIL_MAPPED) have a specification for executor: local. This results in the following error:

Session aborted -- Cause: Local executor requires the use of POSIX compatible file system — offending work directory path: s3://agc-501439698750-us-east-1/project/RNASeq/userid/ray17Xj9Q/context/rnaBigMem/nextflow-execution/runs

The other processes require an s3 path, but these want a local path for the work dir, so we are unable to successfully run the workflow (since we can't specify both). For the command used/output, we are running in AWS batch but I've written the equivalent parameters. Note there are a number of environment variables, included in the log. Most notably NF_WORKDIR=s3://agc-501439698750-us-east-1/project/RNASeq/userid/ray17Xj9Q/context/rnaBigMem/nextflow-execution/runs, an s3 path. Happy to provide any other info you may need.

Command used and terminal output

$ nextflow run nf-core/rnaseq -profile test,docker --outdir s3://bs2aws/nf-outputs/tmp/rnaseq-test

...(full log attached)...
Mar-15 23:43:00.986 [main] DEBUG nextflow.script.ProcessConfig - Config settings `withName:MULTIQC_TSV_FAIL_TRIMMED` matches process NFCORE_RNASEQ:RNASEQ:MULTIQC_TSV_FAIL_TRIMMED
Mar-15 23:43:00.988 [main] DEBUG nextflow.executor.ExecutorFactory - << taskConfig executor: local
Mar-15 23:43:00.988 [main] DEBUG nextflow.executor.ExecutorFactory - >> processorType: 'local'
Mar-15 23:43:00.990 [main] DEBUG nextflow.executor.Executor - [warm up] executor > local
Mar-15 23:43:00.992 [main] DEBUG n.processor.LocalPollingMonitor - Creating local task monitor for executor 'local' > cpus=4; memory=4 GB; capacity=4; pollInterval=100ms; dumpInterval=5m
Mar-15 23:43:01.005 [main] DEBUG nextflow.Session - Session aborted -- Cause: Local executor requires the use of POSIX compatible file system — offending work directory path: s3://agc-501439698750-us-east-1/project/RNASeq/userid/ray17Xj9Q/context/rnaBigMem/nextflow-execution/runs

Relevant files

log-events-viewer-result.csv

System information

• Nextflow version: 23.02.1-edge
• Hardware: AWS Batch
• Executor: awsbatch but some of the processes use local (which causes problem with s3 path)
• Container engine: Docker
• OS: Amazon Linux 2
• nf-core/rnaseq version: v3.10.1-g6e1e448

[pditommaso]

I guess the problem is that the MULTIQC_TSV_FAIL_TRIMMED force the use of local executor, which does not allow S3 as work dir.

[drpatelh]

Fixed in a8f53be

That module has been re-implemented as a custom function, called within a map and then staged as a file for MultiQC.

Not sure this will solve the root issue you are experiencing @rmvgavan . You may also need to specify -work-dir s3://bs2aws/.

[rmvgavan]

Not sure this will solve the root issue you are experiencing @rmvgavan . You may also need to specify -work-dir s3://bs2aws/.

I tried re-running the test command I have above and the original error is resolved. The new error is a little unclear, so I'm hesitant to report on it, but we see a seg fault in NFCORE_RNASEQ:RNASEQ:QUANTIFY_SALMON:SALMON_QUANT:
test-mode-new-error-log.csv

Good news, however, is that your fix did resolve the issue for our "real" job. This succeeded:

["nf-core/rnaseq", "-r", "a8f53be", "--skip_trimming", "--skip_qc", "--input", "s3://bs2aws/biosamples/CD_16H-8pt5_QC/assembly_ds.f285717497744f89aab40ebbf29d942f/CD_16H-8pt5.csv", "--outdir", "s3://bs2aws/nf-outputs/rnaseq-CD_16H-8pt5_dev/", "--fasta", "s3://bs2aws/nf-test/prokka_ds.0c28dbf6c4854bafb1c6a7c4e146a378/scaffolds.fasta", "--gff", "s3://bs2aws/nf-test/prokka_ds.0c28dbf6c4854bafb1c6a7c4e146a378/scaffolds_with_gene_id.gff", "-profile", "docker"]

Anyway, looks like the local executor s3 issue is indeed fixed. Thanks!

[drpatelh]

Great! Thanks for confirming so quickly. I will close this issue but please feel free to create a separate issue for other problems you may encounter.

For real-time help it may be better to join the nf-core Slack Workspace and ask any pipeline related questions in th e #rnaseq channel We also have a channel called #platform-aws in the Nextflow Slack Workspace where you can ask more infrastructure related questions.