Dev -> Master for 2.2 release

[drpatelh]

No description provided.

[JoseEspinosa]

LGTM! Just made some small suggestions!

[JoseEspinosa]

This is more a comment than a change suggestion (I also remembered this was discussed somewhere else but can't find where). After starting the conversion to DSL2 v2.0 of the chipseq pipeline, I was wondering whether maintaining a default value would be good in case someone wants to completely change the behavior of all the processes, otherwise, it will need to modify all the process modes 🤔

[drpatelh]

Yeah, you might be right. But I suspect you may be able to use the power of NF to provide a generic override for that too? 🤔 Not sure exactly what the syntax would be but something like this:

process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
        mode: 'symlink',
        saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
    ]
}

[JoseEspinosa]

The problem is that this won't override all the named process selectors where the mode is harcoded, so then you will have to copy all the rest of the ext.args from the original modules.config, I'd say

[drpatelh]

It should do. The settings above if provided via -c custom.config should override any mode set in the pipeline for all processes. Maybe @mahesh-panchal can confirm.

[JoseEspinosa]

OK, then probably that's the part I missed, fair enough 👍
Will keep testing and I find something not working as expected then we can discuss it further.

[mahesh-panchal]

Confirmed:
main.nf:

nextflow.enable.dsl = 2

workflow {
    FOO()
}

process FOO {

    output:
    path "table.txt", emit: txt

    script:
    """
    touch table.txt
    """
}

nextflow.config:

params.outdir = 'results'
process {
    withName: FOO {
        publishDir = [
            path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
            mode: 'copy',
            saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
        ]
    }
}

custom.config:

params.outdir = 'results'
process {
    publishDir = [
        path: { "${params.outdir}/${task.process.tokenize(':')[-1].tokenize('_')[0].toLowerCase()}" },
        mode: 'symlink',
        saveAs: { filename -> filename.equals('versions.yml') ? null : filename }
    ]
    withName: FOO {
       publishDir = "foo"
    }
}

Output:

$ nextflow run main.nf -c custom.config 
N E X T F L O W  ~  version 21.10.0
Launching `main.nf` [focused_swirles] - revision: de5020ba1d
executor >  local (1)
[25/cdff54] process > FOO [100%] 1 of 1 ✔

$ tree -a
.
├── .nextflow
│   ├── cache
│   │   ├── aeca00af-ea5f-47fc-afe6-de3c6f46a563
│   │   │   ├── db
│   │   │   │   ├── 000003.log
│   │   │   │   ├── CURRENT
│   │   │   │   ├── LOCK
│   │   │   │   └── MANIFEST-000002
│   │   │   └── index.sleepy_mayer
│   │   └── cb644109-0e7f-4657-91fc-3ef9d15345a1
│   │       ├── db
│   │       │   ├── 000003.log
│   │       │   ├── CURRENT
│   │       │   ├── LOCK
│   │       │   └── MANIFEST-000002
│   │       └── index.focused_swirles
│   ├── history
│   └── plr
├── .nextflow.log
├── .nextflow.log.1
├── custom.config
├── foo
│   └── table.txt -> /Users/mahpa906/Downloads/Nextflow_sandbox/test_publish_mode/work/25/cdff5407da55c04b0929d82216982e/table.txt
├── main.nf
├── nextflow.config
├── results
│   └── foo
│       └── table.txt
└── work
    ├── 25
    │   └── cdff5407da55c04b0929d82216982e
    │       ├── .command.begin
    │       ├── .command.err
    │       ├── .command.log
    │       ├── .command.out
    │       ├── .command.run
    │       ├── .command.sh
    │       ├── .exitcode
    │       └── table.txt
    └── bf
        └── fef5f5ebf87254c920c447d82648ff
            ├── .command.begin
            ├── .command.err
            ├── .command.log
            ├── .command.out
            ├── .command.run
            ├── .command.sh
            ├── .exitcode
            └── table.txt

[mahesh-panchal]

I just realised my test didn't confirm mode inheritance. It's using the default setting each time.
If you set it process wide to 'copy', but then override it for a specific instance, then the default mode 'symlink' is still used.

[ggabernet]

Suggested change

	.github/ISSUE_TEMPLATE/bug_report.md
	.github/ISSUE_TEMPLATE/bug_report.md
	.github/ISSUE_TEMPLATE/feature_request.md

[ggabernet]

Suggested change

	"outdir": "s3://{% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %}/{{ short_name }}/{% raw %}results-${{ github.sha }}{% endraw %}"
	"outdir": "s3://{% raw %}${{ secrets.AWS_S3_BUCKET }}{% endraw %}/{{ short_name }}/{% raw %}results-small-tests-${{ github.sha }}{% endraw %}"

[ggabernet]

Just thought about this now, that avoids conflicts with big tests commit sha!

[drpatelh]

Ah, crap! So they would be put in the same directory if we don't change that?