Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add process programmatic configuration #4375

Closed
wants to merge 5 commits into from

Conversation

bentsherman
Copy link
Member

Close #4322

Example test case:

// main.nf

include { FOO ; FOO as BAR } from './module.nf'

params.outdir = 'results'
params.publish_dir_mode = 'copy'

workflow {

    FOO.config.ext.extension = 'foo'
    FOO( Channel.of('1', '2', '3') )
    FOO.out.view { "FOO: $it" }

    BAR.config.ext.extension = 'bar'
    BAR.config.publishDir params.outdir, mode: params.publish_dir_mode
    BAR( Channel.of('1', '2', '3') )
    BAR.out.view { "BAR: $it" }

}
// module.nf

process FOO {
    input:
    val(id)

    output:
    tuple val(id), path("sample${id}.${task.ext.extension}")

    script:
    """
    echo '${id} was here' > sample${id}.${task.ext.extension}
    """
}
$ ./launch.sh run main.nf
N E X T F L O W  ~  version 23.09.2-edge
Launching `main.nf` [astonishing_goldwasser] DSL2 - revision: 9b1341eb15
executor >  local (6)
[cd/f6fe43] process > FOO (3) [100%] 3 of 3 ✔
[16/d722fe] process > BAR (1) [100%] 3 of 3 ✔
BAR: [2, work/ad/d60943813d18751a8ded972ee9d798/sample2.bar]
BAR: [3, work/89/9295138db148949f054a22708895a4/sample3.bar]
FOO: [2, work/48/659fef3cb5e492e2f181c975ea232c/sample2.foo]
FOO: [1, work/db/43db80ff13d8d3c7301467e14184db/sample1.foo]
BAR: [1, work/16/d722fe4b621da24e9bd1e7f30cfd1e/sample1.bar]
FOO: [3, work/cd/f6fe432184777c0f08c9e5c1edee30/sample3.foo]

$ ls -al results/
total 1224
drwxr-xr-x  3 bent bent    4096 Oct  3 12:44 ./
drwxr-xr-x 88 bent bent   12288 Oct  3 12:38 ../
-rw-r--r--  1 bent bent      11 Oct  3 12:38 sample1.bar
-rw-r--r--  1 bent bent      11 Oct  3 12:38 sample2.bar
-rw-r--r--  1 bent bent      11 Oct  3 12:38 sample3.bar

$ cat results/*.bar
1 was here
2 was here
3 was here

@netlify
Copy link

netlify bot commented Oct 4, 2023

Deploy Preview for nextflow-docs-staging ready!

Name Link
🔨 Latest commit 6f103be
🔍 Latest deploy log https://app.netlify.com/sites/nextflow-docs-staging/deploys/65201ca23e63fb00081d3f2f
😎 Deploy Preview https://deploy-preview-4375--nextflow-docs-staging.netlify.app/config
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@@ -183,6 +183,26 @@ workflow {
Optional params for a process input/output are always prefixed with a comma, except for `stdout`. Because `stdout` does not have an associated name or value like other types, the first param should not be prefixed.
:::

### Process config
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally would like to merge #3793 first, then this would become part of the "Workflows" page which is more prominent.

@pditommaso
Copy link
Member

Well done! Look @drpatelh! ☝️

Signed-off-by: Ben Sherman <[email protected]>
Signed-off-by: Ben Sherman <[email protected]>
@bentsherman
Copy link
Member Author

I added the config setting for publishDir defaults as defined in #4186 so that I can further improve the nf-core/rnaseq POC. Doesn't have to be merged with the final PR but it's a simple change.

@drpatelh
Copy link
Contributor

drpatelh commented Oct 8, 2023

This is awesome! Thank you! Will take a closer look when I get a moment. Also, thanks for testing with nf-core/rnaseq!

@bentsherman
Copy link
Member Author

Closing in favor of #4510

@bentsherman bentsherman deleted the 4322-process-programmatic-config branch November 21, 2023 20:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[proposal] Add process programmatic configuration context
3 participants