Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add process programmatic configuration #4375

Closed
wants to merge 5 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -1071,6 +1071,19 @@ manifest {

Read the {ref}`sharing-page` page to learn how to publish your pipeline to GitHub, BitBucket or GitLab.

(config-nextflow)=

### Scope `nextflow`

The `nextflow` scope allows you to define some miscellaneous settings.

The following settings are available:

`nextflow.defaults.publishDir`
: :::{versionadded} 23.10.0
:::
: Default params for the {ref}`process-publishdir` directive. Every instance of this directive will use these defaults unless it overrides them for itself.

(config-notification)=

### Scope `notification`
Expand Down
22 changes: 22 additions & 0 deletions docs/dsl2.md
Original file line number Diff line number Diff line change
Expand Up @@ -183,6 +183,28 @@ workflow {
Optional params for a process input/output are always prefixed with a comma, except for `stdout`. Because `stdout` does not have an associated name or value like other types, the first param should not be prefixed.
:::

(dsl2-process-config)=

### Process config
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally would like to merge #3793 first, then this would become part of the "Workflows" page which is more prominent.


:::{versionadded} 23.10.0
:::

A process can be configured with additional directives from a workflow by using the `config` attribute on a process. For example:

```groovy
include { foo ; foo as bar } from './module.nf'

workflow {
foo()

bar.config.publishDir 'results', mode: 'copy'
bar()
}
```

Process directives can be defined using the same syntax as in a process definition, and they are applied separately to different module aliases. In the above example, the `publishDir` directive is applied to process `bar` but not to process `foo`.

## Workflow

### Workflow definition
Expand Down
6 changes: 6 additions & 0 deletions docs/process.md
Original file line number Diff line number Diff line change
Expand Up @@ -1229,6 +1229,12 @@ By default, directives are evaluated when the process is defined. However, if th

Some directives are generally available to all processes, while others depend on the `executor` currently defined.

Process directives can also be defined in the Nextflow configuration using the {ref}`process <config-process>` scope.

:::{versionadded} 23.10.0
Process directives can also be defined for a particular process from the enclosing workflow. See {ref}`dsl2-process-config` for details.
:::

(process-accelerator)=

### accelerator
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ import java.util.regex.Pattern
import groovy.transform.PackageScope
import groovy.util.logging.Slf4j
import nextflow.Const
import nextflow.Global
import nextflow.Session
import nextflow.NF
import nextflow.ast.NextflowDSLImpl
import nextflow.exception.ConfigParseException
Expand Down Expand Up @@ -842,7 +844,9 @@ class ProcessConfig implements Map<String,Object>, Cloneable {
configProperties.put('publishDir', dirs)
}

dirs.add(params)
final session = Global.session as Session
final defaults = session.config.navigate('nextflow.defaults.publishDir', [:]) as Map
dirs.add(defaults + params)
return this
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,8 @@ class ProcessDef extends BindableDef implements IterableDef, ChainableDef {
this.simpleName = name
this.processName = name
this.baseName = name

this.processConfig = new ProcessConfig(owner,processName)
}

static String stripScope(String str) {
Expand All @@ -96,11 +98,6 @@ class ProcessDef extends BindableDef implements IterableDef, ChainableDef {

protected void initialize() {
log.trace "Process config > $processName"
assert processConfig==null

// the config object
processConfig = new ProcessConfig(owner,processName)

// Invoke the code block which will return the script closure to the executed.
// As side effect will set all the property declarations in the 'taskConfig' object.
processConfig.throwExceptionOnMissingProperty(true)
Expand All @@ -121,6 +118,7 @@ class ProcessDef extends BindableDef implements IterableDef, ChainableDef {
def result = (ProcessDef)super.clone()
result.@taskBody = taskBody?.clone()
result.@rawBody = (Closure)rawBody?.clone()
result.@processConfig = processConfig?.clone()
return result
}

Expand All @@ -145,7 +143,7 @@ class ProcessDef extends BindableDef implements IterableDef, ChainableDef {

String getBaseName() { baseName }

ProcessConfig getProcessConfig() { processConfig }
ProcessConfig getConfig() { processConfig }

ChannelOut getOut() {
if( output==null )
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -98,15 +98,15 @@ class ProcessDefTest extends Specification {
def copy = proc.clone()
copy.initialize()
then:
def cfg1 = copy.processConfig.createTaskConfig()
def cfg1 = copy.config.createTaskConfig()
cfg1.getCpus()==2 // taken from the generic config
cfg1.getMemory().giga == 3 // taken from the `foo` config

when:
copy = proc.cloneWithName('flow1:bar')
copy.initialize()
then:
def cfg2 = copy.processConfig.createTaskConfig()
def cfg2 = copy.config.createTaskConfig()
cfg2.getCpus()==4 // taken from the `bar` config
cfg2.getMemory().giga == 4 // taken from the `bar` config

Expand All @@ -115,7 +115,7 @@ class ProcessDefTest extends Specification {
copy = proc.cloneWithName('flow1:flow2:flow3:bar')
copy.initialize()
then:
def cfg3 = copy.processConfig.createTaskConfig()
def cfg3 = copy.config.createTaskConfig()
cfg3.getCpus()==4 // <-- taken from `withName: foo`
cfg3.getMemory().giga == 8 // <-- taken from `withName: 'flow1:flow2:flow3:bar'`
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -137,8 +137,8 @@ class ParamsDsl2Test extends Dsl2Spec {
def process = ScriptMeta.get(script).getProcess('alpha'); process.initialize()

then:
def inputs = process.processConfig.getInputs()
def outputs = process.processConfig.getOutputs()
def inputs = process.config.getInputs()
def outputs = process.config.getOutputs()
and:
inputs.size() == 1
inputs[0] instanceof StdInParam
Expand Down Expand Up @@ -178,8 +178,8 @@ class ParamsDsl2Test extends Dsl2Spec {
def process = ScriptMeta.get(script).getProcess('beta'); process.initialize()

then:
def inputs = process.processConfig.getInputs()
def outputs = process.processConfig.getOutputs()
def inputs = process.config.getInputs()
def outputs = process.config.getOutputs()
and:
inputs.size() == 1
and:
Expand Down
Loading