From e025a846ecc471c42277ac17e800a4d59c15611b Mon Sep 17 00:00:00 2001 From: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Date: Thu, 21 Nov 2024 16:25:31 +0000 Subject: [PATCH 01/13] Update machineType docs to reflect supported executors (#5533) [ci skip] --- docs/reference/process.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/process.md b/docs/reference/process.md index 7faa8097e7..95a9456ef8 100644 --- a/docs/reference/process.md +++ b/docs/reference/process.md @@ -817,7 +817,7 @@ See also: [resourceLabels](#resourcelabels) :::{versionadded} 19.07.0 ::: -The `machineType` can be used to specify a predefined Google Compute Platform [machine type](https://cloud.google.com/compute/docs/machine-types) when running using the {ref}`Google Life Sciences ` executor. +The `machineType` can be used to specify a predefined Google Compute Platform [machine type](https://cloud.google.com/compute/docs/machine-types) when running using the {ref}`Google Batch ` or {ref}`Google Life Sciences ` executor, or when using the autopools feature of the {ref}`Azure Batch executor`. This directive is optional and if specified overrides the cpus and memory directives: From df69b8f612e4bd85352841838dbcc31adb5d5aa5 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Fri, 22 Nov 2024 17:23:13 +0100 Subject: [PATCH 02/13] Docs: Move `-params-file` config from CLI to config (#5534) [ci skip] Signed-off-by: Christopher Hakkaart --- docs/cli.md | 30 ++++++++++++++++++++++++++++++ docs/config.md | 2 ++ docs/reference/cli.md | 24 +----------------------- 3 files changed, 33 insertions(+), 23 deletions(-) diff --git a/docs/cli.md b/docs/cli.md index 016c4f2075..a6d0b24544 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -243,6 +243,36 @@ $ nextflow run --files "*.fasta" ``` ::: +Parameters specified on the command line can be also specified in a params file using the `-params-file` option. + +```bash +nextflow run main.nf -params-file pipeline_params.yml +``` + +The `-params-file` option loads parameters for your Nextflow pipeline from a JSON or YAML file. Parameters defined in the file are equivalent to specifying them directly on the command line. For example, instead of specifying parameters on the command line: + +```bash +nextflow run main.nf --alpha 1 --beta foo +``` + +Parameters can be represented in YAML format: + +```yaml +alpha: 1 +beta: 'foo' +``` + +Or in JSON format: + +```json +{ + "alpha": 1, + "beta": "foo" +} +``` + +The parameters specified in a params file are merged with the resolved configuration. The values provided via a params file overwrite those of the same name in the Nextflow configuration file, but not those specified on the command line. + ## Managing projects Nextflow seamlessly integrates with popular Git providers, including [BitBucket](http://bitbucket.org/), [GitHub](http://github.com), and [GitLab](http://gitlab.com) for managing Nextflow pipelines as version-controlled Git repositories. diff --git a/docs/config.md b/docs/config.md index ccdc71933a..05fbc1130b 100644 --- a/docs/config.md +++ b/docs/config.md @@ -138,6 +138,8 @@ params { } ``` +See {ref}`cli-params` for information about how to modify these on the command line. + (config-process)= ## Process configuration diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 43ab14c4e2..591d1fa151 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -1172,29 +1172,7 @@ The `run` command is used to execute a local pipeline script or remote pipeline $ nextflow run main.nf -params-file pipeline_params.yml ``` - For example, the following params file in YAML format: - - ```yaml - alpha: 1 - beta: 'foo' - ``` - - Or in JSON format: - - ```json - { - "alpha": 1, - "beta": "foo" - } - ``` - - Is equivalent to the following command line: - - ```console - $ nextflow run main.nf --alpha 1 --beta foo - ``` - - The parameters specified with this mechanism are merged with the resolved configuration (base configuration and profiles). The values provided via a params file overwrite those of the same name in the Nextflow configuration file. + See {ref}`cli-params` for more information about writing custom parameters files. ### `self-update` From b5c63a9f93b9d6b62e4494f45aedc360439ff668 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Fri, 22 Nov 2024 10:28:42 -0600 Subject: [PATCH 03/13] Improve groupTuple docs with scatter/gather example (#5520) [ci skip] Signed-off-by: Ben Sherman --- docs/snippets/grouptuple-groupkey.nf | 19 ++++++++++--------- docs/snippets/grouptuple-groupkey.out | 9 +++++++-- 2 files changed, 17 insertions(+), 11 deletions(-) diff --git a/docs/snippets/grouptuple-groupkey.nf b/docs/snippets/grouptuple-groupkey.nf index fa9f08c6fd..5c8fce8e7a 100644 --- a/docs/snippets/grouptuple-groupkey.nf +++ b/docs/snippets/grouptuple-groupkey.nf @@ -1,12 +1,13 @@ -chr_frequency = ["chr1": 2, "chr2": 3] - Channel.of( - ['region1', 'chr1', '/path/to/region1_chr1.vcf'], - ['region2', 'chr1', '/path/to/region2_chr1.vcf'], - ['region1', 'chr2', '/path/to/region1_chr2.vcf'], - ['region2', 'chr2', '/path/to/region2_chr2.vcf'], - ['region3', 'chr2', '/path/to/region3_chr2.vcf'] + ['chr1', ['/path/to/region1_chr1.vcf', '/path/to/region2_chr1.vcf']], + ['chr2', ['/path/to/region1_chr2.vcf', '/path/to/region2_chr2.vcf', '/path/to/region3_chr2.vcf']], ) - .map { region, chr, vcf -> tuple( groupKey(chr, chr_frequency[chr]), vcf ) } + .flatMap { chr, vcfs -> + vcfs.collect { vcf -> + tuple(groupKey(chr, vcfs.size()), vcf) // preserve group size with key + } + } + .view { v -> "scattered: ${v}" } .groupTuple() - .view() \ No newline at end of file + .map { key, vcfs -> tuple(key.getGroupTarget(), vcfs) } // unwrap group key + .view { v -> "gathered: ${v}" } \ No newline at end of file diff --git a/docs/snippets/grouptuple-groupkey.out b/docs/snippets/grouptuple-groupkey.out index e97159c872..be3f00185b 100644 --- a/docs/snippets/grouptuple-groupkey.out +++ b/docs/snippets/grouptuple-groupkey.out @@ -1,2 +1,7 @@ -[chr1, [/path/to/region1_chr1.vcf, /path/to/region2_chr1.vcf]] -[chr2, [/path/to/region1_chr2.vcf, /path/to/region2_chr2.vcf, /path/to/region3_chr2.vcf]] \ No newline at end of file +scattered: [chr1, /path/to/region1_chr1.vcf] +scattered: [chr1, /path/to/region2_chr1.vcf] +scattered: [chr2, /path/to/region1_chr2.vcf] +scattered: [chr2, /path/to/region2_chr2.vcf] +scattered: [chr2, /path/to/region3_chr2.vcf] +gathered: [chr1, [/path/to/region1_chr1.vcf, /path/to/region2_chr1.vcf]] +gathered: [chr2, [/path/to/region1_chr2.vcf, /path/to/region2_chr2.vcf, /path/to/region3_chr2.vcf]] \ No newline at end of file From b65fc66ef02a37ea778cd1307debd39490480f79 Mon Sep 17 00:00:00 2001 From: Phil Ewels Date: Fri, 22 Nov 2024 17:32:52 +0100 Subject: [PATCH 04/13] Docs: Improve Conda docs - PyPI + lock files (#5531) [ci skip] Signed-off-by: Phil Ewels Co-authored-by: Christopher Hakkaart Co-authored-by: Paolo Di Tommaso --- docs/conda.md | 77 +++++++++++++++++++++++++++++++++++++++------------ docs/wave.md | 2 ++ 2 files changed, 62 insertions(+), 17 deletions(-) diff --git a/docs/conda.md b/docs/conda.md index 6debcec745..8c7405c1a4 100644 --- a/docs/conda.md +++ b/docs/conda.md @@ -6,7 +6,7 @@ Nextflow has built-in support for Conda that allows the configuration of workflow dependencies using Conda recipes and environment files. -This allows Nextflow applications to use popular tool collections such as [Bioconda](https://bioconda.github.io) whilst taking advantage of the configuration flexibility provided by Nextflow. +This allows Nextflow applications to use popular tool collections such as [Bioconda](https://bioconda.github.io) and the [Python Package index](https://pypi.org/), whilst taking advantage of the configuration flexibility provided by Nextflow. ## Prerequisites @@ -22,7 +22,7 @@ Dependencies are specified by using the {ref}`process-conda` directive, providin Conda environments are stored on the file system. By default, Nextflow instructs Conda to save the required environments in the pipeline work directory. The same environment may be created/saved multiple times across multiple executions when using different work directories. ::: -You can specify the directory where the Conda environments are stored using the `conda.cacheDir` configuration property. When using a computing cluster, make sure to use a shared file system path accessible from all compute nodes. See the {ref}`configuration page ` for details about Conda configuration. +You can specify the directory where the Conda environments are stored using the `conda.cacheDir` configuration property. When using a computing cluster, make sure to use a shared file system path accessible from all compute nodes. See the {ref}`configuration page ` for details about Conda configuration. :::{warning} The Conda environment feature is not supported by executors that use remote object storage as a work directory. For example, AWS Batch. @@ -62,6 +62,7 @@ The usual Conda package syntax and naming conventions can be used. The version o The name of the channel where a package is located can be specified prefixing the package with the channel name as shown here `bioconda::bwa=0.7.15`. +(conda-env-files)= ### Use Conda environment files Conda environments can also be defined using one or more Conda environment files. This is a file that lists the required packages and channels structured using the YAML format. For example: @@ -77,20 +78,6 @@ dependencies: - bwa=0.7.15 ``` -This other example shows how to leverage a Conda environment file to install Python packages from the [PyPI repository](https://pypi.org/)), through the `pip` package manager (which must also be explicitly listed as a required package): - -```yaml -name: my-env-2 -channels: - - defaults -dependencies: - - pip - - pip: - - numpy - - pandas - - matplotlib -``` - Read the Conda documentation for more details about how to create [environment files](https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually). The path of an environment file can be specified using the `conda` directive: @@ -110,7 +97,26 @@ process foo { The environment file name **must** have a `.yml` or `.yaml` extension or else it won't be properly recognised. ::: -Alternatively, it is possible to provide the dependencies using a plain text file, just listing each package name as a separate line. For example: +(conda-pypi)= +### Python Packages from PyPI + +Conda environment files can also be used to install Python packages from the [PyPI repository](https://pypi.org/), through the `pip` package manager (which must also be explicitly listed as a required package): + +```yaml +name: my-env-2 +channels: + - defaults +dependencies: + - pip + - pip: + - numpy + - pandas + - matplotlib +``` + +### Conda text files + +It is possible to provide dependencies by listing each package name as a separate line in a plain text file. For example: ``` bioconda::star=2.5.4a @@ -122,6 +128,43 @@ bioconda::multiqc=1.4 Like before, the extension matters. Make sure the dependencies file has a `.txt` extension. ::: +### Conda lock files + +The final way to provide packages to Conda is with [Conda lock files](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#identical-conda-envs). + +These are generated from existing Conda environments using the following command: + +```bash +conda list --explicit > spec-file.txt +``` + +or if using Mamba / Micromamba: + +```bash +micromamba env export --explicit > spec-file.txt +``` + +Conda lock files can also be downloaded from [Wave](https://seqera.io/wave/) build pages. + +These files include every package and their dependencies. As such, no Conda environment resolution step is needed. This is faster and more reproducible. + +The files contain package URLs and an optional md5hash for each download to confirm identity: + +``` +# micromamba env export --explicit +# This file may be used to create an environment using: +# $ conda create --name --file +# platform: linux-64 +@EXPLICIT +https://conda.anaconda.org/conda-forge/linux-64/_libgcc_mutex-0.1-conda_forge.tar.bz2#d7c89558ba9fa0495403155b64376d81 +https://conda.anaconda.org/conda-forge/linux-64/libgomp-13.2.0-h77fa898_7.conda#abf3fec87c2563697defa759dec3d639 +https://conda.anaconda.org/conda-forge/linux-64/_openmp_mutex-4.5-2_gnu.tar.bz2#73aaf86a425cc6e73fcf236a5a46396d +https://conda.anaconda.org/conda-forge/linux-64/libgcc-ng-13.2.0-h77fa898_7.conda#72ec1b1b04c4d15d4204ece1ecea5978 +# .. and so on +``` + +To use with Nextflow, simply set the `conda` directive to the lock file path. + ### Use existing Conda environments If you already have a local Conda environment, you can use it in your workflow specifying the installation directory of such environment by using the `conda` directive: diff --git a/docs/wave.md b/docs/wave.md index 668de765f0..c27c034f6b 100644 --- a/docs/wave.md +++ b/docs/wave.md @@ -90,6 +90,8 @@ conda.channels = 'conda-forge,bioconda' ``` ::: +Packages from the [Python Package Index](https://pypi.org/) can also be added to a Conda `environment.yml` file. See {ref}`Conda and PyPI ` for more information. + (wave-singularity)= ### Build Singularity native images From 508af3b10540b797965701d87adc54a4ba3a390b Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Tue, 26 Nov 2024 12:20:58 +0100 Subject: [PATCH 05/13] Improve installer docs (#5541) [ci skip] Signed-off-by: Paolo Di Tommaso Co-authored-by: Christopher Hakkaart --- docs/install.md | 19 ++++++++++++++----- 1 file changed, 14 insertions(+), 5 deletions(-) diff --git a/docs/install.md b/docs/install.md index cb1f8bcad8..246ce57fcb 100644 --- a/docs/install.md +++ b/docs/install.md @@ -126,9 +126,18 @@ libraries. This distribution is mainly useful for offline environments. Note however the support for cloud services e.g. AWS, Seqera Platform, Wave, etc. still require the download of the corresponding Nextflow plugins. -The installer for the `dist` distribution can be found on the [GitHub releases page](https://github.com/nextflow-io/nextflow/releases), under the "Assets" section for a specific release. The installation procedure is the same as for the standard distribution, only using this URL instead of `https://get.nextflow.io`: +To use the standalone distribution: -```bash -export NXF_VER=24.10.0 -curl -s https://github.com/nextflow-io/nextflow/releases/download/v$NXF_VER/nextflow-$NXF_VER-dist -``` +1. Download it from the [GitHub releases page](https://github.com/nextflow-io/nextflow/releases), under the "Assets" section for a specific + +2. Grant execution permissions to the downloaded file e.g. + + ``` + chmod -x nextflow-24.10.1-dist + ``` + +3. Then you can use it as a drop-in replacement for `nextflow` command. For example: + + ``` + ./nextflow-24.10.1-dist run hello + ``` From b5e31bb05cc923199eb4b13a39e58754e0e2c945 Mon Sep 17 00:00:00 2001 From: Christopher Hakkaart Date: Tue, 26 Nov 2024 16:32:26 +0100 Subject: [PATCH 06/13] Add singularity.libraryDir to config page (#5498) [ci skip] Signed-off-by: Christopher Hakkaart --- docs/container.md | 20 ++++++++++---------- docs/reference/config.md | 6 ++++++ 2 files changed, 16 insertions(+), 10 deletions(-) diff --git a/docs/container.md b/docs/container.md index 8eb8df5c51..ba8a563828 100644 --- a/docs/container.md +++ b/docs/container.md @@ -94,7 +94,7 @@ Read the {ref}`Process scope ` section to learn more about proce Nextflow is able to transparently pull remote container images stored in any Docker compatible registry. -By default when a container name is specified, Nextflow checks if an image file with that name exists in the local file system. If that image file exists, it's used to execute the container. If a matching file does not exist, Nextflow automatically tries to pull an image with the specified name from the container registry. +By default, when a container name is specified, Nextflow checks if an image file with that name exists in the local file system. If that image file exists, it's used to execute the container. If a matching file does not exist, Nextflow automatically tries to pull an image with the specified name from the container registry. If you want Nextflow to check only for local file images, prefix the container name with the `file://` pseudo-protocol. For example: @@ -107,7 +107,7 @@ apptainer.enabled = true Use three `/` slashes to specify an **absolute** file path, otherwise the path will be interpreted as relative to the workflow launch directory. ::: -To pull images from Apptainer Hub or a third party Docker registry, simply prefix the image name with the `shub://`, `docker://` or `docker-daemon://` pseudo-protocol as required by Apptainer. For example: +To pull images from Apptainer Hub or a third party Docker registry, prefix the image name with the `shub://`, `docker://` or `docker-daemon://` pseudo-protocol as required by Apptainer. For example: ```groovy process.container = 'docker://quay.io/biocontainers/multiqc:1.3--py35_2' @@ -120,11 +120,11 @@ You do not need to specify `docker://` to pull from a Docker repository. Nextflo This feature requires the `apptainer` tool to be installed where the workflow execution is launched (as opposed to the compute nodes). ::: -Nextflow caches those images in the `apptainer` directory in the pipeline work directory by default. However it is suggested to provide a centralised cache directory by using either the `NXF_APPTAINER_CACHEDIR` environment variable or the `apptainer.cacheDir` setting in the Nextflow config file. +Nextflow caches Apptainer images in the `apptainer` directory, in the pipeline work directory, by default. However, it is recommended to provide a centralized cache directory using the `NXF_APPTAINER_CACHEDIR` environment variable or the `apptainer.cacheDir` setting in the Nextflow config file. -:::{versionadded} 21.09.0-edge -When looking for a Apptainer image file, Nextflow first checks the *library* directory, and if the image file is not found, the *cache* directory is used as usual. The library directory can be defined either using the `NXF_APPTAINER_LIBRARYDIR` environment variable or the `apptainer.libraryDir` configuration setting (the latter overrides the former). -::: +Nextflow uses the library directory to determine the location of Apptainer containers. The library directory can be defined using the `apptainer.libraryDir` configuration setting or the `NXF_APPTAINER_LIBRARYDIR` environment variable. The configuration file option overrides the environment variable if both are set. + +Nextflow first checks the library directory when searching for the image. If the image is not found it then checks the cache directory. The main difference between the library directory and the cache directory is that the first is assumed to be a read-only container repository, while the latter is expected to be writable path where container images can added for caching purposes. :::{warning} When using a compute cluster, the Apptainer cache directory must reside in a shared filesystem accessible to all compute nodes. @@ -653,11 +653,11 @@ process.container = 'library://library/default/alpine:3.8' The `library://` pseudo-protocol allows you to import Singularity images from a local Docker installation instead of downloading them from a Docker registry. This feature requires the `singularity` tool to be installed where the workflow execution is launched (as opposed to the compute nodes). -Nextflow caches the images in `${NXF_WORK}/singularity` by default. However, it is recommended to define a centralised cache directory using either the `NXF_SINGULARITY_CACHEDIR` environment variable or the `singularity.cacheDir` setting in the Nextflow config file. +Nextflow caches Singularity images in the `singularity` directory, in the pipeline work directory, by default. However, it is recommended to provide a centralized cache directory using the `NXF_SINGULARITY_CACHEDIR` environment variable or the `singularity.cacheDir` setting in the Nextflow config file. -:::{versionadded} 21.09.0-edge -When looking for a Singularity image file, Nextflow first checks the *library* directory, and if the image file is not found, the *cache* directory is used as usual. The library directory can be defined either using the `NXF_SINGULARITY_LIBRARYDIR` environment variable or the `singularity.libraryDir` configuration setting (the latter overrides the former). -::: +Nextflow uses the library directory to determine the location of Singularity images. The library directory can be defined using the `singularity.libraryDir` configuration setting or the `NXF_SINGULARITY_LIBRARYDIR` environment variable. The configuration file option overrides the environment variable if both are set. + +Nextflow first checks the library directory when searching for the image. If the image is not found it then checks the cache directory. The main difference between the library directory and the cache directory is that the first is assumed to be a read-only container repository, while the latter is expected to be writable path where container images can added for caching purposes. :::{warning} When using a compute cluster, the Singularity cache directory must reside in a shared filesystem accessible to all compute nodes. diff --git a/docs/reference/config.md b/docs/reference/config.md index 464ca480ba..0efb1ce6ae 100644 --- a/docs/reference/config.md +++ b/docs/reference/config.md @@ -52,6 +52,9 @@ The following settings are available: `apptainer.envWhitelist` : Comma separated list of environment variable names to be included in the container environment. +`apptainer.libraryDir` +: Directory where remote Apptainer images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. + `apptainer.noHttps` : Pull the Apptainer image with http protocol (default: `false`). @@ -1375,6 +1378,9 @@ The following settings are available: `singularity.envWhitelist` : Comma separated list of environment variable names to be included in the container environment. +`singularity.libraryDir` +: Directory where remote Singularity images are retrieved. When using a computing cluster it must be a shared folder accessible to all compute nodes. + `singularity.noHttps` : Pull the Singularity image with http protocol (default: `false`). From 9248c04d86b30d05f0d34ecc79fda080ecd39fd3 Mon Sep 17 00:00:00 2001 From: Jorge Ejarque Date: Tue, 26 Nov 2024 16:45:23 +0100 Subject: [PATCH 07/13] Fix overlapping conda lock file (#5540) Signed-off-by: jorgee --- .../groovy/nextflow/conda/CondaCache.groovy | 17 +++++++++-------- .../groovy/nextflow/conda/CondaCacheTest.groovy | 15 +++++---------- 2 files changed, 14 insertions(+), 18 deletions(-) diff --git a/modules/nextflow/src/main/groovy/nextflow/conda/CondaCache.groovy b/modules/nextflow/src/main/groovy/nextflow/conda/CondaCache.groovy index 17c605f19b..15234c6039 100644 --- a/modules/nextflow/src/main/groovy/nextflow/conda/CondaCache.groovy +++ b/modules/nextflow/src/main/groovy/nextflow/conda/CondaCache.groovy @@ -251,8 +251,8 @@ class CondaCache { * @return the conda environment prefix {@link Path} */ @PackageScope - Path createLocalCondaEnv(String condaEnv) { - final prefixPath = condaPrefixPath(condaEnv) + Path createLocalCondaEnv(String condaEnv, Path prefixPath) { + if( prefixPath.isDirectory() ) { log.debug "${binaryName} found local env for environment=$condaEnv; path=$prefixPath" return prefixPath @@ -360,17 +360,18 @@ class CondaCache { */ @PackageScope DataflowVariable getLazyImagePath(String condaEnv) { - - if( condaEnv in condaPrefixPaths ) { + final prefixPath = condaPrefixPath(condaEnv) + final condaEnvPath = prefixPath.toString() + if( condaEnvPath in condaPrefixPaths ) { log.trace "${binaryName} found local environment `$condaEnv`" - return condaPrefixPaths[condaEnv] + return condaPrefixPaths[condaEnvPath] } synchronized (condaPrefixPaths) { - def result = condaPrefixPaths[condaEnv] + def result = condaPrefixPaths[condaEnvPath] if( result == null ) { - result = new LazyDataflowVariable({ createLocalCondaEnv(condaEnv) }) - condaPrefixPaths[condaEnv] = result + result = new LazyDataflowVariable({ createLocalCondaEnv(condaEnv, prefixPath) }) + condaPrefixPaths[condaEnvPath] = result } else { log.trace "${binaryName} found local cache for environment `$condaEnv` (2)" diff --git a/modules/nextflow/src/test/groovy/nextflow/conda/CondaCacheTest.groovy b/modules/nextflow/src/test/groovy/nextflow/conda/CondaCacheTest.groovy index 9a8baf952c..f6b7fac271 100644 --- a/modules/nextflow/src/test/groovy/nextflow/conda/CondaCacheTest.groovy +++ b/modules/nextflow/src/test/groovy/nextflow/conda/CondaCacheTest.groovy @@ -197,9 +197,8 @@ class CondaCacheTest extends Specification { when: // the prefix directory exists ==> no conda command is executed - def result = cache.createLocalCondaEnv(ENV) + def result = cache.createLocalCondaEnv(ENV, PREFIX) then: - 1 * cache.condaPrefixPath(ENV) >> PREFIX 0 * cache.isYamlFilePath(ENV) 0 * cache.runCommand(_) result == PREFIX @@ -224,9 +223,8 @@ class CondaCacheTest extends Specification { when: // the prefix directory exists ==> no mamba command is executed - def result = cache.createLocalCondaEnv(ENV) + def result = cache.createLocalCondaEnv(ENV, PREFIX) then: - 1 * cache.condaPrefixPath(ENV) >> PREFIX 0 * cache.isYamlFilePath(ENV) 0 * cache.runCommand(_) result == PREFIX @@ -251,9 +249,8 @@ class CondaCacheTest extends Specification { when: // the prefix directory exists ==> no mamba command is executed - def result = cache.createLocalCondaEnv(ENV) + def result = cache.createLocalCondaEnv(ENV, PREFIX) then: - 1 * cache.condaPrefixPath(ENV) >> PREFIX 0 * cache.isYamlFilePath(ENV) 0 * cache.runCommand(_) result == PREFIX @@ -278,9 +275,8 @@ class CondaCacheTest extends Specification { when: // the prefix directory exists ==> no mamba command is executed - def result = cache.createLocalCondaEnv(ENV) + def result = cache.createLocalCondaEnv(ENV, PREFIX) then: - 1 * cache.condaPrefixPath(ENV) >> PREFIX 0 * cache.isYamlFilePath(ENV) 0 * cache.runCommand(_) result == PREFIX @@ -304,9 +300,8 @@ class CondaCacheTest extends Specification { when: // the prefix directory exists ==> no mamba command is executed - def result = cache.createLocalCondaEnv(ENV) + def result = cache.createLocalCondaEnv(ENV, PREFIX) then: - 1 * cache.condaPrefixPath(ENV) >> PREFIX 0 * cache.isYamlFilePath(ENV) 0 * cache.runCommand(_) result == PREFIX From 308d5a5e893a7b3691d58025cbf673d62e1d7481 Mon Sep 17 00:00:00 2001 From: Ben Sherman Date: Tue, 26 Nov 2024 15:25:07 -0600 Subject: [PATCH 08/13] Update syntax docs (#5542) Signed-off-by: Ben Sherman --- docs/reference/syntax.md | 10 ----- docs/vscode.md | 90 +++++++++++++++++++++++++--------------- 2 files changed, 57 insertions(+), 43 deletions(-) diff --git a/docs/reference/syntax.md b/docs/reference/syntax.md index e454756646..06d7824527 100644 --- a/docs/reference/syntax.md +++ b/docs/reference/syntax.md @@ -622,16 +622,6 @@ A *slashy string* is enclosed by slashes instead of quotes: /no escape!/ ``` -Slashy strings can also span multiple lines: - -```nextflow -/ -Patterns in the code, -Symbols dance to match and find, -Logic unconfined. -/ -``` - :::{note} A slashy string cannot be empty because it would become a line comment. ::: diff --git a/docs/vscode.md b/docs/vscode.md index eb0d6d40eb..11c847ae29 100644 --- a/docs/vscode.md +++ b/docs/vscode.md @@ -230,38 +230,6 @@ if (aligner == 'bowtie2') { } ``` -**Slashy dollar strings** - -Groovy supports a wide variety of strings, including multi-line strings, dynamic strings, slashy strings, multi-line dynamic slashy strings, and more. - -The Nextflow language specification supports single- and double-quoted strings, multi-line strings, and slashy strings. Dynamic slashy strings are not supported: - -```groovy -def logo = /--cl-config 'custom_logo: "${multiqc_logo}"'/ -``` - -Use a double-quoted string instead: - -```nextflow -def logo = "--cl-config 'custom_logo: \"${multiqc_logo}\"'" -``` - -Slashy dollar strings are not supported: - -```groovy -$/ -echo "Hello world!" -/$ -``` - -Use a multi-line string instead: - -```nextflow -""" -echo "Hello world!" -""" -``` - **Implicit environment variables** In Nextflow DSL1 and DSL2, you can reference environment variables directly in strings: @@ -334,6 +302,62 @@ To ease the migration of existing scripts, the language server only reports warn Type annotations and static type checking will be addressed in a future version of the Nextflow language specification. ::: +**Strings** + +Groovy supports a wide variety of strings, including multi-line strings, dynamic strings, slashy strings, multi-line dynamic slashy strings, and more. + +The Nextflow language specification supports single- and double-quoted strings, multi-line strings, and slashy strings. + +Slashy strings cannot be interpolated: + +```nextflow +def id = 'SRA001' +assert 'SRA001.fastq' ~= /${id}\.f(?:ast)?q/ +``` + +Use a double-quoted string instead: + +```nextflow +def id = 'SRA001' +assert 'SRA001.fastq' ~= "${id}\\.f(?:ast)?q" +``` + +Slashy strings cannot span multiple lines: + +```groovy +/ +Patterns in the code, +Symbols dance to match and find, +Logic unconfined. +/ +``` + +Use a multi-line string instead: + +```nextflow +""" +Patterns in the code, +Symbols dance to match and find, +Logic unconfined. +""" +``` + +Dollar slashy strings are not supported: + +```groovy +$/ +echo "Hello world!" +/$ +``` + +Use a multi-line string instead: + +```nextflow +""" +echo "Hello world!" +""" +``` + **Process env inputs/outputs** In Nextflow DSL1 and DSL2, the name of a process `env` input/output can be specified with or without quotes: @@ -481,7 +505,7 @@ includeConfig ({ return 'large.config' else return '/dev/null' -})() +}()) ``` The include source is a closure that is immediately invoked. It includes a different config file based on the return value of the closure. Including `/dev/null` is equivalent to including nothing. From 12fc1d60027d9d2db95170b3780b25e2bfc12a7e Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 27 Nov 2024 15:55:57 +0100 Subject: [PATCH 09/13] Prevent NPE with null AWS Batch response Signed-off-by: Paolo Di Tommaso --- .../main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy b/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy index 4245821a7a..29a2261e25 100644 --- a/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy +++ b/plugins/nf-amazon/src/main/nextflow/cloud/aws/batch/AwsBatchTaskHandler.groovy @@ -198,7 +198,7 @@ class AwsBatchTaskHandler extends TaskHandler implements BatchHandler Date: Wed, 27 Nov 2024 22:12:10 +0100 Subject: [PATCH 10/13] Update wave deps Signed-off-by: Paolo Di Tommaso --- plugins/nf-wave/build.gradle | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plugins/nf-wave/build.gradle b/plugins/nf-wave/build.gradle index bd5da121f0..b6a8367f0d 100644 --- a/plugins/nf-wave/build.gradle +++ b/plugins/nf-wave/build.gradle @@ -36,8 +36,8 @@ dependencies { api 'org.apache.commons:commons-lang3:3.12.0' api 'com.google.code.gson:gson:2.10.1' api 'org.yaml:snakeyaml:2.2' - api 'io.seqera:wave-api:0.13.3' - api 'io.seqera:wave-utils:0.14.1' + api 'io.seqera:wave-api:0.14.0' + api 'io.seqera:wave-utils:0.15.0' testImplementation(testFixtures(project(":nextflow"))) testImplementation "org.apache.groovy:groovy:4.0.24" From ee252173bf823d4bbef5522bd726ccd4df595a7d Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 27 Nov 2024 22:13:09 +0100 Subject: [PATCH 11/13] Fix missing wave response (#5547) [ci fast] Signed-off-by: Paolo Di Tommaso --- .../io/seqera/wave/plugin/WaveClient.groovy | 17 ++++++++++++----- .../io/seqera/wave/plugin/WaveClientTest.groovy | 17 ++++++++--------- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy b/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy index 6c8bb44e8e..c3640ce42d 100644 --- a/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy +++ b/plugins/nf-wave/src/main/io/seqera/wave/plugin/WaveClient.groovy @@ -27,6 +27,7 @@ import java.time.Duration import java.time.Instant import java.time.OffsetDateTime import java.time.temporal.ChronoUnit +import java.util.concurrent.ConcurrentHashMap import java.util.concurrent.Executors import java.util.concurrent.TimeUnit import java.util.function.Predicate @@ -104,7 +105,9 @@ class WaveClient { final private String endpoint - private Cache cache + private Cache cache + + private Map responses = new ConcurrentHashMap<>() private Session session @@ -135,7 +138,7 @@ class WaveClient { this.packer = new Packer().withPreserveTimestamp(config.preserveFileTimestamp()) this.waveRegistry = new URI(endpoint).getAuthority() // create cache - cache = CacheBuilder + this.cache = CacheBuilder .newBuilder() .expireAfterWrite(config.tokensCacheMaxDuration().toSeconds(), TimeUnit.SECONDS) .build() @@ -572,8 +575,12 @@ class WaveClient { final key = assets.fingerprint() log.trace "Wave fingerprint: $key; assets: $assets" // get from cache or submit a new request - final handle = cache.get(key, () -> new Handle(sendRequest(assets),Instant.now()) ) - return new ContainerInfo(assets.containerImage, handle.response.targetImage, key) + final resp = cache.get(key, () -> { + final ret = sendRequest(assets); + responses.put(key,new Handle(ret,Instant.now())); + return ret + }) + return new ContainerInfo(assets.containerImage, resp.targetImage, key) } catch ( UncheckedExecutionException e ) { throw e.cause @@ -633,7 +640,7 @@ class WaveClient { } boolean isContainerReady(String key) { - final handle = cache.getIfPresent(key) + final handle = responses.get(key) if( !handle ) throw new IllegalStateException("Unable to find any container with key: $key") final resp = handle.response diff --git a/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy b/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy index bbd0a397b6..1f54b0a3d7 100644 --- a/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy +++ b/plugins/nf-wave/src/test/io/seqera/wave/plugin/WaveClientTest.groovy @@ -27,7 +27,6 @@ import java.nio.file.attribute.FileTime import java.time.Duration import java.time.Instant -import com.google.common.cache.Cache import com.sun.net.httpserver.HttpExchange import com.sun.net.httpserver.HttpHandler import com.sun.net.httpserver.HttpServer @@ -1303,18 +1302,18 @@ class WaveClientTest extends Specification { def 'should validate isContainerReady' () { given: def sess = Mock(Session) {getConfig() >> [wave: [build:[maxDuration: '500ms']]] } - def cache = Mock(Cache) + def cache = Mock(Map) and: def resp = Mock(SubmitContainerTokenResponse) def handle = new WaveClient.Handle(resp,Instant.now()) - def wave = Spy(new WaveClient(session:sess, cache: cache)) + def wave = Spy(new WaveClient(session:sess, responses: cache)) boolean ready // container succeeded when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.requestId >> '12345' resp.succeeded >> true @@ -1328,7 +1327,7 @@ class WaveClientTest extends Specification { when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.requestId >> '12345' resp.succeeded >> null @@ -1342,7 +1341,7 @@ class WaveClientTest extends Specification { when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.requestId >> '12345' resp.succeeded >> false @@ -1357,7 +1356,7 @@ class WaveClientTest extends Specification { when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.buildId >> 'bd-5678' resp.cached >> false @@ -1371,7 +1370,7 @@ class WaveClientTest extends Specification { when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.requestId >> null resp.buildId >> 'bd-5678' @@ -1386,7 +1385,7 @@ class WaveClientTest extends Specification { when: ready = wave.isContainerReady('xyz') then: - cache.getIfPresent('xyz') >> handle + cache.get('xyz') >> handle and: resp.requestId >> null resp.buildId >> 'bd-5678' From fc5e2c2a8e4608c19d7072f622f3368024420c47 Mon Sep 17 00:00:00 2001 From: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Date: Fri, 29 Nov 2024 18:47:38 +0000 Subject: [PATCH 12/13] Incorrect CPU value in Azure example (#5549) [ci skip] Signed-off-by: Adam Talbot <12817534+adamrtalbot@users.noreply.github.com> Signed-off-by: adamrtalbot <12817534+adamrtalbot@users.noreply.github.com> --- docs/azure.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/azure.md b/docs/azure.md index 6087c18ec2..61269e43ac 100644 --- a/docs/azure.md +++ b/docs/azure.md @@ -167,12 +167,12 @@ To specify multiple Azure machine families, use a comma separated list with glob process.machineType = "Standard_D*d_v5,Standard_E*d_v5" ``` -For example, the following process will create a pool of `Standard_E4d_v5` machines based when using `autoPoolMode`: +For example, the following process will create a pool of `Standard_E8d_v5` machines based when using `autoPoolMode`: ```nextflow process EXAMPLE_PROCESS { machineType "Standard_E*d_v5" - cpus 16 + cpus 8 memory 8.GB script: From ab13ce58b2b176afd6cabbeb7149d8f0611a77f8 Mon Sep 17 00:00:00 2001 From: Paolo Di Tommaso Date: Wed, 27 Nov 2024 22:30:30 +0100 Subject: [PATCH 13/13] Update changelog [ci skip] Signed-off-by: Paolo Di Tommaso --- changelog.txt | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/changelog.txt b/changelog.txt index 3420a4041e..0be1d17adf 100644 --- a/changelog.txt +++ b/changelog.txt @@ -1,5 +1,20 @@ NEXTFLOW CHANGE-LOG =================== +24.10.2 - 27 Nov 2024 +- Prevent NPE with null AWS Batch response [3d491934] +- Fix overlapping conda lock file (#5540) [df66deaa] +- Fix missing wave response (#5547) [eb85cda8] +- Bump nf-wave@1.7.4 [93d09404] +- Bump nf-amazon@2.9.2 [469a35dd] + +24.10.1 - 18 Nov 2024 +- Fix overlapping file lock exception (#5489) [a2566d54] +- Fix isContainerReady when wave is disabled (#5509) [c69e3711] +- Bump nf-wave@1.7.3 [e7709a0f] +- Bump nf-azure@1.10.2 [54496ac4] +- Bump nf-amazon@2.9.1 [fa227933] +- Bump netty-common to version 4.1.115.Final [90623c1e] + 24.10.0 - 27 Oct 2024 - Add `manifest.contributors` config option (#5322) [cf0f9690] - Add wave mirror and scan config [92e69776]