Merge branch 'master' into master

.github/workflows/build.yml

@@ -12,6 +12,7 @@ on:
       - 'master'
       - 'test*'
       - 'dev*'
+      - 'STABLE-*'
   pull_request:
     types: [opened, reopened, synchronize]
   workflow_dispatch:
@@ -38,7 +39,7 @@ jobs:
 
       - name: Get changed files
         id: changed-files
-        uses: tj-actions/changed-files@v35
+        uses: tj-actions/changed-files@v41
         with:
           files_ignore: docs
 

VERSION

@@ -1 +1 @@
-23.11.0-edge
+23.12.0-edge

changelog.txt

@@ -1,5 +1,51 @@
 NEXTFLOW CHANGE-LOG
 ===================
+23.10.1 - 12 Jan 2023 
+- Fix bug with Fusion symlink resolution (#4593) [f28c9e48]
+- Fix Fusion symlinks when publishing files (#4348) [1fa5878a]
+- Fix Inspect command fails with Singularity [25883df3]
+- Fix Allow the use of error built-in function in onComplete handler (#4458) [ci fast] [4be10cd3]
+- Fix Harden regular expression to used to strip secrets in logs (#4563) [ci fast] [0102d4d6]
+- Fix custom notification template [40980bcb]
+- Fix container environment with special chars (#4594) [f4e00601]
+- Fix AZURE_STORAGE_SAS_TOKEN environment variable (#4627) [2e1cb413]
+- Fix azure retry policy (#4638) [2bc3cf0e]
+- Fix BitBucket get source API with custom branch [5ef75e32]
+- Improve error details for AbortOperationException [9e795a62]
+- Bump [email protected] [206f2614]
+- Bump [email protected] [50bcad59]
+
+23.04.5 - 12 Jan 2023
+- Fix container environment with special chars (#4594) [663b2936]
+
+23.12.0-edge - 20 Dec 2023 
+- Add AWS_SESSION_TOKEN to Fusion environment (#4581) [552f29b0]
+- Add ability to disable Cloudinfo service (#4606) [f7251895]
+- Add experimental support for Fargate compute type for AWS Batch  (#3474) [47cf335b]
+- Add support for Instance template to Google Batch [df7ed294]
+- Add support for Singularity/Apptainer auto pull mode for OCI containers [b7f1a192]
+- Fix BitBucket get source API with custom branch [58937831]
+- Fix Fusion tags documentation (#4551) [687e2e96]
+- Fix Harden regular expression to used to strip secrets in logs (#4563) [832bff24]
+- Fix bug with Fusion symlink resolution (#4593) [09e85582]
+- Fix container environment with special chars (#4594) [e0fe952f]
+- Fix custom notification template [ccf4f59e]
+- Fix fusion symlink test (#4604) [681ace86]
+- Fix smoke tests [d3c2f330]
+- Improve GLS tests [58590b1c]
+- Load nf-amazon when AWS SES is enabled [887f06f4]
+- Move build num & timestamp to BuildInfo class [ec8083d4]
+- Move app version to BuildInfo class [c7d749e8]
+- Remove deprecated Wave observer [0e009ef7]
+- Remove undocumented userEmulation mode (#4596) [f6c79788]
+- Remove unused DSL2 check [e9ee3b2c]
+- Replace each iterator with class for [f7662e68]
+- Bump [email protected] [0b40b7b9]
+- Bump [email protected] [bcb20fcf]
+- Bump [email protected] [aa981814]
+- Bump [email protected] and [email protected] [9cb50035]
+- Build optimizations (#4579) [5ad41e44]
+
 23.11.0-edge - 24 Nov 2023
 - Add `fusion.cacheSize` config option (#4518) [2faadc22]
 - Add Topic channel type (experimental) (#4459) [921313d1]
@@ -16,29 +62,29 @@ NEXTFLOW CHANGE-LOG
 - Add support for Azure low-priority pool (#4527) [8320ea10]
 - Add support for FUSION_AWS_REGION (#4481) [8f8b09fa]
 - Add support for Fusion when using Singularity OCI mode (#4508) [4f3aa631]
-- Add support for K8s schedulerName pod spec (#4485) [ci fast] [dfc7b7c8]
+- Add support for K8s schedulerName pod spec (#4485) [dfc7b7c8]
 - Add support for Singularity OCI mode (#4440) [f5362a7b]
-- Allow the use of error built-in function in onComplete handler (#4458) [ci fast] [35a4424b]
-- Fix Bug in JsonSplitter ordering [ci fast] [8ec14dd2]
+- Allow the use of error built-in function in onComplete handler (#4458) [35a4424b]
+- Fix Bug in JsonSplitter ordering [8ec14dd2]
 - Fix Bypass Google Batch Price query if task cpus and memory are defined (#4521) [7f8f20d3]
 - Fix Checkout remote tag if checkout remote branch fails (#4247) [b8907ccb]
 - Fix Fusion symlinks when publishing files (#4348) [89f09fe0]
 - Fix Inspect command fails with Singularity [f5bb829f]
 - Fix ParamsMap copyWith param aliases (#4188) [b480ee0e]
 - Fix Singularity docs [e952299f]
 - Fix container hashing for Singularity + Wave containers [4c6f2e85]
-- Fix detection of Conda local path made by Wave client [ci fast] (#4532) [4d5bc216]
+- Fix detection of Conda local path made by Wave client (#4532) [4d5bc216]
 - Fix doc tests to fail on test failure (#4505) [4d326551]
-- Fix errors when NXF_HOME contains spaces (#4456) [ci fast] [fe5bea99]
+- Fix errors when NXF_HOME contains spaces (#4456) [fe5bea99]
 - Fix Google Batch network/subnetwork docs (#4475) [27d132f3]
-- Fix rounding error with long durations (#4496) [ci fast] [0356178b]
+- Fix rounding error with long durations (#4496) [0356178b]
 - Fix security vulnerabilities (#4513) [a310c777]
 - Fix Use consistently NXF_TASK_WORKDIR (#4484) [48ee3c64]
 - Improve error details for AbortOperationException [35609cb0]
 - Improve operator docs (#4502) [38210e11]
 - Makefile clean to also remove buildSrc/build (#4517) [2ccb05d0]
-- Minor test improvements [ci fast] [171831ea]
-- Minor types improvement for mix operator [ci fast] [91c1ab15]
+- Minor test improvements [171831ea]
+- Minor types improvement for mix operator [91c1ab15]
 - Normalise channel docs [b641d677]
 - Remove deprecated TowerArchiver feature [ff8e06a3]
 - Remove dsl1 deprecated code (part 2) [159effb1]
@@ -138,6 +184,14 @@ NEXTFLOW CHANGE-LOG
 - Bump [email protected] [0cf2476c]
 - Bump [email protected] [d0c47d49]
 
+23.04.4 - 25 Sep 2023
+- Fix use of GITHUB_TOKEN variable to access GitHub repo [108c6b55]
+- Fix allow_other vulnerability preventing google-batch submissions (#4332) [4895d547]
+- Fix Prevent false positive resumable task [aae87715]
+- Fix Always emit publish event for cached task outputs (#4227) [c4cd53c2]
+- Fix Too long Http connection pool timeout [ce5e9930]
+- Bump [email protected] [d881728c]
+
 23.09.1-edge - 11 Sep 2023
 - Revert "Allow setting shell directive when using the trace file (#4210)" [9f9edcdc]
 

docs/conda.md

@@ -76,6 +76,20 @@ dependencies:
   - bwa=0.7.15
 ```
 
+This other example shows how to leverage a Conda environment file to install Python packages from the [PyPI repository](https://pypi.org/)), through the `pip` package manager (which must also be explicitly listed as a required package):
+
+```yaml
+name: my-env-2
+channels:
+  - defaults
+dependencies:
+  - pip
+  - pip:
+    - numpy
+    - pandas
+    - matplotlib
+```
+
 Read the Conda documentation for more details about how to create [environment files](https://conda.io/docs/user-guide/tasks/manage-environments.html#creating-an-environment-file-manually).
 
 The path of an environment file can be specified using the `conda` directive:

docs/conf.py

@@ -64,9 +64,9 @@
 # built documents.
 #
 # The short X.Y version.
-version = '23.11'
+version = '23.12'
 # The full version, including alpha/beta/rc tags.
-release = '23.11.0-edge'
+release = '23.12.0-edge'
 
 # The language for content autogenerated by Sphinx. Refer to documentation
 # for a list of supported languages.

docs/config.md

@@ -880,6 +880,37 @@ The following settings are available for Cloud Life Sciences:
 `google.zone`
 : The Google Cloud zone where jobs are executed. Multiple zones can be provided as a comma-separated list. Cannot be used with the `google.region` option. See the [Google Cloud documentation](https://cloud.google.com/compute/docs/regions-zones/) for a list of available regions and zones.
 
+`google.batch.allowedLocations`
+: :::{versionadded} 22.12.0-edge
+  :::
+: Define the set of allowed locations for VMs to be provisioned. See [Google documentation](https://cloud.google.com/batch/docs/reference/rest/v1/projects.locations.jobs#locationpolicy) for details (default: no restriction).
+
+`google.batch.bootDiskSize`
+: Set the size of the virtual machine boot disk, e.g `50.GB` (default: none).
+
+`google.batch.cpuPlatform`
+: Set the minimum CPU Platform, e.g. `'Intel Skylake'`. See [Specifying a minimum CPU Platform for VM instances](https://cloud.google.com/compute/docs/instances/specify-min-cpu-platform#specifications) (default: none).
+
+`google.batch.installGpuDrivers`
+: :::{versionadded} 23.08.0-edge
+  :::
+: When `true` automatically installs the appropriate GPU drivers to the VM when a GPU is requested (default: `false`). Only needed when using an instance template.
+
+`google.batch.network`
+: Set network name to attach the VM's network interface to. The value will be prefixed with `global/networks/` unless it contains a `/`, in which case it is assumed to be a fully specified network resource URL. If unspecified, the global default network is used.
+
+`google.batch.serviceAccountEmail`
+: Define the Google service account email to use for the pipeline execution. If not specified, the default Compute Engine service account for the project will be used.
+
+`google.batch.spot`
+: When `true` enables the usage of *spot* virtual machines or `false` otherwise (default: `false`).
+
+`google.batch.subnetwork`
+: Define the name of the subnetwork to attach the instance to must be specified here, when the specified network is configured for custom subnet creation. The value is prefixed with `regions/subnetworks/` unless it contains a `/`, in which case it is assumed to be a fully specified subnetwork resource URL.
+
+`google.batch.usePrivateAddress`
+: When `true` the VM will NOT be provided with a public IP address, and only contain an internal IP. If this option is enabled, the associated job can only load docker images from Google Container Registry, and the job executable cannot use external services other than Google APIs (default: `false`).
+
 `google.lifeSciences.bootDiskSize`
 : Set the size of the virtual machine boot disk e.g `50.GB` (default: none).
 
@@ -975,6 +1006,11 @@ The following settings are available:
   :::
 : If you trace the hostname, activate this option (default: `false`).
 
+`k8s.fuseDevicePlugin`
+: :::{versionadded} 24.01.0-edge
+  :::
+: The FUSE device plugin to be used when enabling Fusion in unprivileged mode (default: `['nextflow.io/fuse': 1]`).
+
 `k8s.httpConnectTimeout`
 : :::{versionadded} 22.10.0
   :::
@@ -1415,12 +1451,12 @@ The following settings are available:
 `singularity.ociAutoPull`
 : :::{versionadded} 23.12.0-edge
   :::
-: When enabled, OCI (and Docker) container images are pull and converted to a SIF image file format implicitly by the Singularity run command, instead of Nextflow. Requires Singulairty 3.11 or later (default: `false`).
+: When enabled, OCI (and Docker) container images are pull and converted to a SIF image file format implicitly by the Singularity run command, instead of Nextflow. Requires Singularity 3.11 or later (default: `false`).
 
 `singularity.ociMode`
 : :::{versionadded} 23.12.0-edge
   :::
-: Enable OCI-mode, that allows running native OCI compliant container image with Singularity using `crun` or `runc` as low-level runtime. Note: it requires Singulairty 4 or later. See `--oci` flag in the [Singularity documentation](https://docs.sylabs.io/guides/4.0/user-guide/oci_runtime.html#oci-mode) for more details and requirements (default: `false`).
+: Enable OCI-mode, that allows running native OCI compliant container image with Singularity using `crun` or `runc` as low-level runtime. Note: it requires Singularity 4 or later. See `--oci` flag in the [Singularity documentation](https://docs.sylabs.io/guides/4.0/user-guide/oci_runtime.html#oci-mode) for more details and requirements (default: `false`).
 
 `singularity.pullTimeout`
 : The amount of time the Singularity pull can last, exceeding which the process is terminated (default: `20 min`).

docs/executor.md

@@ -270,7 +270,7 @@ Nextflow manages each process as a separate job that is submitted to the cluster
 
 The pipeline must be launched from a node where the `hq` command is available, which is typically the cluster login node.
 
-To enable the HTCondor executor, set `process.executor = 'hyperqueue'` in the `nextflow.config` file.
+To enable the HyperQueue executor, set `process.executor = 'hq'` in the `nextflow.config` file.
 
 Resource requests and other job characteristics can be controlled via the following process directives:
 

docs/google.md

@@ -90,45 +90,78 @@ Read the {ref}`Google configuration<config-google>` section to learn more about
 
 ### Process definition
 
-Processes can be defined as usual and by default the `cpus` and `memory` directives are used to find the cheapest machine type available at current location that fits the requested resources. If `memory` is not specified, 1GB of memory is allocated per cpu.
+By default, the `cpus` and `memory` directives are used to find the cheapest machine type that is available at the current
+location and that fits the requested resources. If `memory` is not specified, 1 GB of memory is allocated per CPU.
 
-:::{versionadded} 23.02.0-edge
-The `machineType` directive can be a list of patterns separated by comma. The pattern can contain a `*` to match any number of characters and `?` to match any single character. Examples of valid patterns: `c2-*`, `m?-standard*`, `n*`.
-
-Alternatively it can also be used to define a specific predefined Google Compute Platform [machine type](https://cloud.google.com/compute/docs/machine-types) or a custom machine type.
-:::
-
-Examples:
+The `machineType` directive can be used to request a specific VM instance type. It can be any predefined Google Compute
+Platform [machine type](https://cloud.google.com/compute/docs/machine-types) or [custom machine type](https://cloud.google.com/compute/docs/instances/creating-instance-with-custom-machine-type).
 
 ```groovy
-process automatic_resources_task {
+process myTask {
     cpus 8
     memory '40 GB'
 
     """
-    <Your script here>
+    your_command --here
+    """
+}
+
+process anotherTask {
+    machineType 'n1-highmem-8'
+
+    """
+    your_command --here
     """
 }
+```
 
-process allowing_some_series {
+:::{versionadded} 23.02.0-edge
+:::
+
+The `machineType` directive can also be a comma-separated list of patterns. The pattern can contain a `*` to match any
+number of characters and `?` to match any single character. Examples of valid patterns: `c2-*`, `m?-standard*`, `n*`.
+
+```groovy
+process myTask {
     cpus 8
     memory '20 GB'
     machineType 'n2-*,c2-*,m3-*'
 
     """
-    <Your script here>
+    your_command --here
     """
 }
+```
 
-process predefined_resources_task {
-    machineType 'n1-highmem-8'
+:::{versionadded} 23.12.0-edge
+:::
+
+The `machineType` directive can also be an [Instance Template](https://cloud.google.com/compute/docs/instance-templates),
+specified as `template://<instance-template>`. For example:
+
+```groovy
+process myTask {
+    cpus 8
+    memory '20 GB'
+    machineType 'template://my-template'
 
     """
-    <Your script here>
+    your_command --here
     """
 }
 ```
 
+:::{note}
+Using an instance template will overwrite the `accelerator` and `disk` directives, as well as the following Google Batch
+config options: `cpuPlatform`, `preemptible`, and `spot`.
+:::
+
+To use an instance template with GPUs, you must also set the `google.batch.installGpuDrivers` config option to `true`.
+
+To use an instance template with Fusion, the instance template must include a `local-ssd` disk named `fusion` with 375 GB.
+See the [Google Batch documentation](https://cloud.google.com/compute/docs/disks/local-ssd) for more details about local SSDs.
+
+
 :::{versionadded} 23.06.0-edge
 :::
 

docs/kubernetes.md

@@ -120,6 +120,23 @@ Then the pipeline execution can be launched using the usual run command and spec
 nextflow run <YOUR PIPELINE> -work-dir s3://<YOUR-BUCKET>/scratch
 ```
 
+:::{note}
+When using Fusion, pods will run as *privileged* by default.
+:::
+
+To use Fusion with without the need for escalating privileges, it is required to install in the Kubernetes cluster the
+Nextflow [FUSE device plugin](https://github.com/nextflow-io/k8s-fuse-plugin) and add in your Nextflow configuration the following
+setting:
+
+```
+fusion {
+  privileged = false
+}
+```
+
+To use a custom FUSE device plugin, specify it via the setting `k8s.fuseDevicePlugin`. See
+the {ref}`Kubernetes configuration section<config-k8s>` for details.
+
 ### Running in a pod
 
 Nextflow can be executed directly from a pod running in a Kubernetes cluster. In these cases you will need to use the plain Nextflow `run` command and specify the `k8s` executor and the required persistent volume claim in the `nextflow.config` file as shown below:

docs/wave.md

@@ -94,6 +94,16 @@ wave.strategy = ['conda']
 
 The above setting instructs Wave to use the `conda` directive to provision the pipeline containers and ignore the `container` directive and any Dockerfile(s).
 
+:::{tip}
+Some configuration options in the `conda` scope are used when Wave is used to build Conda-based containers.
+For example, the Conda channels and their priority can be set with `conda.channels`:
+
+```groovy
+wave.strategy = ['conda']
+conda.channels = 'seqera,conda-forge,bioconda,defaults'
+```
+:::
+
 ### Build Spack based containers
 
 :::{warning}
@@ -195,7 +205,7 @@ The following configuration options are available:
 : When enabling the container freeze mode, Wave will provision an non-ephemeral container image
 that will be pushed to a container repository your choice. It requires the use of the `wave.build.repository` setting.
 It is also suggested to specify a custom cache repository via the setting `wave.build.cacheRepository`. Note: when using
-container freeze mode, the container repository authentication needs to be managed by the underlying infrastructure.    
+container freeze mode, the container repository authentication needs to be managed by the underlying infrastructure.
 
 `wave.build.repository`
 : The container repository where images built by Wave are uploaded (note: the corresponding credentials must be provided in your Nextflow Tower account).
@@ -230,20 +240,10 @@ container freeze mode, the container repository authentication needs to be manag
 `wave.strategy`
 : The strategy to be used when resolving ambiguous Wave container requirements (default: `'container,dockerfile,conda,spack'`).
 
-`wave.report.enabled` (preview)
-: :::{versionadded} 23.06.0-edge
-  :::
-: Enable the reporting of the Wave containers used during the pipeline execution (default: `false`).
-
-`wave.report.file` (preview)
-: :::{versionadded} 23.06.0-edge
-  :::
-: The name of the containers report file (default: `'containers-<timestamp>.config'`).
-
 `wave.retryPolicy.delay`
 : :::{versionadded} 22.06.0-edge
   :::
-: The initial delay when a failing HTTP request is retried (default: `150ms`). 
+: The initial delay when a failing HTTP request is retried (default: `150ms`).
 
 `wave.retryPolicy.maxDelay`
 : :::{versionadded} 22.06.0-edge