Merge branch 'master' into add/mult_revisions_apr_revision_path

Signed-off-by: Dr Marco Claudio De La Pierre <[email protected]>

VERSION

@@ -1 +1 @@
-24.04.2
+24.05.0-edge

changelog.txt

@@ -1,5 +1,33 @@
 NEXTFLOW CHANGE-LOG
 ===================
+24.05.0-edge - 17 Jun 2024
+- Add Managed Identity to Azure authentication documentation (#5062) [ab5b3eb2]
+- Add support for Azure managed identity (#4897) [21ca16e6]
+- Add support for EC-encrypted private keys for K8s executor [4dbb5193]
+- Add workflow.failOnIgnore config option (#4686) [ci fast] [172edc38]
+- Allow requester pays for S3 buckets (#5027) [0070c1b0]
+- Dump threads on abort condition [e3740977]
+- Enable build with Java 21 (#5030) [4c54db6a]
+- Fix Azure system-assigned managed identity [a639a17d]
+- Fix Java 22 version in launcher [29d6f1df]
+- Fix executions hangs on finalisation exception (#5070) [38434b82]
+- Fix support for Azure managed identity clientId [306814e7]
+- Fix support for s5cmd 2.2.2 (#5069) [7e78bd4d]
+- Fix the cleanup of local secret file (#5061) [e28537ca]
+- Fix typo in error message [f5c2355b]
+- Fix unique operator hangs when applied to value (#5053) [88b8ab0c]
+- Improve retry strategy for Google cloud errors when writing task helper files (#5037) [f8b324ab]
+- Remove GA4GH plugin (#5056) [0471b2e1]
+- Remove ending slashes for publish targets [891ed117]
+- Bump Gradle 8.8 final [ee54606c]
+- Bump aws-sdk 1.12.740 [acad2a1f]
+- Bump azure-compute-batch:1.0.0-beta.2 [c08dc49b]
+- Bump azure-storage-blob 12.26.1 [c76ff5e7]
+- Bump [email protected] [01922f13]
+- Bump [email protected] [c064d209]
+- Bump [email protected] [eeca7d67]
+- Bump [email protected] [d3f53c19]
+
 23.10.3 - 11 Jun 2024
 - Bump [email protected] [6960daab]
 - Bump [email protected] [fbd3c1b4]

docs/azure.md

@@ -405,31 +405,52 @@ The value of the setting must be the identifier of a subnet available in the vir
 Batch Authentication with Shared Keys does not allow to link external resources (like Virtual Networks) to the pool. Therefore, Active Directory Authentication must be used in conjunction with the `virtualNetwork` setting.
 :::
 
-## Microsoft Entra (formerly Active Directory Authentication)
+## Microsoft Entra
 
-:::{versionadded} 22.11.0-edge
-:::
+Using Microsoft Entra for role-based access control is more secure than using access keys and should be used wherever possible. You can authenticate to Azure Entra using a Managed Identity when running on resources within the Azure environment, or by authenticating as an Azure Service Principal when running on external resources.
 
-[Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) credentials can optionally be used instead of Shared Keys for Azure Batch and Storage accounts.
+### Required role assignments
 
-The Service Principal should have the at least the following role assignments:
+To access Azure resources, you must have the relevant role assignments (permissions). Access Azure Blob storage data (e.g. to retrieve data from a private Azure Storage account) you must have the following permissions:
 
-1. Contributor
-2. Storage Blob Data Reader
-3. Storage Blob Data Contributor
+1. Storage Blob Data Reader
+2. Storage Blob Data Contributor
 
-:::{note}
-To assign the necessary roles to the Service Principal, refer to the [official Azure documentation](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current).
+To run Nextflow on Azure Batch you must have the following permissions to create and destroy resources in the Azure Batch account:
+
+1. Batch Contributor
+
+To assign the necessary roles to a Managed Identity or Service Principal, refer to the [official Azure documentation](https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal?tabs=current).
+
+(azure-managed-identities)=
+
+### Managed identities
+
+:::{versionadded} 24.05.0-edge
 :::
 
-The credentials for Service Principal can be specified as follows:
+An Azure [Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) can be used to authenticate with Azure Resources without the use of access keys or credentials. When using a Managed Identity, an Azure resource is able to authenticate because of what _it is_, instead of using access keys. For example, if Nextflow is running on an [Azure Virtual Machine with a managed identity enabled](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-configure-managed-identities?pivots=qs-configure-portal-windows-vm) which had the relevant permissions, it would be able to run a Nextflow workflow on Azure Batch and use data from a private storage account with no additional credentials supplied. This is more secure and reliable than using Access Keys or a Service Principal which can be compromised. The limitation is they are only able to work from within the Azure account, i.e. you cannot use a managed identity from an external service.
+
+An Azure Managed identity comes in two forms, system-assigned and user-assigned. Both are functionally equivalent but have a slightly method
+
+#### System Assigned Managed Identity
+
+When running on an Azure Service such as an Azure Virtual Machine, you can enable system-assigned Managed Identity for that machine. This grants the machine an identity in Azure from which it receives the permissions and role assignments.
+
+1. First we must [enable system-assigned Managed Identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-configure-managed-identities?pivots=qs-configure-portal-windows-vm). Once you have done this the machine has an identity in Azure Entra.
+2. Next we must add the relevant role assignments to this Managed Identity. On the Azure Portal page for the virtual machine, select 'Identity' and then click 'Azure Role Assignments' to modify the existing role assignments. Note you must have `Microsoft.Authorization/roleAssignments/write` to perform this action.
+3. Make sure the identity has the following role assignments:
+    - Storage Blob Data Reader
+    - Storage Blob Data Contributor
+    - Batch Contributor
+4. Save the changes.
+5. Use the following Nextflow configuration to enable Nextflow to adopt the system-assigned identity while running on this machine:
 
 ```groovy
+process.executor = 'azurebatch'
 azure {
-    activeDirectory {
-        servicePrincipalId = '<YOUR SERVICE PRINCIPAL CLIENT ID>'
-        servicePrincipalSecret = '<YOUR SERVICE PRINCIPAL CLIENT SECRET>'
-        tenantId = '<YOUR TENANT ID>'
+    managedIdentity {
+        system = true
     }
 
     storage {
@@ -443,21 +464,52 @@ azure {
 }
 ```
 
-(azure-managed-identities)=
+#### User Assigned Managed Identity
 
-## Managed identities
+A system-assigned managed identity is essentially 'anonymous' and is tied to a single resource. By comparison, a user-assigned managed identity is created by the user and can be assigned to multiple resources, furthermore the lifecycle of a user-assigned managed identity is not tied to the resource. See [the Azure Documentation](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/managed-identity-best-practice-recommendations#choosing-system-or-user-assigned-managed-identities) for further details.
 
-:::{versionadded} 24.05.0-edge
+We can add a user-assigned identity to a resource in a similar manner to a system-assigned identity, but we must create it first.
+
+1. Create a Managed Identity as per [the Azure Documentation](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-manage-user-assigned-managed-identities?pivots=identity-mi-methods-azp).
+2. Assign the relevant role permissions to the Managed Identity as before.
+3. Assign the Managed Identity to the Azure Resource, this can be done at creation time or afterwards. [As an example, see the documentation on how to do this for a virtual machine](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/how-to-configure-managed-identities?pivots=qs-configure-portal-windows-vm#user-assigned-managed-identity).
+4. Retrieve the client ID for the Managed Identity. On the Azure Portal, this can be found on the 'Overview' or 'Properties' page as 'client ID'.
+5. Use the following configuration to enable Nextflow to adopt the user-assigned identity while running on the Azure Resource:
+
+```groovy
+process.executor = 'azurebatch'
+azure {
+    managedIdentity {
+        clientId = '<USER ASSIGNED MANAGED IDENTITY CLIENT ID>'
+    }
+
+    storage {
+        accountName = '<YOUR STORAGE ACCOUNT NAME>'
+    }
+
+    batch {
+        accountName = '<YOUR BATCH ACCOUNT NAME>'
+        location = '<YOUR BATCH ACCOUNT LOCATION>'
+    }
+}
+```
+
+(azure-service-principal)=
+
+### Service Principals
+
+:::{versionadded} 22.11.0-edge
 :::
 
-An Azure [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview) can be used to authenticate with Azure Blob storage without using secrets such as a storage account key.
+[Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) credentials can be used access to Azure Batch and Storage accounts. Similar to a Managed Identity, a Service Principal is an account which can have specific permissions and role based access. However, unlike with Managed Identities you must use a secret key to authenticate as a Service Principal. However, this means you can access authenticate as a Service Principal when operating outside of the Azure account.
 
-The managed identity can be specified as follows:
+The credentials for Service Principal can be specified as follows:
 
 ```groovy
 azure {
-    managedIdentity {
-        clientId = '<YOUR MANAGED IDENTITY>'
+    activeDirectory {
+        servicePrincipalId = '<YOUR SERVICE PRINCIPAL CLIENT ID>'
+        servicePrincipalSecret = '<YOUR SERVICE PRINCIPAL CLIENT SECRET>'
         tenantId = '<YOUR TENANT ID>'
     }
 
@@ -472,8 +524,6 @@ azure {
 }
 ```
 
-Alternatively, you can set `azure.managedIdentity.system = true` to authenticate using the system-assigned identity.
-
 ## Advanced configuration
 
 Read the {ref}`Azure configuration<config-azure>` section to learn more about advanced configuration options.

docs/config.md

@@ -1773,6 +1773,15 @@ The following settings are available:
 `wave.strategy`
 : The strategy to be used when resolving ambiguous Wave container requirements (default: `'container,dockerfile,conda,spack'`).
 
+### Scope `workflow`
+
+The `workflow` scope provides workflow execution options.
+
+`workflow.failOnIgnore`
+: :::{versionadded} 24.05.0-edge
+  :::
+: When `true`, the pipeline will exit with a non-zero exit code if any failed tasks are ignored using the `ignore` error strategy.
+
 (config-miscellaneous)=
 
 ### Miscellaneous

docs/metadata.md

@@ -58,6 +58,11 @@ The following table lists the properties that can be accessed on the `workflow`
 : *Available only in the `workflow.onComplete` and `workflow.onError` handlers*
 : Exit status of the task that caused the workflow execution to fail.
 
+`workflow.failOnIgnore`
+: :::{versionadded} 24.05.0-edge
+  :::
+: Whether the `workflow.failOnIgnore` config option was enabled.
+
 `workflow.fusion.enabled`
 : Whether Fusion is enabled.
 

docs/process.md

@@ -846,6 +846,7 @@ The following output qualifiers are available:
 - `env`: Emit the variable defined in the process environment with the specified name.
 - `stdout`: Emit the `stdout` of the executed process.
 - `tuple`: Emit multiple values.
+- `eval`: Emit the result of a script or command evaluated in the task execution context.
 
 ### Output type `val`
 
@@ -1652,16 +1653,19 @@ The `errorStrategy` directive allows you to define how an error condition is man
 The following error strategies are available:
 
 `terminate` (default)
-: Terminate the execution as soon as an error condition is reported. Pending jobs are killed.
+: When a task fails, terminate the pipeline immediately. Pending and running jobs are killed.
 
 `finish`
-: Initiate an orderly pipeline shutdown when an error condition is raised, waiting for the completion of any submitted jobs.
+: When a task fails, wait for pending and running tasks to finish and then terminate the pipeline.
 
 `ignore`
-: Ignore process execution errors.
+: Ignore all task failures and complete the pipeline execution successfully.
+: :::{versionadded} 24.05.0-edge
+  When the `workflow.failOnIgnore` config option is set to `true`, the pipeline will return a non-zero exit code if one or more failed tasks were ignored.
+  :::
 
 `retry`
-: Re-submit any process that returns an error condition.
+: When a task fails, retry it.
 
 When setting the `errorStrategy` directive to `ignore` the process doesn't stop on an error condition, it just reports a message notifying you of the error event.
 

docs/requirements.txt

@@ -2,4 +2,5 @@ myst-parser==0.18.1
 sphinx==5.3.0
 sphinx-rtd-theme==1.1.1
 sphinxcontrib-mermaid==0.9.2
-sphinxext-rediraffe==0.2.7
+sphinxext-rediraffe==0.2.7
+urllib3>=2.2.2 # not directly required, pinned by Snyk to avoid a vulnerability

modules/nextflow/src/main/groovy/nextflow/Session.groovy

@@ -223,6 +223,8 @@ class Session implements ISession {
 
     private Barrier monitorsBarrier = new Barrier()
 
+    private volatile boolean failOnIgnore
+
     private volatile boolean cancelled
 
     private volatile boolean aborted
@@ -824,7 +826,7 @@ class Session implements ISession {
 
     boolean isCancelled() { cancelled }
 
-    boolean isSuccess() { !aborted && !cancelled }
+    boolean isSuccess() { !aborted && !cancelled && !failOnIgnore }
 
     void processRegister(TaskProcessor process) {
         log.trace ">>> barrier register (process: ${process.name})"
@@ -864,6 +866,10 @@ class Session implements ISession {
         config.navigate('nextflow.enable.moduleBinaries', false) as boolean
     }
 
+    boolean failOnIgnore() {
+        config.navigate('workflow.failOnIgnore', false) as boolean
+    }
+
     @PackageScope VersionNumber getCurrentVersion() {
         new VersionNumber(BuildInfo.version)
     }
@@ -1047,6 +1053,12 @@ class Session implements ISession {
         final trace = handler.safeTraceRecord()
         cache.putTaskAsync(handler, trace)
 
+        // set the pipeline to return non-exit code if specified
+        if( handler.task.errorAction == ErrorStrategy.IGNORE && failOnIgnore() ) {
+            log.debug "Setting fail-on-ignore flag due to ignored task '${handler.task.lazyName()}'"
+            failOnIgnore = true
+        }
+
         // notify the event to the observers
         for( int i=0; i<observers.size(); i++ ) {
             final observer = observers.get(i)

modules/nextflow/src/main/groovy/nextflow/fusion/FusionConfig.groovy

@@ -34,8 +34,8 @@ import nextflow.util.MemoryUnit
 @CompileStatic
 class FusionConfig {
 
-    final static public String DEFAULT_FUSION_AMD64_URL = 'https://fusionfs.seqera.io/releases/v2.3-amd64.json'
-    final static public String DEFAULT_FUSION_ARM64_URL = 'https://fusionfs.seqera.io/releases/v2.3-arm64.json'
+    final static public String DEFAULT_FUSION_AMD64_URL = 'https://fusionfs.seqera.io/releases/v2.4-amd64.json'
+    final static public String DEFAULT_FUSION_ARM64_URL = 'https://fusionfs.seqera.io/releases/v2.4-arm64.json'
     final static public String DEFAULT_TAGS = "[.command.*|.exitcode|.fusion.*](nextflow.io/metadata=true),[*](nextflow.io/temporary=true)"
 
     final static public String FUSION_PATH = '/usr/bin/fusion'

modules/nextflow/src/main/groovy/nextflow/processor/ErrorStrategy.groovy

@@ -23,10 +23,10 @@ package nextflow.processor
  */
 enum ErrorStrategy {
 
-    TERMINATE(false),    // on error, terminate the pipeline execution killing all pending and running tasks
-    FINISH(false),       // on error, terminate the pipeline execution awaiting for previously submitted task to complete
-    IGNORE(true),       // on error, ignore it an go-on
-    RETRY(true);        // on error, retry
+    TERMINATE(false),       // on error, terminate the pipeline execution, killing all pending and running tasks
+    FINISH(false),          // on error, terminate the pipeline execution, waiting for pending and running tasks to complete
+    IGNORE(true),           // on error, ignore it and continue
+    RETRY(true);            // on error, retry
 
     final boolean soft
 

modules/nextflow/src/main/groovy/nextflow/processor/TaskPollingMonitor.groovy

@@ -663,14 +663,24 @@ class TaskPollingMonitor implements TaskMonitor {
 
             // finalize the task asynchronously
             if( enableAsyncFinalizer ) {
-                finalizerPool.submit( ()-> finalizeTask(handler) )
+                finalizerPool.submit( ()-> safeFinalizeTask(handler) )
             }
             else {
                 finalizeTask(handler)
             }
         }
     }
 
+    protected void safeFinalizeTask(TaskHandler handler) {
+        try {
+            finalizeTask(handler)
+        }
+        catch (Throwable t) {
+            log.error "Unexpected error while finalizing task '${handler.task.name}' - cause: ${t.message}"
+            session.abort(t)
+        }
+    }
+
     protected void finalizeTask( TaskHandler handler ) {
         // finalize the task execution
         final fault = handler.task.processor.finalizeTask(handler.task)

modules/nextflow/src/main/groovy/nextflow/processor/TaskProcessor.groovy

@@ -1072,8 +1072,10 @@ class TaskProcessor {
                 errorStrategy = checkErrorStrategy(task, error, taskErrCount, procErrCount, submitRetries)
                 if( errorStrategy.soft ) {
                     def msg = "[$task.hashLog] NOTE: ${submitTimeout ? submitErrMsg : error.message}"
-                    if( errorStrategy == IGNORE ) msg += " -- Error is ignored"
-                    else if( errorStrategy == RETRY ) msg += " -- Execution is retried (${submitTimeout ? submitRetries : taskErrCount})"
+                    if( errorStrategy == IGNORE )
+                        msg += " -- Error is ignored"
+                    else if( errorStrategy == RETRY )
+                        msg += " -- Execution is retried (${submitTimeout ? submitRetries : taskErrCount})"
                     log.info msg
                     task.failed = true
                     task.errorAction = errorStrategy

modules/nextflow/src/main/groovy/nextflow/script/WorkflowMetadata.groovy

@@ -227,6 +227,11 @@ class WorkflowMetadata {
      */
     Manifest manifest
 
+    /**
+     * whenever it should terminate with a failure when one or more task execution failed in an error strategy
+     */
+    boolean failOnIgnore
+
     private Session session
 
     final private List<Closure> onCompleteActions = []
@@ -268,6 +273,7 @@ class WorkflowMetadata {
         this.manifest = session.getManifest()
         this.wave = new WaveMetadata(session)
         this.fusion = new FusionMetadata(session)
+        this.failOnIgnore = session.failOnIgnore()
 
         // check if there's a onComplete action in the config file
         registerConfigAction(session.config.workflow as Map)