Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporate DSL2 features into docs #3793

Merged
merged 13 commits into from
Oct 15, 2023
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Merge branch 'master' into ben-dsl1-migration-guide
Signed-off-by: Ben Sherman <bentshermann@gmail.com>
bentsherman committed May 15, 2023
commit a415a517aa93fd76f0ce103ffeecdb70eef632e2
4 changes: 2 additions & 2 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ jobs:
strategy:
fail-fast: false
matrix:
java_version: [11, 19]
java_version: [11, 17, 19]

steps:
- name: Environment
Expand Down Expand Up @@ -92,7 +92,7 @@ jobs:
any_changed: ${{ steps.changed-files.outputs.any_changed }}

test:
if: ${{ !contains(github.event.head_commit.message, '[ci fast]') && needs.build.outputs.any_changed == 'true' }}
if: ${{ !contains(github.event.head_commit.message, '[ci fast]') }}
needs: build
runs-on: ubuntu-latest
strategy:
Expand Down
5 changes: 5 additions & 0 deletions NOTICE
Original file line number Diff line number Diff line change
Expand Up @@ -140,5 +140,10 @@ This software includes source code and libraries developed by:
Copyright 2008, http://www.snakeyaml.org
Licensed under Apache License, Version 2.0

Spack
https://github.com/spack/spack
Copyright 2013-2023 Lawrence Livermore National Security, LLC and other Spack Project Developers. See the top-level COPYRIGHT file for details.
Licensed under Apache License, Version 2.0

Nextflow is a registered trademark of Seqera Labs, Spain.
See the trademark policy for details https://github.com/nextflow-io/trademark
2 changes: 1 addition & 1 deletion VERSION
Original file line number Diff line number Diff line change
@@ -1 +1 @@
23.04.0
23.05.0-edge
8 changes: 5 additions & 3 deletions build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ allprojects {

java {
toolchain {
languageVersion = JavaLanguageVersion.of(17)
languageVersion = JavaLanguageVersion.of(19)
}
}

Expand Down Expand Up @@ -100,8 +100,8 @@ allprojects {

// Documentation required libraries
groovyDoc 'org.fusesource.jansi:jansi:1.11'
groovyDoc "org.codehaus.groovy:groovy-groovydoc:3.0.16"
groovyDoc "org.codehaus.groovy:groovy-ant:3.0.16"
groovyDoc "org.codehaus.groovy:groovy-groovydoc:3.0.17"
groovyDoc "org.codehaus.groovy:groovy-ant:3.0.17"
}

test {
Expand Down Expand Up @@ -139,6 +139,7 @@ allprojects {
if (JavaVersion.current() >= JavaVersion.VERSION_1_9) {
tasks.withType(Test) {
jvmArgs ([
'--enable-preview',
'--add-opens=java.base/java.lang=ALL-UNNAMED',
'--add-opens=java.base/java.io=ALL-UNNAMED',
'--add-opens=java.base/java.nio=ALL-UNNAMED',
Expand All @@ -154,6 +155,7 @@ allprojects {
'--add-opens=java.base/sun.net.www.protocol.ftp=ALL-UNNAMED',
'--add-opens=java.base/sun.net.www.protocol.file=ALL-UNNAMED',
'--add-opens=java.base/jdk.internal.misc=ALL-UNNAMED',
'--add-opens=java.base/jdk.internal.vm=ALL-UNNAMED',
])
}
}
Expand Down
66 changes: 64 additions & 2 deletions changelog.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,67 @@
NEXTFLOW CHANGE-LOG
===================
23.05.0-edge - 15 May 2023
- Add support for custom custom root directory to resolve relative paths (#3942) [f06bb1f7]
- Add `arch` directive support for Spack (#3786) [62dfc482]
- Add clone deep option to run and pull commands [b44b6453]
- Add fusion.exportStorageCredentials option [acb6aedf]
- Add groovy-yaml to core runtime [003e1f70]
- Add missing test env variables to CI build [a0e501bf]
- Add splitJson Operator (#3830) [4f58f695]
- Add support for Java 20 [3788a689]
- Add support for `time` directive in Azure Batch (#3869) [5c11a0d4]
- Add support for the Spack recipes to Wave build (#3636) [b03cbe70]
- Add Preview support for virtual threads (#3871) [5c429046]
- Convert docs to Markdown (#3196) [3cad402f]
- Deprecate DSL1 syntax for tap operator (#3004) [2be0edfe]
- Enable static compilation to missing classes (#3906) [c3c4b2f0]
- Fix -with-conda option in the docs (#3867) [9b350ba8]
- Fix AWS SSE env propagation to Fusion [e24608c3]
- Fix Apptainer rendering by removing superflous space [36607ed1]
- Fix Azure jobs correctly deleted after completion (#3927) [b173a983]
- Fix Azure pool creation when using scaling formula (#3868) [79984a87]
- Fix DSL2 support in nextflow console (#3864) [0253f8d7]
- Fix K8s handling PodUnschedulableException (#3938) [5b756c63]
- Fix `interval` channel factory (#3848) [d001e4c7]
- Fix invalid warning on func import (#3807) [bb836960]
- Fix minor typo in nextflow subcommand (#3853) [dd3be3d1]
- Fix misleading error message on NoSuchFileException [f3dceba6]
- Fix missing SAS token fusion env for Azure [43015029]
- Fix problem args were not correctly represented in debug yaml (#3904) [eeb4b5c8]
- Fix string comparison in S3 client (#3875) [9344d294]
- Fix use of remote dir with local exec statement [850bc3d9]
- Fix Quote Singularity/Apptainer env values [586417ac]
- Improve handling default out param [91471970]
- Improve plugin test detection [29794bf4]
- Improve plugin updater err handling [f3d72f22]
- Improve task out redirect remove the use of mkfifo (#3863) [efedec74]
- Improve Use Docker stop instead of docker kill [32e6fd85]
- Improve Use separate script for long stage/unstage commands (#3851) [4343e333]
- Increase Azure default maxRetries to 10 [a017139f]
- Keep plugins-info content sorted [d5552ee6]
- Minor change on Docker-spack templates [e0e24bda]
- Refactor the AWS configuration (#3498) [a74e42d9]
- Remove `Channel.from` in favor of `Channel.of` from tests and docs (#3670) [aa8b72fa]
- Rename AmazonS3Client to S3Client [cc59596a]
- Rename com.upplication.s3fs package to nextflow.cloud.aws.nio [a2f3bb24]
- Run build CI tests on push to branches starting with `test*` or `dev*` (#3852) [7032e17c]
- Security fixes [973b7bea]
- Support enums in cache helper (#3901) [e4b977a4]
- Bump slf4j 2.0.7 and logback 1.4.6 [d7eae86e]
- Bump JGit 6.5 and use depth 1 when cloning repos [eda0845c]
- Bump aws-java-sdk-s3:1.12.429 [465171f5]
- Bump fusion 2.2 [f1ebe29a]
- Bump groovy 3.0.17 [cfe4ba56]
- Bump gson:2.10.1 [be555683]
- Bump micromamba:1.4.2 [334df1e0]
- Bump nf-wave@0.9.0 [7d6d0108]
- Bump nf-tower@1.5.13 [b06ffd0c]
- Bump nf-google@1.7.4 [2e0a15cd]
- Bump nf-ga4gh@1.0.6 [0c8fd7df]
- Bump nf-console@1.0.6 [a733b818]
- Bump nf-codecommit@0.1.5 [12edf0e6]
- Bump nf-amazon@2.0.0 [69b6edde]
- Bump nf-azure@1.1.0 [d05d51b4]

23.04.1 - 15 Apr 2023
- Add support for Java 20 [f6933db7]
Expand All @@ -8,7 +70,7 @@ NEXTFLOW CHANGE-LOG
- Fix interval channel factory (#3848) [82ccf6ad]
- Fix invalid warning on func import (#3807) [4678ffe3]
- Improve plugin test detection [57cfb0d9]
- Improve plugin updater err handling [ci skip] [5627c69a]
- Improve plugin updater err handling [5627c69a]
- Security fixes [83e8fd6a]
- Bump nf-wave@0.8.2 [d8cb763c]
- Bump nf-tower@1.5.12 [b2354380]
Expand All @@ -24,7 +86,7 @@ NEXTFLOW CHANGE-LOG
- Security fixes [fc2627bb]
- Add support for testing custom plugin uri [718d4562]
- Improve plugin updater err handling [84042082]
- Improve plugin test detection [ci fast] [e6c60a6c]
- Improve plugin test detection [e6c60a6c]
- Bump nf-google@1.4.6 [26b01b83]
- Bump nf-wave@0.5.5 [2a886fe0]
- Bump nf-ga4gh@1.0.5 [b8769f1a]
Expand Down
2 changes: 1 addition & 1 deletion docker/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM amazoncorretto:17.0.6
FROM amazoncorretto:17.0.7
RUN yum install -y procps-ng shadow-utils

ENV NXF_HOME=/.nextflow
Expand Down
37 changes: 37 additions & 0 deletions docs/_static/theme.css
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,40 @@
margin-top: 0.5rem;
width: 60%;
}

/* Version change admonitions */
div.deprecated, div.versionadded, div.versionchanged {
border-left: 0.2rem solid;
border-radius: 0.25rem;
box-shadow: 0 0.2rem 0.5rem var(--pst-color-shadow),0 0 0.0625rem var(--pst-color-shadow)!important;
margin: 1rem auto;
overflow: hidden;
padding: 0.2rem 0.6rem;
page-break-inside: avoid;
position: relative;
transition: color .25s,background-color .25s,border-color .25s;
vertical-align: middle;
}
div.deprecated>p, div.versionadded>p, div.versionchanged>p {
margin-top: 0.5rem;
margin-bottom: 0.5rem;
}
div.deprecated>p:first-child::before, div.versionadded>p:first-child::before, div.versionchanged>p:first-child::before {
font-family: FontAwesome;
display: inline-block;
font-style: normal;
font-weight: 400;
line-height: 1;
text-decoration: inherit;
margin-right: 0.5rem;
content: "";
-webkit-font-smoothing: antialiased;
padding: 0.4rem 0.6rem;
margin: -0.2rem 0rem -0.2rem -0.6rem;
}
div.versionadded { border-color: #1abc9c; background-color: #dbfaf4; }
div.versionadded p:first-child::before { color: #1abc9c; }
div.versionchanged { border-color: #f0b37e; background-color: #ffedcc; }
div.versionchanged p:first-child::before { color: #f0b37e; }
div.deprecated { border-color: #f29f97; background-color: #fdf3f2; }
div.deprecated p:first-child::before { color: #f29f97; }
24 changes: 23 additions & 1 deletion docs/amazons3.md
Original file line number Diff line number Diff line change
Expand Up @@ -65,11 +65,33 @@ Learn more about [Using IAM Roles to Delegate Permissions to Applications that R
To use an AWS China region, make sure to specify the corresponding AWS API S3 endpoint in the Nextflow configuration file as shown below:

```groovy
aws.client.endpoint = "https://s3.cn-north-1.amazonaws.com.cn"
aws {
client {
endpoint = "https://s3.cn-north-1.amazonaws.com.cn"
}
}
```

Read more about AWS API endpoints in the [AWS documentation](https://docs.aws.amazon.com/general/latest/gr/s3.html)

## S3-compatible storage

To use S3-compatible object storage such as [Ceph](https://ceph.io) or [Minio](https://min.io) specify the endpoint of
your storage provider and enable the [S3 path style access](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#path-style-access)
in your Nextflow configuration as shown below:


```groovy
aws {
accessKey = '<Your access key>'
secretKey = '<Your secret key>'
client {
endpoint = '<Your storage endpoint URL>'
s3PathStyleAccess = true
}
}
```

## Advanced configuration

Read {ref}`AWS configuration<config-aws>` section to learn more about advanced S3 client configuration options.
20 changes: 11 additions & 9 deletions docs/aws.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,10 @@ Different queues bound to the same or different Compute Environments can be conf

## Container Options

As of version `21.12.1-edge`, the {ref}`process-containerOptions` directive can be used to control the properties of the container execution associated with each Batch job.
:::{versionadded} 21.12.1-edge
:::

The {ref}`process-containerOptions` directive can be used to control the properties of the container execution associated with each Batch job.

The following container options are currently supported:

Expand Down Expand Up @@ -306,12 +309,12 @@ aws.batch.cliPath = '/home/ec2-user/miniconda/bin/aws'

Replace the path above with the one matching the location where the `aws` tool is installed in your AMI.

:::{warning}
The grandparent directory of the `aws` tool will be mounted into the container at the same path as the host, e.g. `/home/ec2-user/miniconda`, which will shadow existing files in the container. Make sure you use a path that is not already present in the container.
:::{versionchanged} 19.07.0
The `executor.awscli` config option was replaced by `aws.batch.cliPath`.
:::

:::{note}
In versions of Nextflow prior to 19.07.x, the `executor.awscli` config option should be used instead of `aws.batch.cliPath`.
:::{warning}
The grandparent directory of the `aws` tool will be mounted into the container at the same path as the host, e.g. `/home/ec2-user/miniconda`, which will shadow existing files in the container. Make sure you use a path that is not already present in the container.
:::

### Docker installation
Expand Down Expand Up @@ -416,6 +419,9 @@ With the above configuration, processes with the `bigTask` {ref}`process-label`

### Volume mounts

:::{versionadded} 19.07.0
:::

User provided container volume mounts can be provided as shown below:

```groovy
Expand All @@ -440,10 +446,6 @@ aws {

The above snippet defines two volume mounts for the jobs executed in your pipeline. The first volume mounts the host path `/tmp` to the same path in the container, with the *read-write* access mode. The second volume mounts the host path `/host/path` to `/mnt/path` in the container, with the *read-only* access mode.

:::{note}
This feature requires Nextflow version 19.07.x or later.
:::

### Troubleshooting

**Problem**: The Pipeline execution terminates with an AWS error message similar to the one shown below:
Expand Down
19 changes: 13 additions & 6 deletions docs/azure.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,8 @@

# Azure Cloud

## Requirements

The support for Azure Cloud requires Nextflow version `21.04.0` or later.
:::{versionadded} 21.04.0
:::

(azure-blobstorage)=

Expand Down Expand Up @@ -37,7 +36,9 @@ Once the Blob Storage credentials are set, you can access the files in the blob

## Azure File Shares

As of version `nf-azure@0.11.0`, Nextflow has built-in support also for [Azure Files](https://azure.microsoft.com/en-us/services/storage/files/). Files available in the serverless Azure File shares can be mounted concurrently on the nodes of a pool executing the pipeline. These files become immediately available in the file system and can be referred as local files within the processes. This is especially useful when a task needs to access large amounts of data (such as genome indexes) during its execution. An arbitrary number of File shares can be mounted on each pool node.
*New in `nf-azure` version `0.11.0`*

Nextflow has built-in support also for [Azure Files](https://azure.microsoft.com/en-us/services/storage/files/). Files available in the serverless Azure File shares can be mounted concurrently on the nodes of a pool executing the pipeline. These files become immediately available in the file system and can be referred as local files within the processes. This is especially useful when a task needs to access large amounts of data (such as genome indexes) during its execution. An arbitrary number of File shares can be mounted on each pool node.

The Azure File share must exist in the storage account configured for Blob Storage. The name of the source Azure File share and mount path (the destination path where the files are mounted) must be provided. Additional mount options (see the Azure Files documentation) can be set as well for further customisation of the mounting process.

Expand Down Expand Up @@ -314,7 +315,10 @@ See the {ref}`Azure configuration <config-azure>` section and the [Azure Batch n

### Private container registry

As of version `21.05.0-edge`, a private container registry for Docker images can be specified as follows:
:::{versionadded} 21.05.0-edge
:::

A private container registry for Docker images can be specified as follows:

```groovy
azure {
Expand All @@ -334,7 +338,10 @@ When using containers hosted in a private registry, the registry name must also

## Active Directory Authentication

As of version ``22.11.0-edge``, [Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) credentials can optionally be used instead of Shared Keys for Azure Batch and Storage accounts.
:::{versionadded} 22.11.0-edge
:::

[Service Principal](https://learn.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal) credentials can optionally be used instead of Shared Keys for Azure Batch and Storage accounts.

The Service Principal should have the at least the following role assignments:

Expand Down
17 changes: 7 additions & 10 deletions docs/channel.md
Original file line number Diff line number Diff line change
Expand Up @@ -64,8 +64,8 @@ See also: {ref}`process-multiple-input-channels`.

Channels may be created explicitly using the following channel factory methods.

:::{note}
As of version 20.07.0, `channel` has been introduced as an alias of `Channel`, therefore factory methods can be specified either as `channel.of()` or `Channel.of()`, and so on.
:::{versionadded} 20.07.0
`channel` was introduced as an alias of `Channel`, allowing factory methods to be specified as `channel.of()` or `Channel.of()`, and so on.
:::

(channel-empty)=
Expand All @@ -80,8 +80,8 @@ See also: {ref}`operator-ifempty`.

### from

:::{warning}
This method is deprecated. Use [of](#of) or [fromList](#fromlist) instead.
:::{deprecated} 19.09.0-edge
Use [of](#of) or [fromList](#fromlist) instead.
:::

The `from` method allows you to create a channel emitting any sequence of values that are specified as the method argument, for example:
Expand Down Expand Up @@ -128,8 +128,7 @@ Channel.from( [1, 2], [5,6], [7,9] )

### fromList

:::{note}
This feature requires Nextflow version 19.10.0 or later.
:::{versionadded} 19.10.0
:::

The `fromList` method allows you to create a channel emitting the values provided as a list of elements, for example:
Expand Down Expand Up @@ -311,8 +310,7 @@ Available options:

### fromSRA

:::{note}
This feature requires Nextflow version 19.04.0 or later.
:::{versionadded} 19.04.0
:::

The `fromSRA` method queries the [NCBI SRA](https://www.ncbi.nlm.nih.gov/sra) database and returns a channel emitting the FASTQ files matching the specified criteria i.e project or accession number(s). For example:
Expand Down Expand Up @@ -386,8 +384,7 @@ Available options:

### of

:::{note}
This feature requires Nextflow version 19.10.0 of later.
:::{versionadded} 19.10.0
:::

The `of` method allows you to create a channel that emits the arguments provided to it, for example:
Expand Down
Loading
You are viewing a condensed version of this merge commit. You can view the full changes here.