Skip to content

Commit

Permalink
Merge branch 'dev' of github.com:nf-core/tools into regex
Browse files Browse the repository at this point in the history
  • Loading branch information
ErikDanielsson committed Jun 28, 2022
2 parents f1c1698 + 8877e4a commit 4366c78
Show file tree
Hide file tree
Showing 44 changed files with 1,302 additions and 942 deletions.
18 changes: 18 additions & 0 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,24 @@ Or [version control integration](https://pycqa.github.io/isort/docs/configuratio
There is an automated CI check that runs when you open a pull-request to nf-core/tools that will fail if
any code does not adhere to isort formatting.

### pre-commit hooks

This repository comes with [pre-commit](https://pre-commit.com/) hooks for black, isort and Prettier. pre-commit automatically runs checks before a commit is committed into the git history. If all checks pass, the commit is made, if files are changed by the pre-commit hooks, the user is informed and has to stage the changes and attempt the commit again.

You can use the pre-commit hooks if you like, but you don't have to. The CI on Github will run the same checks as the tools installed with pre-commit. If the pre-commit checks pass, then the same checks in the CI will pass, too.

You can install the pre-commit hooks into the development environment by running the following command in the root directory of the repository.

```bash
pre-commit install --install-hooks
```

You can also run all pre-commit hooks without making a commit:

```bash
pre-commit run --all
```

## API Documentation

We aim to write function docstrings according to the [Google Python style-guide](https://github.com/google/styleguide/blob/gh-pages/pyguide.md#38-comments-and-docstrings). These are used to automatically generate package documentation on the nf-core website using Sphinx.
Expand Down
23 changes: 6 additions & 17 deletions .github/workflows/create-lint-wf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ on: [push, pull_request]

env:
NXF_ANSI_LOG: false
CAPSULE_LOG: none

jobs:
MakeTestWorkflow:
Expand All @@ -12,14 +11,9 @@ jobs:
NXF_ANSI_LOG: false
strategy:
matrix:
# Nextflow versions
include:
# Test pipeline minimum Nextflow version
- NXF_VER: "21.10.3"
NXF_EDGE: ""
# Test latest edge release of Nextflow
- NXF_VER: ""
NXF_EDGE: "1"
NXF_VER:
- "21.10.3"
- "latest-everything"
steps:
# Get the repo code
- uses: actions/checkout@v2
Expand All @@ -38,14 +32,9 @@ jobs:
# Set up Nextflow
- name: Install Nextflow
env:
NXF_VER: ${{ matrix.NXF_VER }}
# Uncomment only if the edge release is more recent than the latest stable release
# See https://github.com/nextflow-io/nextflow/issues/2467
# NXF_EDGE: ${{ matrix.NXF_EDGE }}
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
uses: nf-core/setup-nextflow@v1
with:
version: ${{ matrix.NXF_VER }}

# Install the Prettier linting tools
- uses: actions/setup-node@v2
Expand Down
23 changes: 6 additions & 17 deletions .github/workflows/create-test-wf.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@ on: [push, pull_request]

env:
NXF_ANSI_LOG: false
CAPSULE_LOG: none

jobs:
RunTestWorkflow:
Expand All @@ -12,14 +11,9 @@ jobs:
NXF_ANSI_LOG: false
strategy:
matrix:
# Nextflow versions
include:
# Test pipeline minimum Nextflow version
- NXF_VER: "21.10.3"
NXF_EDGE: ""
# Test latest edge release of Nextflow
- NXF_VER: ""
NXF_EDGE: "1"
NXF_VER:
- "21.10.3"
- "latest-everything"
steps:
- uses: actions/checkout@v2
name: Check out source-code repository
Expand All @@ -35,14 +29,9 @@ jobs:
pip install .
- name: Install Nextflow
env:
NXF_VER: ${{ matrix.NXF_VER }}
# Uncomment only if the edge release is more recent than the latest stable release
# See https://github.com/nextflow-io/nextflow/issues/2467
# NXF_EDGE: ${{ matrix.NXF_EDGE }}
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
uses: nf-core/setup-nextflow@v1
with:
version: ${{ matrix.NXF_VER }}

- name: Run nf-core/tools
run: |
Expand Down
11 changes: 3 additions & 8 deletions .github/workflows/pytest.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,6 @@ on:
push:
pull_request:

# Uncomment if we need an edge release of Nextflow again
# env: NXF_EDGE: 1

jobs:
pytest:
runs-on: ubuntu-latest
Expand All @@ -30,11 +27,9 @@ jobs:
pip install -e .
- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
wget -qO- get.nextflow.io | bash
sudo mv nextflow /usr/local/bin/
uses: nf-core/setup-nextflow@v1
with:
version: "latest-everything"

- name: Test with pytest
run: python3 -m pytest tests/ --color=yes --cov-report=xml --cov-config=.github/.coveragerc --cov=nf_core
Expand Down
13 changes: 3 additions & 10 deletions .github/workflows/sync.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,6 @@ on:
description: Only run on nf-core/testpipeline?
required: true

# Uncomment if we need an edge release of Nextflow again
# env: NXF_EDGE: 1

jobs:
get-pipelines:
runs-on: ubuntu-latest
Expand Down Expand Up @@ -57,13 +54,9 @@ jobs:
pip install .
- name: Install Nextflow
env:
CAPSULE_LOG: none
run: |
mkdir /tmp/nextflow
cd /tmp/nextflow
wget -qO- get.nextflow.io | bash
sudo ln -s /tmp/nextflow/nextflow /usr/local/bin/nextflow
uses: nf-core/setup-nextflow@v1
with:
version: "latest-everything"

- name: Run synchronisation
if: github.repository == 'nf-core/tools'
Expand Down
19 changes: 19 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
repos:
- repo: https://github.com/psf/black
rev: 22.3.0
hooks:
- id: black
language_version: python3.10
- repo: https://github.com/pycqa/isort
rev: 5.10.1
hooks:
- id: isort
- repo: https://github.com/pre-commit/mirrors-prettier
rev: "v2.6.2"
hooks:
- id: prettier
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: "v4.3.0"
hooks:
- id: name-tests-test
args: [--pytest-test-first]
26 changes: 17 additions & 9 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,33 @@

### Template

- Fix bug in pipeline readme logo URL ([#1589](https://github.com/nf-core/tools/issues/1589))
- Fix bug in pipeline readme logo URL ([#1590](https://github.com/nf-core/tools/pull/1590))
- Switch CI to use [setup-nextflow](https://github.com/nf-core/setup-nextflow) action to install Nextflow ([#1650](https://github.com/nf-core/tools/pull/1650))

### Linting

- Check that the `.prettierignore` file exists and that starts with the same content.
- Update `readme.py` nf version badge validation regexp to accept any signs before version number ([#1613](https://github.com/nf-core/tools/issues/1613))
- Add isort configuration and GitHub workflow ([#1538](https://github.com/nf-core/tools/pull/1538))
- Use black also to format python files in workflows ([#1563](https://github.com/nf-core/tools/pull/1563))

### General

- Add `--fail-warned` flag to `nf-core lint` to make warnings fail [#1593](https://github.com/nf-core/tools/pull/1593)
- Add `--fail-warned` flag to pipeline linting workflow [#1593](https://github.com/nf-core/tools/pull/1593)
- Update `readme` to drop `--key` option from `nf-core modules list` and add the new pattern syntax
- Add `--fail-warned` flag to `nf-core lint` to make warnings fail ([#1593](https://github.com/nf-core/tools/pull/1593))
- Add `--fail-warned` flag to pipeline linting workflow ([#1593](https://github.com/nf-core/tools/pull/1593))
- Updated the package requirements to prevent defunct installations of nf-core ([#1620](https://github.com/nf-core/tools/pull/1620))
- Fix and improve broken test for Singularity container download ([#1622](https://github.com/nf-core/tools/pull/1622))
- Use [`$XDG_CACHE_HOME`](https://specifications.freedesktop.org/basedir-spec/basedir-spec-latest.html) or `~/.cache` instead of `$XDG_CONFIG_HOME` or `~/config/` as base directory for API cache
- Switch CI to use [setup-nextflow](https://github.com/nf-core/setup-nextflow) action to install Nextflow ([#1650](https://github.com/nf-core/tools/pull/1650))

### Modules

- Fix a bug in the regex extracting the version from biocontainers URLs [#1598](https://github.com/nf-core/tools/pull/1598)

### Linting

- Add isort configuration and GitHub workflow ([#1538](https://github.com/nf-core/tools/pull/1538))
- Add `--fix-version` flag to `nf-core modules lint` command to update modules to the latest version ([#1588](https://github.com/nf-core/tools/pull/1588))
- Fix a bug in the regex extracting the version from biocontainers URLs ([#1598](https://github.com/nf-core/tools/pull/1598))
- Update how we interface with git remotes. ([#1626](https://github.com/nf-core/tools/issues/1626))
- Add prompt for module name to `nf-core modules info` ([#1644](https://github.com/nf-core/tools/issues/1644))
- Update docs with example of custom git remote ([#1645](https://github.com/nf-core/tools/issues/1645))

## [v2.4.1 - Cobolt Koala Patch](https://github.com/nf-core/tools/releases/tag/2.4) - [2022-05-16]

Expand Down Expand Up @@ -60,7 +68,7 @@
- Add a new command `nf-core modules test` which runs pytests locally.
- Print include statement to terminal when `modules install` ([#1520](https://github.com/nf-core/tools/pull/1520))
- Allow follow links when generating `test.yml` file with `nf-core modules create-test-yml` ([1570](https://github.com/nf-core/tools/pull/1570))
- Escaped test run output before logging it, to avoid a rich ` MarkupError`
- Escaped test run output before logging it, to avoid a rich `MarkupError`

### Linting

Expand Down
46 changes: 17 additions & 29 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

[![Python tests](https://github.com/nf-core/tools/workflows/Python%20tests/badge.svg?branch=master&event=push)](https://github.com/nf-core/tools/actions?query=workflow%3A%22Python+tests%22+branch%3Amaster)
[![codecov](https://codecov.io/gh/nf-core/tools/branch/master/graph/badge.svg)](https://codecov.io/gh/nf-core/tools)
[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
[![code style: prettier](https://img.shields.io/badge/code%20style-prettier-ff69b4.svg)](https://github.com/prettier/prettier)
[![Imports: isort](https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&labelColor=ef8336)](https://pycqa.github.io/isort/)

[![install with Bioconda](https://img.shields.io/badge/install%20with-bioconda-brightgreen.svg)](https://bioconda.github.io/recipes/nf-core/README.html)
Expand Down Expand Up @@ -910,42 +911,29 @@ The nf-core DSL2 modules repository is at <https://github.com/nf-core/modules>

The modules supercommand comes with two flags for specifying a custom remote:

- `--github-repository <github repo>`: Specify the repository from which the modules should be fetched. Defaults to `nf-core/modules`.
- `--branch <branch name>`: Specify the branch from which the modules shoudl be fetched. Defaults to `master`.
- `--git-remote <git remote url>`: Specify the repository from which the modules should be fetched as a git URL. Defaults to the github repository of `nf-core/modules`.
- `--branch <branch name>`: Specify the branch from which the modules should be fetched. Defaults to the default branch of your repository.

Note that a custom remote must follow a similar directory structure to that of `nf-core/moduleś` for the `nf-core modules` commands to work properly.

### Private remote modules
For example, if you want to install the `fastqc` module from the repository `nf-core/modules-test` hosted at `gitlab.com`, you can use the following command:

In order to get access to your private modules repo, you need to create
the `~/.config/gh/hosts.yml` file, which is the same file required by
[GitHub CLI](https://cli.github.com/) to deal with private repositories.
Such file is structured as follow:

```conf
github.com:
oauth_token: <your github access token>
user: <your github user>
git_protocol: <ssh or https are valid choices>
```terminal
nf-core modules --git-remote [email protected]:nf-core/modules-test.git install fastqc
```

The easiest way to create this configuration file is through _GitHub CLI_: follow
its [installation instructions](https://cli.github.com/manual/installation)
and then call:
Note that a custom remote must follow a similar directory structure to that of `nf-core/moduleś` for the `nf-core modules` commands to work properly.

The modules commands will during initalisation try to pull changes from the remote repositories. If you want to disable this, for example
due to performance reason or if you want to run the commands offline, you can use the flag `--no-pull`. Note however that the commands will
still need to clone repositories that have previously not been used.

```bash
gh auth login
```
### Private remote repositories

After that, you will be able to list and install your private modules without
providing your github credentials through command line, by using `--github-repository`
and `--branch` options properly.
See the documentation on [gh auth login](https://cli.github.com/manual/gh_auth_login>)
to get more information.
You can use the modules command with private remote repositories. Make sure that your local `git` is correctly configured with your private remote
and then specify the remote the same way you would do with a public remote repository.

### List modules

The `nf-core modules list` command provides the subcommands `remote` and `local` for listing modules installed in a remote repository and in the local pipeline respectively. Both subcommands come with the `--key <keywords>` option for filtering the modules by keywords.
The `nf-core modules list` command provides the subcommands `remote` and `local` for listing modules installed in a remote repository and in the local pipeline respectively. Both subcommands allow to use a pattern for filtering the modules by keywords eg: `nf-core modules list <subcommand> <keyword>`.

#### List remote modules

Expand Down Expand Up @@ -1078,7 +1066,7 @@ There are three additional flags that you can use when installing a module:

- `--force`: Overwrite a previously installed version of the module.
- `--prompt`: Select the module version using a cli prompt.
- `--sha <commit_sha>`: Install the module at a specific commit from the `nf-core/modules` repository.
- `--sha <commit_sha>`: Install the module at a specific commit.

### Update modules in a pipeline

Expand Down
Loading

0 comments on commit 4366c78

Please sign in to comment.