Skip to content

Commit 7fcaf43

Browse files
authored
Merge pull request #91 from nf-cmgg/dev
Release PR v2.0
2 parents 660d3d6 + 161dde8 commit 7fcaf43

File tree

319 files changed

+16906
-6581
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

319 files changed

+16906
-6581
lines changed

.devcontainer/devcontainer.json

+3
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,8 @@
22
"name": "nfcore",
33
"image": "nfcore/gitpod:latest",
44
"remoteUser": "gitpod",
5+
"runArgs": ["--privileged"],
6+
57
// Configure tool-specific properties.
68
"customizations": {
79
// Configure properties specific to VS Code.
@@ -18,6 +20,7 @@
1820
"python.linting.pydocstylePath": "/opt/conda/bin/pydocstyle",
1921
"python.linting.pylintPath": "/opt/conda/bin/pylint"
2022
},
23+
2124
// Add the IDs of extensions you want installed when the container is created.
2225
"extensions": ["ms-python.python", "ms-python.vscode-pylance", "nf-core.nf-core-extensionpack"]
2326
}

.editorconfig

+15-2
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@ trim_trailing_whitespace = true
88
indent_size = 4
99
indent_style = space
1010

11-
[*.{md,yml,yaml,html,css,scss,js,cff}]
11+
[*.{md,yml,yaml,html,css,scss,js}]
1212
indent_size = 2
1313

1414
# These files are edited and tested upstream in nf-core/modules
@@ -18,7 +18,20 @@ end_of_line = unset
1818
insert_final_newline = unset
1919
trim_trailing_whitespace = unset
2020
indent_style = unset
21-
indent_size = unset
21+
[/subworkflows/nf-core/**]
22+
charset = unset
23+
end_of_line = unset
24+
insert_final_newline = unset
25+
trim_trailing_whitespace = unset
26+
indent_style = unset
2227

2328
[/assets/email*]
2429
indent_size = unset
30+
31+
# ignore Readme
32+
[README.md]
33+
indent_style = unset
34+
35+
# ignore python
36+
[*.{py,md}]
37+
indent_style = unset

.github/CONTRIBUTING.md

+15-9
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,20 @@
1-
# CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing: Contributing Guidelines
1+
# nf-cmgg/preprocessing: Contributing Guidelines
22

33
Hi there!
4-
Many thanks for taking an interest in improving CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing.
4+
Many thanks for taking an interest in improving nf-cmgg/preprocessing.
55

6-
We try to manage the required tasks for CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing using GitHub issues, you probably came to this page when creating one.
6+
We try to manage the required tasks for nf-cmgg/preprocessing using GitHub issues, you probably came to this page when creating one.
77
Please use the pre-filled template to save time.
88

99
However, don't be put off by this template - other more general issues and suggestions are welcome!
1010
Contributions to the code are even more welcome ;)
1111

1212
## Contribution workflow
1313

14-
If you'd like to write some code for CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing, the standard workflow is as follows:
14+
If you'd like to write some code for nf-cmgg/preprocessing, the standard workflow is as follows:
1515

16-
1. Check that there isn't already an issue about your idea in the [CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing issues](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
17-
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing repository](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing) to your GitHub account
16+
1. Check that there isn't already an issue about your idea in the [nf-cmgg/preprocessing issues](https://github.com/nf-cmgg/preprocessing/issues) to avoid duplicating work. If there isn't one already, please create one so that others know you're working on this
17+
2. [Fork](https://help.github.com/en/github/getting-started-with-github/fork-a-repo) the [nf-cmgg/preprocessing repository](https://github.com/nf-cmgg/preprocessing) to your GitHub account
1818
3. Make the necessary changes / additions within your forked repository following [Pipeline conventions](#pipeline-contribution-conventions)
1919
4. Use `nf-core schema build` and add any new parameters to the pipeline JSON schema (requires [nf-core tools](https://github.com/nf-core/tools) >= 1.10).
2020
5. Submit a Pull Request against the `dev` branch and wait for the code to be reviewed and merged
@@ -23,6 +23,12 @@ If you're not used to this workflow with git, you can start with some [docs from
2323

2424
## Tests
2525

26+
You have the option to test your changes locally by running the pipeline. For receiving warnings about process selectors and other `debug` information, it is recommended to use the debug profile. Execute all the tests with the following command:
27+
28+
```bash
29+
nf-test test --profile debug,test,docker --verbose
30+
```
31+
2632
When you create a pull request with changes, [GitHub Actions](https://github.com/features/actions) will run automatic tests.
2733
Typically, pull-requests are only fully reviewed when these tests are passing, though of course we can help out before then.
2834

@@ -52,7 +58,7 @@ These tests are run both with the latest available version of `Nextflow` and als
5258

5359
## Pipeline contribution conventions
5460

55-
To make the CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.
61+
To make the nf-cmgg/preprocessing code and processing logic more understandable for new contributors and to ensure quality, we semi-standardise the way the code and other contributions are written.
5662

5763
### Adding a new step
5864

@@ -79,7 +85,7 @@ Once there, use `nf-core schema build` to add to `nextflow_schema.json`.
7985

8086
Sensible defaults for process resource requirements (CPUs / memory / time) for a process should be defined in `conf/base.config`. These should generally be specified generic with `withLabel:` selectors so they can be shared across multiple processes/steps of the pipeline. A nf-core standard set of labels that should be followed where possible can be seen in the [nf-core pipeline template](https://github.com/nf-core/tools/blob/master/nf_core/pipeline-template/conf/base.config), which has the default process as a single core-process, and then different levels of multi-core configurations for increasingly large memory requirements defined with standardised labels.
8187

82-
The process resources can be passed on to the tool dynamically within the process with the `${task.cpu}` and `${task.memory}` variables in the `script:` block.
88+
The process resources can be passed on to the tool dynamically within the process with the `${task.cpus}` and `${task.memory}` variables in the `script:` block.
8389

8490
### Naming schemes
8591

@@ -102,7 +108,7 @@ This repo includes a devcontainer configuration which will create a GitHub Codes
102108

103109
To get started:
104110

105-
- Open the repo in [Codespaces](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/codespaces)
111+
- Open the repo in [Codespaces](https://github.com/nf-cmgg/preprocessing/codespaces)
106112
- Tools installed
107113
- nf-core
108114
- Nextflow

.github/ISSUE_TEMPLATE/bug_report.yml

+1-2
Original file line numberDiff line numberDiff line change
@@ -49,7 +49,6 @@ body:
4949
5050
* OS _(eg. CentOS Linux, macOS, Linux Mint)_
5151
52-
* Version of CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing _(eg. 1.1,
53-
1.5, 1.8.2)_
52+
* Version of nf-cmgg/preprocessing _(eg. 1.1, 1.5, 1.8.2)_
5453
5554
"

.github/ISSUE_TEMPLATE/feature_request.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
name: Feature request
2-
description: Suggest an idea for the CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing pipeline
2+
description: Suggest an idea for the nf-cmgg/preprocessing pipeline
33
labels: enhancement
44
body:
55
- type: textarea

.github/PULL_REQUEST_TEMPLATE.md

+5-4
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,24 @@
11
<!--
2-
# CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing pull request
2+
# nf-cmgg/preprocessing pull request
33
4-
Many thanks for contributing to CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing!
4+
Many thanks for contributing to nf-cmgg/preprocessing!
55
66
Please fill in the appropriate checklist below (delete whatever is not relevant).
77
These are the most common things requested on pull requests (PRs).
88
99
Remember that PRs should be made against the dev branch, unless you're preparing a pipeline release.
1010
11-
Learn more about contributing: [CONTRIBUTING.md](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/tree/master/.github/CONTRIBUTING.md)
11+
Learn more about contributing: [CONTRIBUTING.md](https://github.com/nf-cmgg/preprocessing/tree/master/.github/CONTRIBUTING.md)
1212
-->
1313

1414
## PR checklist
1515

1616
- [ ] This comment contains a description of changes (with reason).
1717
- [ ] If you've fixed a bug or added code that should be tested, add tests!
18-
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/tree/master/.github/CONTRIBUTING.md)
18+
- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/nf-cmgg/preprocessing/tree/master/.github/CONTRIBUTING.md)
1919
- [ ] Make sure your code lints (`nf-core lint`).
2020
- [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
21+
- [ ] Check for unexpected warnings in debug mode (`nextflow run . -profile debug,test,docker --outdir <OUTDIR>`).
2122
- [ ] Usage Documentation in `docs/usage.md` is updated.
2223
- [ ] Output Documentation in `docs/output.md` is updated.
2324
- [ ] `CHANGELOG.md` is updated.

.github/workflows/branch.yml

+3-3
Original file line numberDiff line numberDiff line change
@@ -11,15 +11,15 @@ jobs:
1111
steps:
1212
# PRs to the nf-core repo master branch are only ok if coming from the nf-core repo `dev` or any `patch` branches
1313
- name: Check PRs
14-
if: github.repository == 'CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing'
14+
if: github.repository == 'nf-cmgg/preprocessing'
1515
run: |
16-
{ [[ ${{github.event.pull_request.head.repo.full_name }} == CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing ]] && [[ $GITHUB_HEAD_REF == "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
16+
{ [[ ${{github.event.pull_request.head.repo.full_name }} == nf-cmgg/preprocessing ]] && [[ $GITHUB_HEAD_REF == "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
1717
1818
# If the above check failed, post a comment on the PR explaining the failure
1919
# NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets
2020
- name: Post PR comment
2121
if: failure()
22-
uses: mshick/add-pr-comment@v1
22+
uses: mshick/add-pr-comment@b8f338c590a895d50bcbfa6c5859251edc8952fc # v2
2323
with:
2424
message: |
2525
## This PR is against the `master` branch :x:

.github/workflows/ci.yml

+67-12
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,10 @@ on:
1010

1111
env:
1212
NXF_ANSI_LOG: false
13+
NFT_VER: "0.8.4"
14+
NFT_WORKDIR: "~"
15+
NFT_DIFF: "pdiff"
16+
NFT_DIFF_ARGS: "--line-numbers --expand-tabs=2"
1317
AWS_ACCESS_KEY_ID: ${{ secrets.UGENT_S3_ACCESS_KEY }}
1418
AWS_SECRET_ACCESS_KEY: ${{ secrets.UGENT_S3_SECRET_KEY }}
1519

@@ -19,29 +23,80 @@ concurrency:
1923

2024
jobs:
2125
test:
22-
name: Run pipeline with test data
26+
name: ${{ matrix.tags }} ${{ matrix.profile }} NF-${{ matrix.NXF_VER }}
2327
# Only run on push if this is the nf-core dev branch (merged PRs)
24-
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing') }}"
28+
if: "${{ github.event_name != 'push' || (github.event_name == 'push' && github.repository == 'nf-cmgg/preprocessing') }}"
2529
runs-on: ubuntu-latest
2630
strategy:
2731
matrix:
2832
NXF_VER:
29-
- 23.04.0
30-
aligner:
31-
#- "bowtie2"
32-
- "bwamem"
33-
#- "bwamem2"
34-
#- "dragmap"
35-
#- "snap"
33+
- "24.04.2"
34+
- "latest-everything"
35+
tags:
36+
- "modules/local/panelcoverage"
37+
- "subworkflows/local/bam_qc"
38+
- "subworkflows/local/coverage"
39+
- "subworkflows/local/fastq_to_aligned_cram"
40+
- "subworkflows/local/fastq_align_rna"
41+
- "subworkflows/local/fastq_to_unaligned_cram"
42+
- "workflows/preprocessing"
43+
- "pipeline"
44+
profile:
45+
- "test,docker"
3646
steps:
3747
- name: Check out pipeline code
38-
uses: actions/checkout@v3
48+
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4
3949

4050
- name: Install Nextflow
4151
uses: nf-core/setup-nextflow@v1
4252
with:
4353
version: "${{ matrix.NXF_VER }}"
4454

45-
- name: Run pipeline with test data
55+
- name: Disk space cleanup
56+
uses: jlumbroso/free-disk-space@54081f138730dfa15788a46383842cd2f914a1be # v1.3.1
57+
58+
- uses: actions/setup-python@v4
59+
with:
60+
python-version: "3.11"
61+
architecture: "x64"
62+
63+
- name: Install pdiff to see diff between nf-test snapshots
64+
run: |
65+
python -m pip install --upgrade pip
66+
pip install pdiff
67+
68+
- name: Cache nf-test installation
69+
id: cache-software
70+
uses: actions/cache@v3
71+
with:
72+
path: |
73+
/usr/local/bin/nf-test
74+
/home/runner/.nf-test/nf-test.jar
75+
key: ${{ runner.os }}-${{ env.NFT_VER }}-nftest
76+
77+
- name: Install nf-test
78+
if: steps.cache-software.outputs.cache-hit != 'true'
4679
run: |
47-
nextflow run ${GITHUB_WORKSPACE} -profile test,docker --aligner ${{ matrix.aligner }} --outdir ./results
80+
wget -qO- https://code.askimed.com/install/nf-test | bash
81+
sudo mv nf-test /usr/local/bin/
82+
83+
- name: Run nf-test
84+
run: |
85+
nf-test test --verbose --tag "${{ matrix.tags }}" --profile "+${{ matrix.profile }}" --junitxml=test.xml --tap=test.tap
86+
87+
- uses: pcolby/tap-summary@v1
88+
with:
89+
path: >-
90+
test.tap
91+
92+
- name: Output log on failure
93+
if: failure()
94+
run: |
95+
sudo apt install bat > /dev/null
96+
batcat --decorations=always --color=always ${{ github.workspace }}/.nf-test/tests/*/meta/nextflow.log
97+
98+
- name: Publish Test Report
99+
uses: mikepenz/action-junit-report@v3
100+
if: always() # always run even if the previous step fails
101+
with:
102+
report_paths: test.xml

.github/workflows/clean-up.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ jobs:
1010
issues: write
1111
pull-requests: write
1212
steps:
13-
- uses: actions/stale@v7
13+
- uses: actions/stale@28ca1036281a5e5922ead5184a1bbf96e5fc984e # v9
1414
with:
1515
stale-issue-message: "This issue has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment otherwise this issue will be closed in 20 days."
1616
stale-pr-message: "This PR has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment if it is still useful."
+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
name: Test successful pipeline download with 'nf-core download'
2+
3+
# Run the workflow when:
4+
# - dispatched manually
5+
# - when a PR is opened or reopened to master branch
6+
# - the head branch of the pull request is updated, i.e. if fixes for a release are pushed last minute to dev.
7+
on:
8+
workflow_dispatch:
9+
inputs:
10+
testbranch:
11+
description: "The specific branch you wish to utilize for the test execution of nf-core download."
12+
required: true
13+
default: "dev"
14+
pull_request:
15+
types:
16+
- opened
17+
branches:
18+
- master
19+
pull_request_target:
20+
branches:
21+
- master
22+
23+
env:
24+
NXF_ANSI_LOG: false
25+
26+
jobs:
27+
download:
28+
runs-on: ubuntu-latest
29+
steps:
30+
- name: Install Nextflow
31+
uses: nf-core/setup-nextflow@b9f764e8ba5c76b712ace14ecbfcef0e40ae2dd8 # v1
32+
33+
- uses: actions/setup-python@0a5c61591373683505ea898e09a3ea4f39ef2b9c # v5
34+
with:
35+
python-version: "3.11"
36+
architecture: "x64"
37+
- uses: eWaterCycle/setup-singularity@931d4e31109e875b13309ae1d07c70ca8fbc8537 # v7
38+
with:
39+
singularity-version: 3.8.3
40+
41+
- name: Install dependencies
42+
run: |
43+
python -m pip install --upgrade pip
44+
pip install git+https://github.com/nf-core/tools.git@dev
45+
46+
- name: Get the repository name and current branch set as environment variable
47+
run: |
48+
echo "REPO_LOWERCASE=${GITHUB_REPOSITORY,,}" >> ${GITHUB_ENV}
49+
echo "REPOTITLE_LOWERCASE=$(basename ${GITHUB_REPOSITORY,,})" >> ${GITHUB_ENV}
50+
echo "REPO_BRANCH=${{ github.event.inputs.testbranch || 'dev' }}" >> ${GITHUB_ENV}
51+
52+
- name: Download the pipeline
53+
env:
54+
NXF_SINGULARITY_CACHEDIR: ./
55+
run: |
56+
nf-core download ${{ env.REPO_LOWERCASE }} \
57+
--revision ${{ env.REPO_BRANCH }} \
58+
--outdir ./${{ env.REPOTITLE_LOWERCASE }} \
59+
--compress "none" \
60+
--container-system 'singularity' \
61+
--container-library "quay.io" -l "docker.io" -l "ghcr.io" \
62+
--container-cache-utilisation 'amend' \
63+
--download-configuration
64+
65+
- name: Inspect download
66+
run: tree ./${{ env.REPOTITLE_LOWERCASE }}
67+
68+
- name: Run the downloaded pipeline
69+
env:
70+
NXF_SINGULARITY_CACHEDIR: ./
71+
NXF_SINGULARITY_HOME_MOUNT: true
72+
run: nextflow run ./${{ env.REPOTITLE_LOWERCASE }}/$( sed 's/\W/_/g' <<< ${{ env.REPO_BRANCH }}) -stub -profile test,singularity --outdir ./results

0 commit comments

Comments
 (0)