Merge pull request #59 from CenterForMedicalGeneticsGhent/dev

Release PR

Author: nvnieuwk

.github/CONTRIBUTING.md

@@ -110,4 +110,3 @@ To get started:
 Devcontainer specs:
 
 - [DevContainer config](.devcontainer/devcontainer.json)
-- [Dockerfile](.devcontainer/Dockerfile)

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -38,13 +38,14 @@ body:
     id: system
     attributes:
       label: System information
-      description: "* Nextflow version _(eg. 22.10.1)_
+      description: "* Nextflow version _(eg. 23.04.0)_
 
         * Hardware _(eg. HPC, Desktop, Cloud)_
 
         * Executor _(eg. slurm, local, awsbatch)_
 
-        * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter or Charliecloud)_
+        * Container engine: _(e.g. Docker, Singularity, Conda, Podman, Shifter, Charliecloud,
+        or Apptainer)_
 
         * OS _(eg. CentOS Linux, macOS, Linux Mint)_
 

.github/ISSUE_TEMPLATE/config.yml

@@ -1,7 +1,6 @@
 name: Feature request
 description: Suggest an idea for the CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing pipeline
-labels:
-  - enhancement
+labels: enhancement
 body:
   - type: textarea
     id: description

.github/ISSUE_TEMPLATE/feature_request.yml

@@ -15,8 +15,7 @@ Learn more about contributing: [CONTRIBUTING.md](https://github.com/CenterForMed
 
 - [ ] This comment contains a description of changes (with reason).
 - [ ] If you've fixed a bug or added code that should be tested, add tests!
-  - [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/tree/master/.github/CONTRIBUTING.md)
-  - [ ] If necessary, also make a PR on the CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing _branch_ on the [nf-core/test-datasets](https://github.com/nf-core/test-datasets) repository.
+- [ ] If you've added a new tool - have you followed the pipeline conventions in the [contribution docs](https://github.com/CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing/tree/master/.github/CONTRIBUTING.md)
 - [ ] Make sure your code lints (`nf-core lint`).
 - [ ] Ensure the test suite passes (`nextflow run . -profile test,docker --outdir <OUTDIR>`).
 - [ ] Usage Documentation in `docs/usage.md` is updated.

.github/PULL_REQUEST_TEMPLATE.md

@@ -13,7 +13,7 @@ jobs:
       - name: Check PRs
         if: github.repository == 'CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing'
         run: |
-          { [[ ${{github.event.pull_request.head.repo.full_name }} == CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing ]] && [[ $GITHUB_HEAD_REF = "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
+          { [[ ${{github.event.pull_request.head.repo.full_name }} == CenterForMedicalGeneticsGhent/nf-cmgg-preprocessing ]] && [[ $GITHUB_HEAD_REF == "dev" ]]; } || [[ $GITHUB_HEAD_REF == "patch" ]]
 
       # If the above check failed, post a comment on the PR explaining the failure
       # NOTE - this doesn't currently work if the PR is coming from a fork, due to limitations in GitHub actions secrets

.github/workflows/branch.yml

@@ -13,6 +13,10 @@ env:
   AWS_ACCESS_KEY_ID: ${{ secrets.UGENT_S3_ACCESS_KEY }}
   AWS_SECRET_ACCESS_KEY: ${{ secrets.UGENT_S3_SECRET_KEY }}
 
+concurrency:
+  group: "${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}"
+  cancel-in-progress: true
+
 jobs:
   test:
     name: Run pipeline with test data
@@ -22,12 +26,7 @@ jobs:
     strategy:
       matrix:
         NXF_VER:
-          - "22.10.0"
-          #- "latest-everything"
-        test:
-          - "test_flowcell"
-          #- "test_fastq"
-          #- "test_cram"
+          - 23.04.0
         aligner:
           #- "bowtie2"
           - "bwamem"
@@ -36,7 +35,7 @@ jobs:
           #- "snap"
     steps:
       - name: Check out pipeline code
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
 
       - name: Install Nextflow
         uses: nf-core/setup-nextflow@v1
@@ -45,4 +44,4 @@ jobs:
 
       - name: Run pipeline with test data
         run: |
-          nextflow run ${GITHUB_WORKSPACE} -profile ${{ matrix.test }},docker  --aligner ${{ matrix.aligner }} --outdir ./results
+          nextflow run ${GITHUB_WORKSPACE} -profile test,docker  --aligner ${{ matrix.aligner }} --outdir ./results

.github/workflows/ci.yml

@@ -0,0 +1,24 @@
+name: "Close user-tagged issues and PRs"
+on:
+  schedule:
+    - cron: "0 0 * * 0" # Once a week
+
+jobs:
+  clean-up:
+    runs-on: ubuntu-latest
+    permissions:
+      issues: write
+      pull-requests: write
+    steps:
+      - uses: actions/stale@v7
+        with:
+          stale-issue-message: "This issue has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment otherwise this issue will be closed in 20 days."
+          stale-pr-message: "This PR has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor. Remove stale label or add a comment if it is still useful."
+          close-issue-message: "This issue was closed because it has been tagged as awaiting-changes or awaiting-feedback by an nf-core contributor and then staled for 20 days with no activity."
+          days-before-stale: 30
+          days-before-close: 20
+          days-before-pr-close: -1
+          any-of-labels: "awaiting-changes,awaiting-feedback"
+          exempt-issue-labels: "WIP"
+          exempt-pr-labels: "WIP"
+          repo-token: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/clean-up.yml

@@ -24,7 +24,7 @@ jobs:
         env:
           GITHUB_TOKEN: ${{ secrets.nf_core_bot_auth_token }}
 
-      - uses: actions/setup-node@v2
+      - uses: actions/setup-node@v3
 
       - name: Install Prettier
         run: npm install -g prettier @prettier/plugin-php
@@ -34,10 +34,11 @@ jobs:
         id: prettier_status
         run: |
           if prettier --check ${GITHUB_WORKSPACE}; then
-            echo "::set-output name=result::pass"
+            echo "result=pass" >> $GITHUB_OUTPUT
           else
-            echo "::set-output name=result::fail"
+            echo "result=fail" >> $GITHUB_OUTPUT
           fi
+
       - name: Run 'prettier --write'
         if: steps.prettier_status.outputs.result == 'fail'
         run: prettier --write ${GITHUB_WORKSPACE}

.github/workflows/fix-linting.yml

@@ -78,13 +78,13 @@ jobs:
 
       - uses: actions/setup-python@v4
         with:
-          python-version: "3.7"
+          python-version: "3.11"
           architecture: "x64"
 
       - name: Install dependencies
         run: |
           python -m pip install --upgrade pip
-          pip install --upgrade --force-reinstall git+https://github.com/nf-core/tools.git@dev
+          pip install nf-core
 
       - name: Run nf-core lint
         env:

.github/workflows/linting.yml

@@ -1,4 +1,9 @@
 image: nfcore/gitpod:latest
+tasks:
+  - name: Update Nextflow and setup pre-commit
+    command: |
+      pre-commit install --install-hooks
+      nextflow self-update
 
 vscode:
   extensions: # based on nf-core.nf-core-extensionpack

.gitpod.yml

@@ -1,23 +1,31 @@
 repository_type: pipeline
+template:
+  name: nf-cmgg-preprocessing
+  description: Preprocessing workflow for sequencing data at CMGG
+  author: matthdsm
+  prefix: CenterForMedicalGeneticsGhent
 lint:
   files_exist:
-    - .github/workflows/awsfulltest.yml
-    - .github/workflows/awstest.yml
-    - assets/nf-core-nf-cmgg-preprocessing_logo_dark.png
     - assets/nf-core-nf-cmgg-preprocessing_logo_light.png
+    - conf/test.config
     - conf/test_full.config
-    - docs/images/nf-core-nf-cmgg-preprocessing_logo_dark.png
+    - CODE_OF_CONDUCT.md
     - docs/images/nf-core-nf-cmgg-preprocessing_logo_light.png
+    - docs/images/nf-core-nf-cmgg-preprocessing_logo_dark.png
+    - lib/nfcore_external_java_deps.jar
+    - .github/ISSUE_TEMPLATE/config.yml
+    - .github/workflows/awstest.yml
+    - .github/workflows/awsfulltest.yml
   files_unchanged:
-    - .github/ISSUE_TEMPLATE/feature_request.yml
-    - .github/PULL_REQUEST_TEMPLATE.md
-    - .github/workflows/linting.yml
+    - CODE_OF_CONDUCT.md
+    - assets/nf-core-nf-cmgg-preprocessing_logo_light.png
+    - docs/images/nf-core-nf-cmgg-preprocessing_logo_light.png
+    - docs/images/nf-core-nf-cmgg-preprocessing_logo_dark.png
+    - .github/ISSUE_TEMPLATE/bug_report.yml
     - .gitignore
-    - docs/README.md
+  multiqc_config:
+    - report_comment
   nextflow_config:
+    - params.input
     - manifest.name
     - manifest.homePage
-  multiqc_config:
-    - report_comment
-  readme:
-    - nextflow_badge

.nf-core.yml

@@ -0,0 +1,5 @@
+repos:
+  - repo: https://github.com/pre-commit/mirrors-prettier
+    rev: "v2.7.1"
+    hooks:
+      - id: prettier

.pre-commit-config.yaml

@@ -3,6 +3,24 @@
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v1.2.0
+
+- Add fix for sample replicates with different split sizes
+- Add option to use samtools for duplicate marking
+- Updated modules
+- Updated to nf-core template v2.8
+- Fix issue where unaligned samples were not merged and fixed naming
+- Add minimal support for sample "tags" in sample sheet
+- Add panel coverage analysis
+- Limit panel coverage analyses to samples with WES/WGS tags
+- Make BAM_ARCHIVE accept channels with cram files
+- Drop ELPREP support
+- Add support for cram and index outputs in SAMTOOLS_SORMADUP
+- Add option to disable marking duplicates
+- Update to nf-core template v2.9
+- Start using nf-validation for input parsing
+- Drop support for generating aligner indices
+
 ## v1.1.2
 
 - Support for bamprocessing with ELPREP

CHANGELOG.md

@@ -15,9 +15,11 @@
 - [Biobambam2](https://gitlab.com/german.tischler/biobambam2)
   > Tischler, G., Leonard, S. biobambam: tools for read pair collation based algorithms on BAM files. Source Code Biol Med 9, 13 (2014). <https://doi.org/10.1186/1751-0473-9-13>
 - [Bowtie2](http://bowtie-bio.sourceforge.net/bowtie2/index.shtml)
+
   > - Langmead B, Wilks C., Antonescu V., Charles R. Scaling read aligners to hundreds of threads on general-purpose processors. Bioinformatics. bty648.
   > - Langmead B, Salzberg S. Fast gapped-read alignment with Bowtie 2. Nature Methods. 2012, 9:357-359.
   > - Langmead B, Trapnell C, Pop M, Salzberg SL. Ultrafast and memory-efficient alignment of short DNA sequences to the human genome. Genome Biology 10:R25.
+
 - [BWA](http://bio-bwa.sourceforge.net/)
   > Li H. and Durbin R. (2009) Fast and accurate short read alignment with Burrows-Wheeler Transform. Bioinformatics, 25:1754-60. [PMID: 19451168]
 - [FastP](https://github.com/OpenGene/fastp)
@@ -33,9 +35,16 @@
   > Petr Danecek, James K Bonfield, Jennifer Liddle, John Marshall, Valeriu Ohan, Martin O Pollard, Andrew Whitwham, Thomas Keane, Shane A McCarthy, Robert M Davies, Heng Li
   > GigaScience, Volume 10, Issue 2, February 2021, giab008, <https://doi.org/10.1093/gigascience/giab008>
 - [Snap](http://snap.cs.berkeley.edu)
+
   > - Faster and More Accurate Sequence Alignment with SNAP. Matei Zaharia, William J. Bolosky, Kristal Curtis, Armando Fox, David Patterson, Scott Shenker, Ion Stoica, Richard M. Karp, and Taylor Sittler. arXiv:1111.5572v1, November 2011.
   > - Fuzzy set intersection based paired-end short-read alignment. William J. Bolosky, Arun Subramaniyan, Matei Zaharia, Ravi Pandya, Taylor Sittler, and David Patterson. BioRxiv, November 2021.
 
+  > Andrews, S. (2010). FastQC: A Quality Control Tool for High Throughput Sequence Data [Online].
+
+- [MultiQC](https://pubmed.ncbi.nlm.nih.gov/27312411/)
+
+  > Ewels P, Magnusson M, Lundin S, Käller M. MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics. 2016 Oct 1;32(19):3047-8. doi: 10.1093/bioinformatics/btw354. Epub 2016 Jun 16. PubMed PMID: 27312411; PubMed Central PMCID: PMC5039924.
+
 ## Software packaging/containerisation tools
 
 - [Anaconda](https://anaconda.com)
@@ -52,5 +61,8 @@
 
 - [Docker](https://dl.acm.org/doi/10.5555/2600239.2600241)
 
+  > Merkel, D. (2014). Docker: lightweight linux containers for consistent development and deployment. Linux Journal, 2014(239), 2. doi: 10.5555/2600239.2600241.
+
 - [Singularity](https://pubmed.ncbi.nlm.nih.gov/28494014/)
+
   > Kurtzer GM, Sochat V, Bauer MW. Singularity: Scientific containers for mobility of compute. PLoS One. 2017 May 11;12(5):e0177459. doi: 10.1371/journal.pone.0177459. eCollection 2017. PubMed PMID: 28494014; PubMed Central PMCID: PMC5426675.