fastq > cram using nf-core

[ellendejong]

What has changed?

DxNextflowRNA workflow to create a cram file (output) from fastq files (input).
Since this is a major refactor with current workflow in dev/main, the code review should focus on the new files instead of comparing the differences.

The branch name does not reflect the feature and is not conform our standards, I apologize. :)

Noteworthy in my opinion.

1. Main workflow

I defined two subworkflows;

• fastq > cram (trimming, filter or rRNA, alignment, removal of duplicates, umi deduplication)
• QC tools

workflow.OnComplete is responsible to send an email when workflow completed. I used nf-core functionality, which means emails differ from format compared to DxNextflowWES for example. This can be configured using templates, but is considered out-of-scope for v1.0.0.

CRAM output
If cram files will be used as input in follow-up analyses, the same reference files are required to be able to process cram files. Therefore, the reference files used to create the cram files are added to the multiqc report.

2. Naming conventions

I tried to use the naming conventions and guidelines of the nf-core community. As well as the code styling guidelines.
I decided that linting usingruff and pre-commit configuration will be part of future releases.

3. MultiQC

MultiQC is added to the main workflow, to enable a single report per analysis.
Sample grouping is used (available since MultiQC v1.25), although it is not supported for all tools used in this pipeline.

4. Reference files

Instructions to generate/download the required reference files are added to the README.md.

5. Dynamic resources

I used previous versions of the pipeline to calculate dynamic resources, since most tools are used in both pipelines. The current settings might need some tweaking over time.

6. SortMeRNA

SortMeRna is available as module in nf-core. However, I encountered a bug when using this tool. It appears I needed to update it to version >= 4.3.7. Solution: override container via modules.config. Once the update is available in nf-core, I should switch and remove the override again.

Index SortMeRNA.
Runtimes of SortMeRNA are optimized by creating a sortmerna index first. This is done by executing the tool with settings as configured SORTMERNA_INDEX. Convenietly, I created a workflow to do just that. I am not certain if I should add the workflow to this repo, or whether instructions in README.md would be sufficient. Please let me know your opinion :)

7. Completion email

Waiting for fix in nf-core/tools#3304.
Implemented the fixes locally (moved nf-core/utils_nfcore_pipeline to local).
Until pull-request with fix is merged, local implementation is required.

8. Considered out-of-scope for release v1.0.0

• Email layout adjustments will be something for the future :)
• docs (usage.md, output.md, README.md). Currently limited to either the template version created by nf-core tools or edited in case of necessary information only.

[mraves2]

The current pipeline (main.nf in repo main dir) is divided into 3 steps:

• subworkflow for pipeline initiation
• DXNEXTFLOWRNA, a umcugenetics-specific workflow
• subworkflow for pipeline completion

There are three folders that contain pieces of the workflow:

• modules
• subworkflows
• workflows

The flow can be followed from the main.nf scripts in eacht (sub)folder.
main.nf: include { DXNEXTFLOWRNA } from './workflows/dxnextflowrna'
include { PIPELINE_INITIALISATION } from './subworkflows/local/utils_umcugenetics_dxnextflowrna_pipeline'
include { PIPELINE_COMPLETION } from './subworkflows/local/utils_umcugenetics_dxnextflowrna_pipeline'

The DXNEXTFLOWRNA workflow consists of modules, subworkflows and functions:
// MODULES
include { MULTIQC } from '../modules/nf-core/multiqc/main'
// SUBWORKFLOWS
include { FASTQ_BAM_QC } from '../subworkflows/local/fastq_bam_qc'
include { FASTQ_TRIM_FILTER_ALIGN_DEDUP } from '../subworkflows/local/fastq_trim_filter_align_dedup'

workflows/dxnextflowrna.nf: definition of reference channels, input channels and workflows
SUBWORKFLOW: Run fastq_trim_filter_align_dedup contains TrimGalore, SortMeRNA, STAR and SAMTOOLS and UMITOOLS (ends with CRAM files)
SUBWORKFLOW: Run fastq_bam_qc contains FastQC, PICARD and PRESEQ voor QC
MODULE: MultiQC

Modules are unmodified. Whenever modifications have been made, files have been moved from a nf-core folder to a local folder. Do the files in nf-core folder still need to be maintained? This seems to superfluous.
For example, there is no functional difference between subworkflows/nf-core/utils_nfcore_pipeline and subworkflows/local/utils_nfcore_pipeline; can one of the folders be removed?

Names of processes and parameters look informative and are consistent with the conventions in https://nf-co.re/docs/guidelines/components/overview

Currently, the workflow cannot be tested by anyone other than the author, because of the temporary fix for SortMeRNA, which requires access to the file
references/sortmerna/sortmerna_v4.3.4_db/smr_v4.3_sensitive_db.fasta

It is unclear to me at this moment whether follow-up analysis tools can work with CRAM files or will in the future; featureCounts for exammple (see https://www.biostars.org/p/9577607/). Is it possible for the time being to also export the bam files?

Alignment of blocks of similar entries is sometimes done as described in https://nf-co.re/docs/contributing/code_editors_and_styling/harshil_alignment and sometimes not. For example, the include blocks are nicely aligned on { and }, the workflow take channels are aligned on // but the emit channels are only aligned on = but not on //. Be consistent. Personally, I like this kind of aligment; it makes it easier to read, but some in our team do not agree. The nf-core team uses this alignment (see for example subworkflows/nf-core/bam_dedup_stats_samtools_umitools/main.nf), so I guess we should too.

Nice to see the versions of each module in the MultiQC html file. Just one looks strange: FastQC (, '0.12.1') Perhaps this can be fixed?

[ellendejong]

Modules are unmodified. Whenever modifications have been made, files have been moved from a nf-core folder to a local folder. Do the files in nf-core folder still need to be maintained?

Yes / No. Files are a local copy of the github repositories of each module/subworkflow. Would be nice to keep up-to-date and sync everything once in a while.

For example, there is no functional difference between subworkflows/nf-core/utils_nfcore_pipeline and subworkflows/local/utils_nfcore_pipeline; can one of the folders be removed?

Yes. I used a local copy to debug sending emails with param summary and multiqc attachment. In the end, no changes were required and forgot to remove the local copy.

Currently, the workflow cannot be tested by anyone other than the author, because of the temporary fix for SortMeRNA, which requires access to the file references/sortmerna/sortmerna_v4.3.4_db/smr_v4.3_sensitive_db.fasta

Oops. Fixed.

It is unclear to me at this moment whether follow-up analysis tools can work with CRAM files or will in the future; featureCounts for exammple (see https://www.biostars.org/p/9577607/). Is it possible for the time being to also export the bam files?

Quite likely not all downstream tools will support cram. I don't want to export bam files as well, since those will be included on the archive as well. The way to go is to include a cram to bam process in downstream (sub)workflows.

If those tools will be added as an extention of current workflow dxnextflowrna, the bam files can be used ofcourse. (no export needed to forward files between workdirs).

Alignment of blocks of similar entries is sometimes done as described in https://nf-co.re/docs/contributing/code_editors_and_styling/harshil_alignment and sometimes not. For example, the include blocks are nicely aligned on { and }, the workflow take channels are aligned on // but the emit channels are only aligned on = but not on //. Be consistent. Personally, I like this kind of aligment; it makes it easier to read, but some in our team do not agree. The nf-core team uses this alignment (see for example subworkflows/nf-core/bam_dedup_stats_samtools_umitools/main.nf), so I guess we should too.

Yes. Nf-core requires the harshil alignment. I am one of those peoples that are not a fan :D It might grow on me over time, who knows. I used an extension in VSCode to do the harshil alignment for me, so nothing manually. I could run the formatter again. If it doesnot change anything, I am hesitant to change the results myself / manually. Especially if it would revert those changes everytime i use the formatter...

• Check Harshil alignment formatter on local files (nf-core files excluded).

Nice to see the versions of each module in the MultiQC html file. Just one looks strange: FastQC (, '0.12.1') Perhaps this can be fixed?

Might be, although no clue how :)
output of multiqc shows indeed this weird format of version:
grep -i fastqc /hpc/diaggen/users/ellen/rna/nf_add_trimgalore_v1.0.0/work/0c/6cb4fb44f2003f1e888b1d9077c4ea/nf_add_trimgalore_v1.0.0_multiqc_report_data/multiqc_software_versions.txt
FastQC (<Version('0.12.1')>, '0.12.1')

Can't find this in workflow software versions file:
/hpc/diaggen/users/ellen/rna/nf_add_trimgalore_v1.0.0/pipeline_info/nf_core_pipeline_software_mqc_versions.yml

Neither in fastqcs output:
/hpc/diaggen/users/ellen/rna/nf_add_trimgalore_v1.0.0/work/3b/b81fe58855391114ffd1870fce4c11/versions.yml

Bit of a mystery to me. I will ask around in our team to see if someone can help me out :)

• Solve version format of fastqc.

[mraves2]

Approved.