Subworkflow for organelles #18

ksenia-krasheninnikova · 2023-06-29T12:24:42Z

Implements assembling mito from raw reads and from assembly

PR checklist

github-actions · 2023-06-29T12:26:31Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit 10093d9

+| ✅ 131 tests passed       |+
#| ❔  18 tests were ignored |#
!| ❗  14 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes.config
readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
pipeline_todos - TODO string in methods_description_template.yml: #Update the HTML below to your prefered methods description, e.g. add publication citation for this pipeline
pipeline_todos - TODO string in WorkflowMain.groovy: Add Zenodo DOI for pipeline after first release
pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.
pipeline_todos - TODO string in output.md: Write this documentation describing your workflow's output
pipeline_todos - TODO string in usage.md: Add documentation about anything specific to running your pipeline. For general topics, please point to (and add to) the main nf-core website.
pipeline_todos - TODO string in genomeassembly.nf: Add all file path parameters for the pipeline to the list below
system_exit - System.exit in WorkflowSanger-tol-genomeassembly.groovy: System.exit(1) [line 51]
schema_description - No description provided in schema for parameter: timestamp
schema_description - No description provided in schema for parameter: hifiasm
schema_description - No description provided in schema for parameter: hifiasmhic

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowGenomeassembly.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-genomeassembly_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '1.0dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (215 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: ci.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2023-10-30 10:20:45

priyanka-surana

Let's test this after merging the other two PRs and then good to move forward. You might want to add associated documentation to this PR.

priyanka-surana · 2023-07-18T11:43:53Z

modules/local/mitohifi.nf

Is this the module you are trying to add to nf-core?

Yes, this is the one

priyanka-surana · 2023-07-18T11:45:59Z

subworkflows/local/organelles.nf

+                        code)    
+    }
+
+    emit:


Can some of the channel engineering be moved above for a cleaner emit section? It is very hard to follow right now.

Seems it looks better now.

priyanka-surana

Do you want to make the changes and merge this PR? And do the changes for CI testing in a separate PR?

subworkflows/local/organelles.nf

ksenia-krasheninnikova · 2023-10-17T15:03:49Z

Removed all redundant files from the emit section. I think it can be merged in now. Thanks @priyanka-surana

priyanka-surana

A couple minor changes requested.

priyanka-surana · 2023-10-17T15:36:57Z

nextflow.config

@@ -19,6 +19,7 @@ params {

    // Assembly options
    hifiasm_hic_on             = false
+    organelles_on              = true


We are recommended to keep the default as false and then turn on via command line flags or the config.

priyanka-surana · 2023-10-17T15:37:27Z

subworkflows/local/organelles.nf

+
+    emit:
+
+    versions = ch_versions


Now there are no output files. Is this intentional?

While we need this files in the final output, we don't need them down the workflow. Initially I thought the emit section has to do something with the files being published, but actually it works fine the way it is now.

muffato · 2023-10-17T19:55:45Z

I find this emit section confusing and difficult to navigate. Could be just me since I am not familiar with MitoHifi. Looking for a more informed view @muffato since you are familiar with the tool.

I'm not that familiar with MitoHifi :) !

I can see you cleaned up the entire emit section. It simplifies things a lot ! For the record, I had some questions about that structure:

Could you make the MitoHifi module take a single input like this:
1. Define it as tuple val(meta), path(input) rather than the mutually-exclusive reads and contigs
2. Use $args $input in the MitoHifi command-line rather than ${run_type}
3. First check that $args does contain either -c or -r (one of them is mandatory)
Then, do you need this one subworkflow to choose between the assembly and read mode and duplicate the input and output channels ? Could you imagine having a single subworkflow that is agnostic of the mode (just one input channel and one et of output channels), and in the parent workflow you would import it twice: once as ORGANELLES_READS and once as ORGANELLES_ASM

ksenia-krasheninnikova · 2023-10-20T12:15:38Z

Could you make the MitoHifi module take a single input like this:

Define it as tuple val(meta), path(input) rather than the mutually-exclusive reads and contigs

Use $args $input in the MitoHifi command-line rather than ${run_type}

First check that $args does contain either -c or -r (one of them is mandatory)

Then, do you need this one subworkflow to choose between the assembly and read mode and duplicate the input and output channels ? Could you imagine having a single subworkflow that is agnostic of the mode (just one input channel and one et of output channels), and in the parent workflow you would import it twice: once as ORGANELLES_READS and once as ORGANELLES_ASM

Thank you @muffato for suggestions. I've updated the code, could you have a look if it's what you mean please? I had to put -r/-c into ext.args2 because the input file should follow the corresponding option immediately:

usage: MitoHiFi [-h] (-r <reads>.fasta | -c <contigs>.fasta) -f
                <relatedMito>.fasta -g <relatedMito>.gbk -t <THREADS> [-d]
                [-a {animal,plant,fungi}] [-p <PERC>] [-m <BLOOM FILTER>]
                [--max-read-len MAX_READ_LEN] [--mitos]
                [--circular-size CIRCULAR_SIZE]
                [--circular-offset CIRCULAR_OFFSET] [-winSize WINSIZE]
                [-covMap COVMAP] [-v] [-o <GENETIC CODE>]

The purpose of the original implementation of the organelles workflow was to separate the whole organelles logic into the separate workflow to lighten up the main workflow. My concern though is that the main workflow getting rather long. Is it a standard practice to keep it that way? It will become more readable when I add documentation.

muffato

I like this. I find it much more straightforward !
Can you propose your updated mitohifi/mitohifi module to nf-core (can do that after this PR) and suggest mahesh-panchal and scorreard for review ? They both work on genome-assembly pipelines.

modules/local/get_calcuts_params.nf

modules/nf-core/mitohifi/findmitoreference/main.nf

ksenia-krasheninnikova · 2023-10-30T10:42:50Z

@muffato
Merged it updated mitohifi module from nf-core. All the review items must have been addressed for now!
PR ready for the final review.
Thanks!

muffato

As you said, the module is now in sync with nf-core, meaning no local .diff file.

All good !

Ksenia Krasheninnikova added 7 commits June 15, 2023 09:27

Initial commit for mitohifi

8146f6a

Organelle subworkflow

aa65ab4

Work in progress mitohifi

a9c9622

Updates to mitohifi output and testing of the full pipeline

59801ad

Code cleanup

1b874bf

Prettier checks

2fe331e

Switch polishing on in the config

a2f987b

ksenia-krasheninnikova added the enhancement New feature or request label Jun 29, 2023

ksenia-krasheninnikova requested a review from priyanka-surana June 29, 2023 12:24

priyanka-surana reviewed Jul 18, 2023

View reviewed changes

Ksenia Krasheninnikova and others added 7 commits July 25, 2023 15:53

Refactor emit section for organelles subworkflow

57a953b

Merge branch 'dev' into mitohifi

b8390a2

Fix issues after automatic merge

ccac98f

Lint and prettify

d4336a1

Concat reads for mitohifi input

57beb02

Fix merging discrepancies

06e304e

Prettier fix

53de040

ksenia-krasheninnikova requested a review from priyanka-surana October 6, 2023 12:58

Ksenia Krasheninnikova added 6 commits October 11, 2023 11:16

CI testing

e0b1d62

Fix syntax error

46c609e

Add github testing profile to config

7833a35

Fix spelling error

e46fc77

Make organelles optional

4e91612

Switch off extra runs for github test

21aba97

priyanka-surana reviewed Oct 17, 2023

View reviewed changes

subworkflows/local/organelles.nf Outdated Show resolved Hide resolved

Drop redundant lines in emit section

d1d9816

ksenia-krasheninnikova requested a review from priyanka-surana October 17, 2023 15:01

priyanka-surana reviewed Oct 17, 2023

View reviewed changes

Ksenia Krasheninnikova added 2 commits October 17, 2023 16:44

Switch off bloom filter for hifiasm

c820100

Organelles on

c9d98f3

Ksenia Krasheninnikova added 9 commits October 19, 2023 10:44

Switch off bloom filter for hifiasm

e1803af

Prettier modules fix

0c4b577

Fix error

dfe8372

Fix CI testing errors

e68eb98

Swtich on polishing and hifiasm hic for githuub

ab56821

Fix image path

7c021e8

Switch off polishing

0f4f79c

Change logic of mitohifi input

c3df695

Clean code

1e80e13

ksenia-krasheninnikova requested a review from muffato October 20, 2023 12:16

muffato reviewed Oct 23, 2023

View reviewed changes

modules/local/get_calcuts_params.nf Outdated Show resolved Hide resolved

modules/nf-core/mitohifi/findmitoreference/main.nf Show resolved Hide resolved

Ksenia Krasheninnikova added 3 commits October 23, 2023 10:37

Change container path to more basic

c6d7c9a

Update mitohifi to nf-core

fbbbc59

Prettier checks

10093d9

ksenia-krasheninnikova requested a review from muffato October 30, 2023 10:39

ksenia-krasheninnikova mentioned this pull request Oct 31, 2023

Mitohifi channel input nf-core/modules#4254

Merged

14 tasks

muffato approved these changes Oct 31, 2023

View reviewed changes

ksenia-krasheninnikova merged commit 814f6cd into dev Oct 31, 2023
6 checks passed

muffato deleted the mitohifi branch November 21, 2023 22:32

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Subworkflow for organelles #18

Subworkflow for organelles #18

ksenia-krasheninnikova commented Jun 29, 2023

github-actions bot commented Jun 29, 2023 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

priyanka-surana left a comment

priyanka-surana Jul 18, 2023

ksenia-krasheninnikova Jul 25, 2023

priyanka-surana Jul 18, 2023

ksenia-krasheninnikova Jul 25, 2023

priyanka-surana left a comment

ksenia-krasheninnikova commented Oct 17, 2023

priyanka-surana left a comment

priyanka-surana Oct 17, 2023

ksenia-krasheninnikova Oct 17, 2023

priyanka-surana Oct 17, 2023

ksenia-krasheninnikova Oct 17, 2023

ksenia-krasheninnikova Oct 17, 2023

muffato commented Oct 17, 2023

ksenia-krasheninnikova commented Oct 20, 2023 •

edited

Loading

muffato left a comment

ksenia-krasheninnikova commented Oct 30, 2023

muffato left a comment

+                                      code)
+                  }
+                  emit:

Subworkflow for organelles #18

Subworkflow for organelles #18

Conversation

ksenia-krasheninnikova commented Jun 29, 2023

PR checklist

github-actions bot commented Jun 29, 2023 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

priyanka-surana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

priyanka-surana left a comment

Choose a reason for hiding this comment

ksenia-krasheninnikova commented Oct 17, 2023

priyanka-surana left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

muffato commented Oct 17, 2023

ksenia-krasheninnikova commented Oct 20, 2023 • edited Loading

muffato left a comment

Choose a reason for hiding this comment

ksenia-krasheninnikova commented Oct 30, 2023

muffato left a comment

Choose a reason for hiding this comment

github-actions bot commented Jun 29, 2023 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️

ksenia-krasheninnikova commented Oct 20, 2023 •

edited

Loading