Trio #55

yumisims · 2024-09-23T20:10:18Z

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

github-actions · 2024-09-23T20:12:51Z

`nf-core lint` overall result: Passed ✅ ⚠️

Posted for pipeline commit a6519c6

+| ✅ 130 tests passed       |+
#| ❔  19 tests were ignored |#
!| ❗   3 tests had warnings |!

❗ Test warnings:

files_exist - File not found: conf/igenomes.config
pipeline_todos - TODO string in base.config: Check the defaults for all processes
pipeline_todos - TODO string in base.config: Customise requirements for specific processes.

❔ Tests ignored:

files_exist - File is ignored: assets/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_light.png
files_exist - File is ignored: docs/images/nf-core-genomeassembly_logo_dark.png
files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
files_exist - File is ignored: .github/workflows/awstest.yml
files_exist - File is ignored: .github/workflows/awsfulltest.yml
nextflow_config - Config variable ignored: manifest.name
nextflow_config - Config variable ignored: manifest.homePage
files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_unchanged - File ignored due to lint config: .github/CONTRIBUTING.md
files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
files_unchanged - File does not exist: assets/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_light.png
files_unchanged - File does not exist: docs/images/nf-core-genomeassembly_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/genomeassembly/genomeassembly/.github/workflows/awstest.yml

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: conf/modules.config
files_exist - File found: conf/test.config
files_exist - File found: conf/test_full.config
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: lib/WorkflowGenomeassembly.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-genomeassembly_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '0.10.1dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - lib/NfcoreSchema.groovy matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
readme - README Zenodo placeholder was replaced with DOI.
pipeline_name_conventions - Name adheres to nf-core convention
template_strings - Did not find any Jinja template strings (262 files)
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/yaml'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: sanger_test_full.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: branch.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: sanger_test.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2025-03-17 13:16:36

ksenia-krasheninnikova · 2024-09-26T12:46:37Z

Hi @yumisims
Looks good to me in general, thanks for your work! What has to be changed - we don't have to run purge_dups when hifiasm is run in the trio mode. The logic is similar to the the hap1/hap2 assembly - each of the two hifiasm trio-phased files is to be scaffolded up directly:

genomeassembly/workflows/genomeassembly.nf

Lines 388 to 430 in 36730fc

    
           if ( hifiasm_hic_on ) { 
        
               // 
        
               // SUBWORKFLOW: MAP HIC DATA TO THE HAP1 CONTIGS 
        
               // 
        
               HIC_MAPPING_HAP1 ( RAW_ASSEMBLY.out.hap1_hic_contigs, crams_ch, hic_aligner_ch, 'hap1' ) 
        
               ch_versions = ch_versions.mix(HIC_MAPPING_HAP1.out.versions) 
        
               // 
        
               // SUBWORKFLOW: SCAFFOLD HAP1 
        
               // 
        
               SCAFFOLDING_HAP1( HIC_MAPPING_HAP1.out.bed, RAW_ASSEMBLY.out.hap1_hic_contigs, cool_bin, 'hap1' ) 
        
               ch_versions = ch_versions.mix(SCAFFOLDING_HAP1.out.versions) 
        
               // 
        
               // SUBWORKFLOW: MAP HIC DATA TO THE HAP2 CONTIGS 
        
               // 
        
               HIC_MAPPING_HAP2 ( RAW_ASSEMBLY.out.hap2_hic_contigs, crams_ch, hic_aligner_ch, 'hap2' ) 
        
               ch_versions = ch_versions.mix(HIC_MAPPING_HAP2.out.versions) 
        
               // 
        
               // SUBWORKFLOW: SCAFFOLD HAP2 
        
               // 
        
               SCAFFOLDING_HAP2( HIC_MAPPING_HAP2.out.bed, RAW_ASSEMBLY.out.hap2_hic_contigs, cool_bin, 'hap2' ) 
        
               ch_versions = ch_versions.mix(SCAFFOLDING_HAP2.out.versions) 
        
               // 
        
               // LOGIC: CREATE A CHANNEL FOR THE FULL HAP1/HAP2 ASSEMBLY 
        
               // 
        
               SCAFFOLDING_HAP1.out.fasta.combine(SCAFFOLDING_HAP2.out.fasta) 
        
                           .map{meta_s, fasta_s, meta_h, fasta_h -> [ [id:meta_h.id], fasta_s, fasta_h ]} 
        
                           .set{ stats_haps_input_ch } 
        
               // 
        
               // SUBWORKFLOW: CALCULATE ASSEMBLY STATISTICS FOR HAP1/HAP2 ASSEMBLY 
        
               // 
        
               GENOME_STATISTICS_SCAFFOLDS_HAPS( stats_haps_input_ch,  
        
                              PREPARE_INPUT.out.busco, 
        
                              GENOMESCOPE_MODEL.out.hist, 
        
                              GENOMESCOPE_MODEL.out.ktab, 
        
                              [], 
        
                              [], 
        
                              set_busco_alts 
        
           )

yumisims · 2024-11-05T14:53:02Z

Hi @yumisims Looks good to me in general, thanks for your work! What has to be changed - we don't have to run purge_dups when hifiasm is run in the trio mode. The logic is similar to the the hap1/hap2 assembly - each of the two hifiasm trio-phased files is to be scaffolded up directly:

genomeassembly/workflows/genomeassembly.nf

Lines 388 to 430 in 36730fc

if ( hifiasm_hic_on ) {

//

// SUBWORKFLOW: MAP HIC DATA TO THE HAP1 CONTIGS

//

HIC_MAPPING_HAP1 ( RAW_ASSEMBLY.out.hap1_hic_contigs, crams_ch, hic_aligner_ch, 'hap1' )

ch_versions = ch_versions.mix(HIC_MAPPING_HAP1.out.versions)

//

// SUBWORKFLOW: SCAFFOLD HAP1

//

SCAFFOLDING_HAP1( HIC_MAPPING_HAP1.out.bed, RAW_ASSEMBLY.out.hap1_hic_contigs, cool_bin, 'hap1' )

ch_versions = ch_versions.mix(SCAFFOLDING_HAP1.out.versions)

//

// SUBWORKFLOW: MAP HIC DATA TO THE HAP2 CONTIGS

//

HIC_MAPPING_HAP2 ( RAW_ASSEMBLY.out.hap2_hic_contigs, crams_ch, hic_aligner_ch, 'hap2' )

ch_versions = ch_versions.mix(HIC_MAPPING_HAP2.out.versions)

//

// SUBWORKFLOW: SCAFFOLD HAP2

//

SCAFFOLDING_HAP2( HIC_MAPPING_HAP2.out.bed, RAW_ASSEMBLY.out.hap2_hic_contigs, cool_bin, 'hap2' )

ch_versions = ch_versions.mix(SCAFFOLDING_HAP2.out.versions)

//

// LOGIC: CREATE A CHANNEL FOR THE FULL HAP1/HAP2 ASSEMBLY

//

SCAFFOLDING_HAP1.out.fasta.combine(SCAFFOLDING_HAP2.out.fasta)

.map{meta_s, fasta_s, meta_h, fasta_h -> [ [id:meta_h.id], fasta_s, fasta_h ]}

.set{ stats_haps_input_ch }

//

// SUBWORKFLOW: CALCULATE ASSEMBLY STATISTICS FOR HAP1/HAP2 ASSEMBLY

//

GENOME_STATISTICS_SCAFFOLDS_HAPS( stats_haps_input_ch,

PREPARE_INPUT.out.busco,

GENOMESCOPE_MODEL.out.hist,

GENOMESCOPE_MODEL.out.ktab,

[],

[],

set_busco_alts

)

Hi Ksenia, could you please take a look it again? I have add in scaffolding for trio case. Thank you

yumisims · 2024-11-05T14:54:06Z

@gq1 the same error appear again, could you please take a look at the editorconfig? thanks

gq1 · 2024-11-05T16:07:00Z

@gq1 the same error appear again, could you please take a look at the editorconfig? thanks

https://github.com/sanger-tol/genomeassembly/actions/runs/11686036439/workflow?pr=55#L22
You need to use the old version as other pipelines you did? not the latest version.

ksenia-krasheninnikova · 2024-11-06T11:08:44Z

Hi Yumi

Thank you for the updates! There are some considerations:

hifiasm_trio_on should be set to false in base.conf - similar how it's implemented for hifiasm_hic_on
the test based on test.yaml runs incomplete. The output files of the hifiasm in trio mode are not picked up because the hifiasm module refers to the wrong names of the gfa files: it has to be ".asm.dip.hap1.p_ctg.gfa" not ".asm.hic.hap1.p_ctg.gfa" etc (you can find them in the hifiasm run in the nextflow workdir). Hence the jobs gfa_to_fasta, assembly stats/busco/merqury and scaffolding are not picked up by nextflow.
purge_dups should not be run on an assembly with trio data but these jobs are being scheduled
If hifiasm was run in trio mode it has to be visible from it's name: baUndUnlc1.hifiasm-trio.20241106
I suggest to move the trio mode in a separate test rather than keeping it in test.conf

yumisims · 2024-11-13T15:49:42Z

@ksenia-krasheninnikova Could you please have a look at the pr again? thanks.
I have added a separate config for trio mode, and make sure no trio mode in the test.config. Please let me know.
Thanks

ksenia-krasheninnikova · 2025-01-06T11:27:15Z

Thanks @yumisims
I can see in the latest commit trio mode is switched off by default.

The primary assembly is still incomplete in the test mode.
I suggest to move the trio mode to a separate test case which means there has to be a specific test file created for the trio mode. There are paths to trio data in test.yaml which is not needed anymore if this functionality is not tested. I can see multiple tests configuration files were updated by adding trio files for sarscov but that wouldn't work for testing because the HiFi datasets in parents/offspring won't match.
Among the new changes - the naming of the output folder has to be changed as suggested in the previous post, because currently it refers to hifiasm-hic folder output. For this reason it's not possible to run the pipeline with --hifiasm_trio_on True as it fails with an error.

prototaxites

Hi Yumi,

Have had a look over - I think this all looks fine! There are things in the code that I think could be tidied, but on the whole I think this can happen in a bigger refactoring of the codebase. I've left a few comments though, which probably aren't very arduous.

That said, I'm still not too familiar with the codebase, so I'll wait for @ksenia-krasheninnikova to give an OK. In the meantime can you share either a path to the output of a test run, or a command I could run to run a trio test?

assets/test_trio.yaml

conf/modules.config

modules/nf-core/merquryfk/merquryfk/main.nf

subworkflows/local/raw_assembly.nf

modules/local/hapmaker.nf

prototaxites · 2025-03-07T11:03:20Z

modules/nf-core/merquryfk/merquryfk/main.nf


    when:
    task.ext.when == null || task.ext.when

    script:
+    // Exit if running this module with -profile conda / -profile mamba
+    if (workflow.profile.tokenize(',').intersect(['conda', 'mamba']).size() >= 1) {


Not something to fix in this PR but there is a bioconda package for MerquryFK now

Oh nice ! In fact, it's not just the fact that there's a bioconda package, it's importantly that there are now releases https://github.com/thegenemyers/MERQURY.FK/releases !

yumisims · 2025-03-08T18:50:17Z

@prototaxites, Hi Jim, thanks for the commit, I will do some cleaning up based on your comment. My next step is to refactor three nf-core modules and write up the hapmaker module to submit it to nf-core (see issues created).

ksenia-krasheninnikova · 2025-03-10T16:12:23Z

assets/test_gsMetZobe1.yaml

+    matreads:
+      - https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/fastq/test_1.fastq.gz
+    patreads:
+      - https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/fastq/test_2.fastq.gz


As discussed, there is a separate trio test implemented. The 'trio:' section has to be removed here.

ok, thanks both @ksenia-krasheninnikova @prototaxites I will incorperate all your comments.

assets/test_gfLaeSulp1.yaml

ksenia-krasheninnikova · 2025-03-10T16:13:00Z

assets/test_iyVesGerm1.yaml

+    matreads:
+      - https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/fastq/test_1.fastq.gz
+    patreads:
+      - https://raw.githubusercontent.com/nf-core/test-datasets/modules/data/genomics/sarscov2/illumina/fastq/test_2.fastq.gz


As discussed, there is a separate trio test implemented. The 'trio:' section has to be removed here.

yumisims added 3 commits September 16, 2024 15:58

trio mode

067cd1f

updates

f07e8bd

add trio mode

1f37e38

yumisims requested review from ksenia-krasheninnikova and gq1 September 23, 2024 20:10

yumisims added 6 commits September 23, 2024 21:18

revert hifiasm_hic

4ff0e7c

map ktabs

ea18322

map ktabs

60d5fc1

map ktabs

ee46527

map ktabs

47ccc76

add version for trio process

36730fc

yumisims added 2 commits November 5, 2024 14:10

scaffolding trio assembly

47e93c2

scaffolding trio assembly

392a454

upgrade editconfig

7383927

yumisims added 2 commits November 13, 2024 14:53

re-adjusted configuration for trio mode

35299c8

add boolen in schema

fac5b0a

yumisims added 5 commits March 3, 2025 14:46

changes for trio mode

cab7050

corrected merqury arguments

1facb3e

fixed linting

64bdad6

create patches for modules

be64b6e

prettier

560d577

yumisims requested a review from prototaxites March 3, 2025 20:59

avoid purging in trio mode

ad09059

prototaxites reviewed Mar 7, 2025

View reviewed changes

ksenia-krasheninnikova reviewed Mar 10, 2025

View reviewed changes

assets/test_gfLaeSulp1.yaml Outdated Show resolved Hide resolved

ksenia-krasheninnikova reviewed Mar 10, 2025

View reviewed changes

yumisims and others added 7 commits March 14, 2025 12:03

fixed pr

ca85627

changed hapermaker module name

581165d

fixed yaml

9507d99

adjusted yaml

a62fef3

prettier

db2a099

Merge branch 'dev' into trio

4e02638

Fix inputs to Mitohifi modules to remove meta

a6519c6

prototaxites approved these changes Mar 18, 2025

View reviewed changes

prototaxites merged commit 0c2aae8 into dev Mar 20, 2025
6 checks passed

prototaxites deleted the trio branch March 27, 2025 10:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Trio #55

Trio #55

yumisims commented Sep 23, 2024 •

edited

Loading

github-actions bot commented Sep 23, 2024 •

edited

Loading

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

ksenia-krasheninnikova commented Sep 26, 2024

yumisims commented Nov 5, 2024

yumisims commented Nov 5, 2024

gq1 commented Nov 5, 2024

ksenia-krasheninnikova commented Nov 6, 2024

yumisims commented Nov 13, 2024

ksenia-krasheninnikova commented Jan 6, 2025

prototaxites left a comment

prototaxites Mar 7, 2025

muffato Mar 7, 2025

yumisims commented Mar 8, 2025

ksenia-krasheninnikova Mar 10, 2025

yumisims Mar 10, 2025

ksenia-krasheninnikova Mar 10, 2025

Trio #55

Trio #55

Conversation

yumisims commented Sep 23, 2024 • edited Loading

PR checklist

github-actions bot commented Sep 23, 2024 • edited Loading

nf-core lint overall result: Passed ✅ ⚠️

❗ Test warnings:

❔ Tests ignored:

✅ Tests passed:

Run details

ksenia-krasheninnikova commented Sep 26, 2024

yumisims commented Nov 5, 2024

yumisims commented Nov 5, 2024

gq1 commented Nov 5, 2024

ksenia-krasheninnikova commented Nov 6, 2024

yumisims commented Nov 13, 2024

ksenia-krasheninnikova commented Jan 6, 2025

prototaxites left a comment

Choose a reason for hiding this comment

prototaxites Mar 7, 2025

Choose a reason for hiding this comment

muffato Mar 7, 2025

Choose a reason for hiding this comment

yumisims commented Mar 8, 2025

ksenia-krasheninnikova Mar 10, 2025

Choose a reason for hiding this comment

yumisims Mar 10, 2025

Choose a reason for hiding this comment

ksenia-krasheninnikova Mar 10, 2025

Choose a reason for hiding this comment

yumisims commented Sep 23, 2024 •

edited

Loading

github-actions bot commented Sep 23, 2024 •

edited

Loading

`nf-core lint` overall result: Passed ✅ ⚠️