Sentieon-haplotyper and joint-germline with Sentieon Genotyper and VQSR #1007

asp8200 · 2023-04-27T08:51:38Z

Ready for review.

The sentieon-based haplotyper-flow should be able to output vcf, gvcf and both.
joint-germline with Sentieon's Genotyper and VQSR (VarCal and ApplyVarCal).
Add Sentieon's Genotyper, VarCal and ApplyVarCal as modules to nf-core's module-repo.
pytest for using sentieon_haplotyper_skip_filter and sentieon_haplotyper_out_format
pytest for VQSR (both GATK and Sentieon). This turned out to be difficult. Created Pytest for VQSR-flow #1027 for that.
Update the Sentieon-modules so that they also accepts license-files. (See email from Don.)
cleanup

Note: The sentieon-bwamem module actually generate index-files (default behaviour, cannot be disabled) but currently Sarek recomputes the index-files, since that is what is done for the other aligners. It would be great if we could avoid the redundant index generation when Sarek uses Sentieon-bwamem for alignment. (I made a separate issue for this : #1046 )

PR checklist

This comment contains a description of changes (with reason).
If you've fixed a bug or added code that should be tested, add tests!
If you've added a new tool - have you followed the pipeline conventions in the contribution docs- [ ] If necessary, also make a PR on the nf-core/sarek branch on the nf-core/test-datasets repository.
Make sure your code lints (nf-core lint).
Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
Usage Documentation in docs/usage.md is updated.
Output Documentation in docs/output.md is updated.
CHANGELOG.md is updated.
README.md is updated (including new tool citations and authors/contributors).

…ne, cleanup and more testing.

github-actions · 2023-04-27T08:57:08Z

`nf-core lint` overall result: Passed ✅

Posted for pipeline commit cc45a80

+| ✅ 153 tests passed       |+
#| ❔   9 tests were ignored |#

❔ Tests ignored:

files_exist - File is ignored: conf/modules.config
files_exist - File is ignored: conf/test.config
files_exist - File is ignored: conf/test_full.config
files_unchanged - File ignored due to lint config: assets/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_light.png
files_unchanged - File ignored due to lint config: docs/images/nf-core-sarek_logo_dark.png
files_unchanged - File ignored due to lint config: lib/NfcoreSchema.groovy
files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
template_strings - template_strings

✅ Tests passed:

files_exist - File found: .gitattributes
files_exist - File found: .gitignore
files_exist - File found: .nf-core.yml
files_exist - File found: .editorconfig
files_exist - File found: .prettierignore
files_exist - File found: .prettierrc.yml
files_exist - File found: CHANGELOG.md
files_exist - File found: CITATIONS.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: CODE_OF_CONDUCT.md
files_exist - File found: LICENSE or LICENSE.md or LICENCE or LICENCE.md
files_exist - File found: nextflow_schema.json
files_exist - File found: nextflow.config
files_exist - File found: README.md
files_exist - File found: .github/.dockstore.yml
files_exist - File found: .github/CONTRIBUTING.md
files_exist - File found: .github/ISSUE_TEMPLATE/bug_report.yml
files_exist - File found: .github/ISSUE_TEMPLATE/config.yml
files_exist - File found: .github/ISSUE_TEMPLATE/feature_request.yml
files_exist - File found: .github/PULL_REQUEST_TEMPLATE.md
files_exist - File found: .github/workflows/branch.yml
files_exist - File found: .github/workflows/ci.yml
files_exist - File found: .github/workflows/linting_comment.yml
files_exist - File found: .github/workflows/linting.yml
files_exist - File found: assets/email_template.html
files_exist - File found: assets/email_template.txt
files_exist - File found: assets/sendmail_template.txt
files_exist - File found: assets/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_light.png
files_exist - File found: docs/images/nf-core-sarek_logo_dark.png
files_exist - File found: docs/output.md
files_exist - File found: docs/README.md
files_exist - File found: docs/README.md
files_exist - File found: docs/usage.md
files_exist - File found: lib/nfcore_external_java_deps.jar
files_exist - File found: lib/NfcoreSchema.groovy
files_exist - File found: lib/NfcoreTemplate.groovy
files_exist - File found: lib/Utils.groovy
files_exist - File found: lib/WorkflowMain.groovy
files_exist - File found: main.nf
files_exist - File found: assets/multiqc_config.yml
files_exist - File found: conf/base.config
files_exist - File found: conf/igenomes.config
files_exist - File found: .github/workflows/awstest.yml
files_exist - File found: .github/workflows/awsfulltest.yml
files_exist - File found: lib/WorkflowSarek.groovy
files_exist - File found: modules.json
files_exist - File found: pyproject.toml
files_exist - File not found check: Singularity
files_exist - File not found check: parameters.settings.json
files_exist - File not found check: .nf-core.yaml
files_exist - File not found check: bin/markdown_to_html.r
files_exist - File not found check: conf/aws.config
files_exist - File not found check: .github/workflows/push_dockerhub.yml
files_exist - File not found check: .github/ISSUE_TEMPLATE/bug_report.md
files_exist - File not found check: .github/ISSUE_TEMPLATE/feature_request.md
files_exist - File not found check: docs/images/nf-core-sarek_logo.png
files_exist - File not found check: .markdownlint.yml
files_exist - File not found check: .yamllint.yml
files_exist - File not found check: lib/Checks.groovy
files_exist - File not found check: lib/Completion.groovy
files_exist - File not found check: lib/Workflow.groovy
files_exist - File not found check: .travis.yml
nextflow_config - Config variable found: manifest.name
nextflow_config - Config variable found: manifest.nextflowVersion
nextflow_config - Config variable found: manifest.description
nextflow_config - Config variable found: manifest.version
nextflow_config - Config variable found: manifest.homePage
nextflow_config - Config variable found: timeline.enabled
nextflow_config - Config variable found: trace.enabled
nextflow_config - Config variable found: report.enabled
nextflow_config - Config variable found: dag.enabled
nextflow_config - Config variable found: process.cpus
nextflow_config - Config variable found: process.memory
nextflow_config - Config variable found: process.time
nextflow_config - Config variable found: params.outdir
nextflow_config - Config variable found: params.input
nextflow_config - Config variable found: params.show_hidden_params
nextflow_config - Config variable found: params.schema_ignore_params
nextflow_config - Config variable found: manifest.mainScript
nextflow_config - Config variable found: timeline.file
nextflow_config - Config variable found: trace.file
nextflow_config - Config variable found: report.file
nextflow_config - Config variable found: dag.file
nextflow_config - Config variable (correctly) not found: params.nf_required_version
nextflow_config - Config variable (correctly) not found: params.container
nextflow_config - Config variable (correctly) not found: params.singleEnd
nextflow_config - Config variable (correctly) not found: params.igenomesIgnore
nextflow_config - Config variable (correctly) not found: params.name
nextflow_config - Config variable (correctly) not found: params.enable_conda
nextflow_config - Config timeline.enabled had correct value: true
nextflow_config - Config report.enabled had correct value: true
nextflow_config - Config trace.enabled had correct value: true
nextflow_config - Config dag.enabled had correct value: true
nextflow_config - Config manifest.name began with nf-core/
nextflow_config - Config variable manifest.homePage began with https://github.com/nf-core/
nextflow_config - Config dag.file ended with .html
nextflow_config - Config variable manifest.nextflowVersion started with >= or !>=
nextflow_config - Config manifest.version ends in dev: '3.2dev'
nextflow_config - Config params.custom_config_version is set to master
nextflow_config - Config params.custom_config_base is set to https://raw.githubusercontent.com/nf-core/configs/master
nextflow_config - Lines for loading custom profiles found
files_unchanged - .gitattributes matches the template
files_unchanged - .prettierrc.yml matches the template
files_unchanged - CODE_OF_CONDUCT.md matches the template
files_unchanged - LICENSE matches the template
files_unchanged - .github/.dockstore.yml matches the template
files_unchanged - .github/CONTRIBUTING.md matches the template
files_unchanged - .github/ISSUE_TEMPLATE/bug_report.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/config.yml matches the template
files_unchanged - .github/ISSUE_TEMPLATE/feature_request.yml matches the template
files_unchanged - .github/PULL_REQUEST_TEMPLATE.md matches the template
files_unchanged - .github/workflows/branch.yml matches the template
files_unchanged - .github/workflows/linting_comment.yml matches the template
files_unchanged - .github/workflows/linting.yml matches the template
files_unchanged - assets/email_template.html matches the template
files_unchanged - assets/email_template.txt matches the template
files_unchanged - assets/sendmail_template.txt matches the template
files_unchanged - docs/README.md matches the template
files_unchanged - lib/nfcore_external_java_deps.jar matches the template
files_unchanged - .gitignore matches the template
files_unchanged - .prettierignore matches the template
files_unchanged - pyproject.toml matches the template
actions_ci - '.github/workflows/ci.yml' is triggered on expected events
actions_ci - '.github/workflows/ci.yml' checks minimum NF version
actions_awstest - '.github/workflows/awstest.yml' is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml is triggered correctly
actions_awsfulltest - .github/workflows/awsfulltest.yml does not use -profile test
readme - README Nextflow minimum version badge matched config. Badge: 22.10.1, Config: 22.10.1
readme - README Zenodo placeholder was replaced with DOI.
pipeline_todos - No TODO strings found
pipeline_name_conventions - Name adheres to nf-core convention
schema_lint - Schema lint passed
schema_lint - Schema title + description lint passed
schema_lint - Input mimetype lint passed: 'text/csv'
schema_params - Schema matched params returned from nextflow config
system_exit - No System.exit calls found
actions_schema_validation - Workflow validation passed: clean-up.yml
actions_schema_validation - Workflow validation passed: linting_comment.yml
actions_schema_validation - Workflow validation passed: ci.yml
actions_schema_validation - Workflow validation passed: awstest.yml
actions_schema_validation - Workflow validation passed: awsfulltest_germline.yml
actions_schema_validation - Workflow validation passed: linting.yml
actions_schema_validation - Workflow validation passed: fix-linting.yml
actions_schema_validation - Workflow validation passed: pytest-workflow.yml
actions_schema_validation - Workflow validation passed: awsfulltest.yml
actions_schema_validation - Workflow validation passed: branch.yml
merge_markers - No merge markers found in pipeline files
modules_json - Only installed modules found in modules.json
multiqc_config - 'assets/multiqc_config.yml' follows the ordering scheme of the minimally required plugins.
multiqc_config - 'assets/multiqc_config.yml' contains a matching 'report_comment'.
multiqc_config - 'assets/multiqc_config.yml' contains 'export_plots: true'.
modules_structure - modules directory structure is correct 'modules/nf-core/TOOL/SUBTOOL'

Run details

nf-core/tools version 2.8
Run at 2023-05-26 09:38:14

…md5sums.

… for Sentieon-Haplotyper

.github/workflows/pytest-workflow.yml

nextflow_schema.json

subworkflows/local/bam_joint_calling_germline_sentieon/main.nf

tests/config/tags.yml

subworkflows/local/bam_variant_calling_germline_all/main.nf

subworkflows/local/bam_variant_calling_haplotypecaller/main.nf

maxulysse

Looking good, almost ready to merge

Co-authored-by: Maxime U Garcia <[email protected]>

…ection

.github/workflows/pytest-workflow.yml

subworkflows/local/bam_variant_calling_haplotypecaller/main.nf

Co-authored-by: Maxime U Garcia <[email protected]>

…into sentieon_haplotyper

WIP: Sentieon-haplotyper vcf-flow working. TO-DO: gvcf, joint-germeli…

f15ad55

…ne, cleanup and more testing.

asp8200 requested a review from maxulysse as a code owner April 27, 2023 08:51

Merge branch 'sentieon' into sentieon_haplotyper

4d45b72

asp8200 added 26 commits April 27, 2023 12:51

Adding sentieon/dedup

c4343cf

Adding sentieon/haplotyper

0f0e5d8

Adding skip filter functionality for sentieon-haplotyper

a1803a5

Merge branch 'sentieon' into sentieon_haplotyper

88606f9

Removing md5sum checks on TsTv.qual-files. Those files have unstable …

e38ca4f

…md5sums.

Changing option skip_haplotypecaller_filter to skip_haplotyper_filter…

ecc0da1

… for Sentieon-Haplotyper

haplotypecaller -> haplotyper

395b584

WIP: joint-germline with Sentieon

b520780

Removing debug print statement

58e88f5

Adding tests for sentieon joint-germline

72cb3f0

prettier

22be85b

Removing import of unused subworkflow BAM_MERGE_INDEX_SAMTOOLS

607eb1f

Adding option sentieon_haplotyper_out_format

eecde5f

Removing conversion of sentieon_haplotyper_out_format to value channel

da38969

Adding check for gvcf-files

3cd76e3

Setting option sentieon_haplotyper_out_format to gvcf

cfa1b84

Adding some check for running joint_germline with sentieon_haplotyper

4651282

Fixing name of output gvcf-files

58e1560

Adding tests for sentieon_haplotyper_skip_filter

c3b1a29

Removing unstable md5 checks

03280a5

WIP: Setting up subworkflow for Sentieon-based joint-calling-germline

c92bc46

Removing exception for Sentieon's haplotyper

c8adea8

Adding tests for --sentieon_haplotyper_out_format gvcf

7cdf463

Creating csv for gvcf-output from Sentieon haplotyper

2389593

Clean up

4633afa

Adding pytest for both vcf and gvcf

56d3499

asp8200 force-pushed the sentieon_haplotyper branch from ae73440 to 7efeeb6 Compare May 25, 2023 11:30

Skipping singularity-based test for CI-test concatenate_vcfs

71f5b05

asp8200 changed the title ~~DRAFT: Sentieon-haplotyper and joint-germline with Sentieon Genotyper and VQSR~~ Sentieon-haplotyper and joint-germline with Sentieon Genotyper and VQSR May 25, 2023