Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pipeline configuration with a Yaml file #80

Closed
wants to merge 6 commits into from
Closed

Pipeline configuration with a Yaml file #80

wants to merge 6 commits into from

Conversation

muffato
Copy link
Member

@muffato muffato commented Jul 12, 2023

Hi,

I've been wondering a lot how to reconcile the parameters that the Nextflow processes use (typically by setting params.* and ext.args) and the BTK Yaml file. To avoid redundancy, I think there should be a single source.

Here is a proof on concept for using a BTK-style Yaml file as the main configuration mechanism (through Nextflow's native -params-file), from which the pipeline takes all its parameters by setting ext.args.

This is a draft PR and should remain that way for now. The unit test seems to work, but I've removed some parameter validation and the code would likely crash if some key parameters are missing.
I didn't try defining the reads in the Yaml file, so there is still a CSV sample sheet, but that would be the next natural step. Then, to complete the Yaml file that goes into the blobdir, we'd still have to add the software versions (can be done by adding a process that takes ch_versions), and some other genome metadata stuff I think.

It's just one way of doing. Alternatively the config file could be given through an option like --btk-config /path/to/yaml which is parsed and flown into the jobs through meta ?

Comments welcome 😄

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • Make sure your code lints (nf-core lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@muffato muffato added the question Further information is requested label Jul 12, 2023
@github-actions
Copy link

github-actions bot commented Jul 12, 2023 •
edited
Loading

nf-core lint overall result: Failed ❌

Posted for pipeline commit 36cd215

+| ✅ 128 tests passed       |+
#| ❔  18 tests were ignored |#
!| ❗   4 tests had warnings |!
-| ❌   6 tests failed       |-

❌ Test failures:

  • schema_params - Param fasta from nextflow config not found in nextflow_schema.json
  • schema_params - Param accession from nextflow config not found in nextflow_schema.json
  • schema_params - Param taxon from nextflow config not found in nextflow_schema.json
  • schema_params - Param taxa_file from nextflow config not found in nextflow_schema.json
  • schema_params - Param taxdump from nextflow config not found in nextflow_schema.json
  • schema_params - Param busco from nextflow config not found in nextflow_schema.json

❗ Test warnings:

  • files_exist - File not found: conf/igenomes.config
  • files_exist - File not found: .github/workflows/awstest.yml
  • files_exist - File not found: .github/workflows/awsfulltest.yml
  • nextflow_config - Config manifest.version should end in dev: '0.2.0'

❔ Tests ignored:

  • files_exist - File is ignored: assets/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_exist - File is ignored: docs/images/nf-core-blobtoolkit_logo_dark.png
  • files_exist - File is ignored: .github/ISSUE_TEMPLATE/config.yml
  • nextflow_config - Config variable ignored: manifest.name
  • nextflow_config - Config variable ignored: manifest.homePage
  • files_unchanged - File ignored due to lint config: LICENSE or LICENSE.md or LICENCE or LICENCE.md
  • files_unchanged - File ignored due to lint config: .github/ISSUE_TEMPLATE/bug_report.yml
  • files_unchanged - File does not exist: .github/ISSUE_TEMPLATE/config.yml
  • files_unchanged - File ignored due to lint config: .github/workflows/linting.yml
  • files_unchanged - File ignored due to lint config: assets/sendmail_template.txt
  • files_unchanged - File does not exist: assets/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-blobtoolkit_logo_light.png
  • files_unchanged - File does not exist: docs/images/nf-core-blobtoolkit_logo_dark.png
  • files_unchanged - File ignored due to lint config: lib/NfcoreTemplate.groovy
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore or pyproject.toml
  • actions_ci - actions_ci
  • actions_awstest - 'awstest.yml' workflow not found: /home/runner/work/blobtoolkit/blobtoolkit/.github/workflows/awstest.yml

✅ Tests passed:

Run details

  • nf-core/tools version 2.8
  • Run at 2023-07-12 12:35:16

@priyanka-surana priyanka-surana linked an issue Jul 12, 2023 that may be closed by this pull request
Improved generation of the summary Yaml file #77
Closed
@priyanka-surana priyanka-surana changed the base branch from release to alignment September 18, 2023 07:16
@priyanka-surana
Copy link
Contributor

@muffato Should we attempt to merge this into #83 for testing?

@muffato
Copy link
Member Author

muffato commented Sep 18, 2023

I think there's too much to do for this config file to work. I'd rather merge #83 first, and then work on the config file properly.

Base automatically changed from alignment to release October 18, 2023 09:42
@priyanka-surana priyanka-surana deleted the branch release December 22, 2023 09:46
@muffato muffato deleted the config_params_file branch May 20, 2024 09:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
question Further information is requested
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Improved generation of the summary Yaml file
2 participants
@muffato @priyanka-surana