Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Download pipelines for Tower #2247

Merged
merged 43 commits into from
Jun 1, 2023
Merged
Changes from 1 commit
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
f9d07ef
Add -t / --tower option to 'nf-core download'.
MatthiasZepper Oct 5, 2022
e94dce0
Intermediate commit
MatthiasZepper Dec 5, 2022
cba336b
Implement logic for the Tower download in DownloadWorkflow:download_w…
MatthiasZepper Feb 15, 2023
15c0588
Extend ModulesRepo:setup_local_repo() with a cache_only bool, so we c…
MatthiasZepper Feb 15, 2023
2be88c4
Create WorkflowRepo subclass of ModuleRepo and initialise local clone.
MatthiasZepper Feb 15, 2023
d72169d
TypeError: HEAD is a detached symbolic reference as it points to ...
MatthiasZepper Feb 15, 2023
a6b3492
Split history ./modules/modules_repo.py to synced_repo.py
MatthiasZepper Feb 21, 2023
8984a63
Split history ./modules/modules_repo.py to synced_repo.py
MatthiasZepper Feb 21, 2023
33d0381
Split history ./modules/modules_repo.py to synced_repo.py
MatthiasZepper Feb 21, 2023
caef187
Duplication of ModulesRepo to SyncedRepo done.
MatthiasZepper Feb 21, 2023
2367ae7
Strip ModulesRepo class of the methods moved to new superclass.
MatthiasZepper Feb 21, 2023
f5f0df2
Rebase to current dev.
MatthiasZepper Feb 21, 2023
f852159
Local caching of the repo works now.
MatthiasZepper Feb 21, 2023
af4754e
Started implementing the config download.
MatthiasZepper Feb 22, 2023
3bc97c5
Started to implement the multiple revision selection for the Tower do…
MatthiasZepper Feb 24, 2023
e17a8e9
Rewrite get_revision_hash() function to accomodate multiple revisions.
MatthiasZepper Feb 28, 2023
ecaabf8
The 2nd revivial of the config choice. Now available for archives wit…
MatthiasZepper Feb 28, 2023
6d04ec8
Inclusion of the revision in the output file name is problematic with…
MatthiasZepper Mar 4, 2023
7642e4f
Allow multiple instances of the -r argument. Needed for scripted down…
MatthiasZepper Mar 8, 2023
7f93edb
Finished updating the prompts for the dialogues.
MatthiasZepper Mar 28, 2023
12bf942
Converted the self.wf_download_url into a dict.
MatthiasZepper Apr 13, 2023
2ff62f3
Enable multi-revision classic download.
MatthiasZepper Apr 14, 2023
986f791
Small tweaks to ensure that tools doesn't bail out if there is no sym…
MatthiasZepper Apr 17, 2023
6f95829
Initialise the Git repo clone of the workflow.
MatthiasZepper Apr 18, 2023
760fcaa
WorkflowRepo attributes and functions.
MatthiasZepper Apr 19, 2023
c381776
Finished the Tower download branch.
MatthiasZepper Apr 21, 2023
526a26e
Minor tweaks to the container download functionality.
MatthiasZepper Apr 24, 2023
f4b9e67
Updating docs and changelog, fixing linting errors.
MatthiasZepper Apr 24, 2023
2bf14bd
Hopefully fixed the existing tests. New ones still need to be written.
MatthiasZepper Apr 24, 2023
8de588e
Refactor the CLI commands for the Singularity Cache Dir
MatthiasZepper Apr 25, 2023
d729bde
Readme updates for the new remote Singularity cache feature.
MatthiasZepper Apr 26, 2023
0f58c29
Add interactive check in retry for parsing the index.
MatthiasZepper Apr 27, 2023
6294d74
Incorporating some suggestions by @mashehu.
MatthiasZepper Apr 27, 2023
8d327a4
Apply suggestions from code review @mashehu
MatthiasZepper Apr 27, 2023
340c519
Writing additional tests for the --tower download functionality.
MatthiasZepper Apr 27, 2023
f599237
Move alterations from Version 2.8 (which this PR didn't make anymore)…
MatthiasZepper May 2, 2023
2518a4b
Adding the info about remote containers to the summary log rather tha…
MatthiasZepper May 2, 2023
4f390be
Moved the notification about remote containers to summary_log.
MatthiasZepper May 5, 2023
f8e5068
Apply suggestions from code review
MatthiasZepper May 9, 2023
e512878
Fixes suggested by @mirpedrol during review. Thanks!
MatthiasZepper May 9, 2023
315b9a3
@mashehu suggested that downloading the containers should not be opti…
MatthiasZepper May 9, 2023
6a806ee
Bugfix: WorkflowRepo.tidy_tags() did indeed only tidy tags. However, …
MatthiasZepper May 26, 2023
259afa6
Merge branch 'dev' into DownloadForTower
MatthiasZepper May 31, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Incorporating some suggestions by @mashehu.
MatthiasZepper committed May 25, 2023
commit 6294d74559cabebad29d6cb08d3aed6b4c57756b
20 changes: 10 additions & 10 deletions nf_core/download.py
Original file line number Diff line number Diff line change
@@ -148,8 +148,7 @@ def download_workflow(self):
self.container = "singularity"
self.prompt_singularity_cachedir_creation()
self.prompt_singularity_cachedir_utilization()
self.prompt_singularity_cachedir_remote(retry=False)
self.read_remote_containers()
self.prompt_singularity_cachedir_remote()
# Nothing meaningful to compress here.
if not self.tower:
self.prompt_compression_type()
@@ -177,8 +176,8 @@ def download_workflow(self):
if not self.tower:
# Only show entry, if option was prompted.
summary_log.append(f"Include default institutional configuration: '{self.include_configs}'")

summary_log.append(f"Enabled for seqeralabs® Nextflow Tower: '{self.tower}'")
else:
summary_log.append(f"Enabled for seqeralabs® Nextflow Tower: '{self.tower}'")

# Check that the outdir doesn't already exist
if os.path.exists(self.outdir):
@@ -203,9 +202,9 @@ def download_workflow(self):
if self.tower:
self.download_workflow_tower()
else:
self.download_workflow_classic()
self.download_workflow_static()

def download_workflow_classic(self):
def download_workflow_static(self):
"""Downloads a nf-core workflow from GitHub to the local file system in a self-contained manner."""

# Download the centralised configs first
@@ -290,7 +289,8 @@ def prompt_revision(self):
Prompt for pipeline revision / branch
Prompt user for revision tag if '--revision' was not set
If --tower is specified, allow to select multiple revisions
Also the classic download allows for multiple revisions, but
Also the static download allows for multiple revisions, but
we do not prompt this option interactively.
"""
if not bool(self.revision):
(choice, tag_set) = nf_core.utils.prompt_pipeline_release_branch(
@@ -459,7 +459,7 @@ def prompt_singularity_cachedir_utilization(self):
style=nf_core.utils.nfcore_question_style,
).unsafe_ask()

def prompt_singularity_cachedir_remote(self, retry):
def prompt_singularity_cachedir_remote(self):
"""Prompt about the index of a remote $NXF_SINGULARITY_CACHEDIR"""
if (
self.container == "singularity"
@@ -485,8 +485,8 @@ def prompt_singularity_cachedir_remote(self, retry):
cachedir_index = None
if cachedir_index:
self.singularity_cache_index = cachedir_index
if retry: # invoke parsing the file again.
self.read_remote_containers()
# in any case read the remote containers, even if no prompt was shown.
self.read_remote_containers()

def read_remote_containers(self):
"""Reads the file specified as index for the remote Singularity cache dir"""