Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

stage-in/stage-out from HPC #1143

Open
kosarko opened this issue Jan 13, 2025 · 1 comment
Open

stage-in/stage-out from HPC #1143

kosarko opened this issue Jan 13, 2025 · 1 comment
Labels
NRP On hold This should be done at some point; there are some other concerns that block it from being planned
Milestone

Comments

@kosarko
Copy link
Member

kosarko commented Jan 13, 2025

We need to download data (incl restricted) into (stage-in) a computational cluster (there might be some wait time before resources are available, expect a non-interactive session)
We need to add data from computational cluster to a repository workspace/workflow/item (or create a new one). This will be one of the last steps of a computational workflow definition (ie. again non interactive).

As a first MVP one possible solution is that the user generates some sort of token(s) and uses these in the workflow (perhaps one of the modes in "Optimize uploads/downloads" might be useful).
At some later point e-infra aai might be involved.

The lifetime of the tokens is expected to be in order of days (should be configurable at least on a global level)

@kosarko kosarko added this to the 2025 milestone Jan 13, 2025
@kosarko kosarko changed the title [dtq-next] stage-in/stage-out from HPC stage-in/stage-out from HPC Jan 13, 2025
@stranak
Copy link
Member

stranak commented Jan 20, 2025

See my comment at #1151. I would wait with this for the situation with OS II AI sub-project to clarify.

@kosarko kosarko added NRP On hold This should be done at some point; there are some other concerns that block it from being planned labels Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NRP On hold This should be done at some point; there are some other concerns that block it from being planned
Projects
None yet
Development

No branches or pull requests

2 participants