Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example: extending workflow #5947

Draft
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

oliver-sanders
Copy link
Member

@oliver-sanders oliver-sanders commented Jan 30, 2024

Document the preferred approach to "extending" a workflow.

Closes: cylc/cylc-doc#525
Partially addresses: #5875

Context:

A common pattern in some areas is to run a workflow until it stops. Then modify the final cycle point to extend the workflow and restart.

This isn't an easy working pattern to port to Cylc 8 due to the difficulty of re-populating the start tasks. We have a plan to make this easier, see #5416, however, this working pattern is needlessly complex in the first place. It would be much simpler if the workflow didn't "finish" in the first place.

To do this, swap final cycle point for stop after cycle point, now the workflow will not "finish" so the pool will not need re-polulating for the workflow to continue. This is much nicer for Cylc (no discontinuity), the user (no extra commands) and provenance (direct continuation of previous run).

Blockers.

Check List

  • I have read CONTRIBUTING.md and added my name as a Code Contributor.
  • Contains logically grouped changes (else tidy your branch by rebase).
  • Does not contain off-topic changes (use other PRs for other changes).
  • Applied any dependency changes to both setup.cfg (and conda-environment.yml if present).
  • Tests are included (or explain why tests are not needed).
  • CHANGES.md - docs only
  • Cylc-Doc pull request opened if required at cylc/cylc-doc/pull/XXXX.
  • If this is a bug fix, PR should be raised against the relevant ?.?.x branch.

@oliver-sanders oliver-sanders added doc Documentation BLOCKED This can't happen until something else does labels Jan 30, 2024
@oliver-sanders oliver-sanders added this to the cylc-8.3.0 milestone Jan 30, 2024
@oliver-sanders oliver-sanders self-assigned this Jan 30, 2024
@hjoliver
Copy link
Member

hjoliver commented Feb 15, 2024

Agreed on documenting use of a stop point rather than final cycle point, to allow extending a workflow run more easily.

But, a stop point requires trickier graph config if any graph structure is tied to the stop point, and users may not know in advance that they will want to extend their run. So I guess we'll still have to document how to do it the hard way as well, i.e., extending the final point.

As revealed by the discussion on #5952, it will be crucial to document that changing a final cycle point, or a stop point if the graph is tied to it, can implicitly change the structure of the graph, with potential consequences for the restart.

The simple example in that issue can be handled automatically I think, but I'm not sure that's universal.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Feb 15, 2024

So I guess we'll still have to document how to do it the hard way as well

We'd rather not, the FCP approach is conceptually awkward and are keen to move users away from it.

We've got a lot of users who rely on this pattern which was very simple at Cylc 7 but is very difficult at Cylc 8, we need to provide a simple solution for this use case.

@hjoliver
Copy link
Member

hjoliver commented Feb 16, 2024

We'd rather not, the FCP approach is conceptually awkward and are keen to move users away from it.

That's all very well, but it's going to happen anyway, as in "Help, my workflow finished, how do I move the FCP and continue it",

this pattern which was very simple at Cylc 7 but is very difficult at Cylc 8,

Hmm, moving the FCP was always a bit ill-defined and dangerous, even if it did work most of the time.

It's not so difficult now: restart with --pause and trigger the first cycle after the original FCP.

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Feb 16, 2024

It's not so difficult now: restart with --pause and trigger the first cycle after the original FCP.

It really is, I find it hard! The triggering is often non-trivial and requires inspection of the workflow configuration, we cannot presently provide generic advice to do this or a single command like we used to have.

@hjoliver
Copy link
Member

hjoliver commented Mar 3, 2024

We'd rather not, the FCP approach is conceptually awkward and are keen to move users away from it.

That's all very well, but it's going to happen anyway, as in "Help, my workflow finished, how do it move the FCP and continue it",

Case in point https://cylc.discourse.group/t/extending-and-restarting-a-a-workflow/911

@oliver-sanders
Copy link
Member Author

oliver-sanders commented Mar 4, 2024

True but unhelpful, we would like to avoid using this pattern for workflows which are intended to be extended. Stop after cycle point is a very clean solution which cuts out the discontinuity between runs completely.

Will come back to this one and the associated issues when I get the chance.

@hjoliver
Copy link
Member

I don't think we're disagreeing on the fundamental point here.

I'm just saying in addition to that, we still need to document how to restart after extending the FCP if you do get yourself into that particular fix.

Recommending a better way isn't much help if users don't anticipate the need for extending a workflow, or they don't see the advice, before starting a run.

@hjoliver hjoliver modified the milestones: cylc-8.3.0, cylc-8.4.0 Mar 14, 2024
@oliver-sanders oliver-sanders mentioned this pull request Jul 11, 2024
8 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BLOCKED This can't happen until something else does doc Documentation
Projects
None yet
Development

Successfully merging this pull request may close these issues.

How to "continue" a completed workflow
2 participants