Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: add asynchronous cycle which can deal with long-running evaluations #238

Closed
hollandjg opened this issue Jan 13, 2023 · 4 comments
Closed
Assignees
Labels
enhancement New feature or request. priority 1 - needed These are highly desirable to be fixed, ideally within 2 weeks.

Comments

@hollandjg
Copy link
Member

hollandjg commented Jan 13, 2023

... and if everything crashes, doesn't completely lose its data.

It could be:

  • a message queue
  • a lightweight program which can run on any machine
  • a lightweight program which runs as a cron-job

It needs to be able to:

@hollandjg
Copy link
Member Author

Our current approach relies on the Controller and Cycle objects, and running within python. We've demonstrated saving and loading (though it's not yet in a PR which is ready to go), as well as being able to handle many tasks like seeding and different processes depending on the current state within the framework of the Controller.

⛔️ However, we still don't have robust SLURM dispatch, which is a standard problem for data processing pipelines. It turns out that this is hard.

We should switch gears and look at pipeline tools. Can we set up the cycle to run under:

  • cylc?
  • snakemake?
  • make?

@hollandjg
Copy link
Member Author

We have a working version of the workflow module which does this. Open:

  • Thorough testing
  • more documentation

@hollandjg hollandjg self-assigned this Jun 20, 2023
@gtdang gtdang removed the full cycle label Jul 18, 2023
@gtdang gtdang added the priority 1 - needed These are highly desirable to be fixed, ideally within 2 weeks. label Aug 15, 2023
@hollandjg
Copy link
Member Author

This is implemented with autora-workflow version 0.5.0 with the State mechanism. Needs testing in the real world.

@hollandjg
Copy link
Member Author

This is now possible using cylc and the workflow package. Hurrah!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request. priority 1 - needed These are highly desirable to be fixed, ideally within 2 weeks.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants