Scheduling periodic tasks

[mivade]

I'm looking to schedule periodic tasks on workers and thus far it looks like there's not a canonical way to do this. My current thinking is to combine apscheduler and a Dask scheduler in a wrapper script, something like this:

import asyncio

from apscheduler.schedulers.background import BackgroundScheduler as APScheduler
from apscheduler.triggers.interval import IntervalTrigger

async def main():
    apscheduler = APScheduler()
    apscheduler.add_job(lambda: print("ping"), trigger=IntervalTrigger(seconds=1))
    apscheduler.start()

    async with Scheduler() as scheduler:
        print(scheduler.identity())
        await asyncio.sleep(60)

asyncio.run(main())

This works but it feels like a bit of a hack since now I have to run this script instead of just running dask-scheduler on the command line with appropriate options (aside: Scheduler itself does not appear to be documented along with the rest of the API which does not instill great confidence in using this approach). Is there a better approach?

For some more context I have a Tornado web app and want to use Dask to offload some tasks to. Currently I have just a single server process so I could kick off the periodic tasks from there. However I'd like to have the option to add additional server processes in the future so it would make more sense to have the scheduler itself launching these tasks.

[fjetter]

Scheduler itself does not appear to be documented along with

Indeed, the scheduler as a class is not documented. It is something which changes fairly frequently and I would advise caution to rely on any specifics which are not advertised otherwise as part of the API

I'm looking to schedule periodic tasks on workers and thus far it looks like there's not a canonical way to do this. My current thinking is to combine apscheduler and a Dask scheduler in a wrapper script, something like this:

I would go for a tornado PeriodicCallback which we are using in distributed all over the place.

To start this, I suggest to implement a plugin for this, which is our way to offer users a stable method to hook into what is going on underneath. There are a few hooks for specific events but if it is just about spawning a PC, you can just do this upon plugin initialisation and don't need to implement any further methods.

If the CLI preload to start the plugin does not work for you, there is also a way to do this rather dynamically, see https://distributed.dask.org/en/latest/api.html#distributed.Client.register_worker_plugin

[mivade]

To start this, I suggest to implement a plugin for this

This was eventually the conclusion I came to. For others who would find this helpful, the way I'm doing this is to launch the scheduler via the CLI with something like

dask-scheduler --preload my.plugin

I would go for a tornado PeriodicCallback which we are using in distributed all over the place.

I've used these extensively before. The main downside here is that there's only one way to specify a schedule whereas apscheduler (which can integrate with asyncio/Tornado event loops) provides more flexibility (e.g., cron-style scheduling).