Spawn background processes across HPC cluster nodes

[angel-bee2018]

Good day! 😊

Is it currently possible for users to set up a callr command such as callr::r_bg to spawn background tasks across different nodes of a compute cluster?
If not, could the existing workings behind callr::r_bg be by any chance easily applied across multiple nodes?

I've been trying to figure out how to make this potentially work on a large HPC cluster which runs on openmpi as the interface for inter-node communication. From the documentation, it seems like callr::r_bg is a wrapper for processx::process, which is itself based around calling system so it theoretically it should work right ...? however, I am not familiar enough with what callr::bg, for example, adds on top of the existing processx::process function that may make this harder or easier.

I would be interested to know any thoughts about this / suggested ways we can currently achieve the goal
cheers!

[gaborcsardi]

processx::process does not use system. More importantly the IPC in processx is a Unix socket or (a Windows named pipe), so it is not going to work across machines.