load in parallel

[ViralBShah]

What should load do in parallel? Do we assume an NFS type filesystem? It seems to me that the right thing to do would be for the client to preprocess/compile the code and send it to the rest of the Workers.

[JeffBezanson]

It would be much easier to implement pload that sends the file everywhere so you don't need a shared FS.

[ViralBShah]

Do we really need a pload? Can't load just do that automatically, if it detects parallel mode?

[StefanKarpinski]

In general, I think that we need a simpler eval_file() primitive that just evals the contents of a file like load() originally did. Then load() needs to be a much more sophisticated mechanism on top of that which

1. Searches a system load_path variable for where the file to load actually resides.
2. Sends the file across the network to other nodes so that they can also load the file.

Is there a single node which "owns" all file system access? Or is this where we need a distributed file system?

[JeffBezanson]

Agree.

[ViralBShah]

I prefer that there is a single node that owns all the filesystem access. Until then, we assume a distributed FS available, so that we can just do

@bcast load(file)

[StefanKarpinski]

I feel like this is backwards — it makes sense to have a single node own accesses until there's a distributed filesystem to rely on. No?

[ViralBShah]

Well, most clusters we are likely to run on in early stages will probably have NFS. Also, the multi-core mode where you run multiple julia processes have the same fs (although not distributed).

[StefanKarpinski]

Maybe we should just assume that some fs will take care of this then and
just do the load on each machine without worrying about it.

On Fri, Jun 24, 2011 at 12:10 PM, ViralBShah <
[email protected]>wrote:

Well, most clusters we are likely to run on in early stages will probably
have NFS. Also, the multi-core mode where you run multiple julia processes
have the same fs (although not distributed).

Reply to this email directly or view it on GitHub:
#81 (comment)

[ViralBShah]

Except that it breaks down, when your client is your laptop, and the computation is in the cloud. But for now, we can assume that some fs will figure it out.

-viral

On Jun 24, 2011, at 9:59 PM, StefanKarpinski wrote:

Maybe we should just assume that some fs will take care of this then and
just do the load on each machine without worrying about it.

On Fri, Jun 24, 2011 at 12:10 PM, ViralBShah <
[email protected]>wrote:

Well, most clusters we are likely to run on in early stages will probably
have NFS. Also, the multi-core mode where you run multiple julia processes
have the same fs (although not distributed).

Reply to this email directly or view it on GitHub:
#81 (comment)

Reply to this email directly or view it on GitHub:
#81 (comment)

[StefanKarpinski]

Right. That's an excellent case for using the local copy and shipping it
elsewhere for evaluation. In fact, I find that completely convincing for
development work.

On Fri, Jun 24, 2011 at 12:38 PM, ViralBShah <
[email protected]>wrote:

Except that it breaks down, when your client is your laptop, and the
computation is in the cloud. But for now, we can assume that some fs will
figure it out.

-viral

On Jun 24, 2011, at 9:59 PM, StefanKarpinski wrote:

Maybe we should just assume that some fs will take care of this then and
just do the load on each machine without worrying about it.

On Fri, Jun 24, 2011 at 12:10 PM, ViralBShah <
[email protected]>wrote:

Well, most clusters we are likely to run on in early stages will
probably
have NFS. Also, the multi-core mode where you run multiple julia
processes
have the same fs (although not distributed).

Reply to this email directly or view it on GitHub:
#81 (comment)

Reply to this email directly or view it on GitHub:
#81 (comment)

Reply to this email directly or view it on GitHub:
#81 (comment)