Create SharedArray from disk file #12560
Conversation
Also use #10525 for indexing operations
This mmaps the file on all workers
| workermode = mode == "w+" ? "r+" : mode # workers don't truncate! | ||
|
|
||
| # Ensure the file will be readable | ||
| mode in ("r", "r+", "w+", "a+") || error("mode must be readable, but I got $mode") |
There was a problem hiding this comment.
can these be ArgumentError's? Also, using a personal pronoun in an error message is a bit weird.
There was a problem hiding this comment.
Good points all round, will fix.
|
OK, the remaining AppVeyor error seems unrelated ( |
|
Where are the throws coming from? If it's really the same condition then it would make sense to me to make it the same exception type if that's not much work. |
Create SharedArray from disk file
|
|
||
| # Mapping an existing file | ||
| fn = tempname() | ||
| open(fn, "w") do io |
There was a problem hiding this comment.
these should probably clean up after themselves when they're done
There was a problem hiding this comment.
Fixed in 13846e5
The third commit here allows you to share data from a pre-existing disk file (or a newly-created one) across all workers.
The first commit is just a cleanup (and net code deletion): it leverages #10525 for indexing, and makes all fields of a SharedArray concrete (I think the important ones were already). The second is #12423 without the documentation changes (because docs can always be added later, and we're in a bit of doc-paralysis at the moment). That commit should still be good even after this merges.
CC @amitmurthy