Add save/load ndarray to release notes, small bug fix#2981
Add save/load ndarray to release notes, small bug fix#2981junpenglao merged 2 commits intopymc-devs:masterfrom
Conversation
pymc3/backends/ndarray.py
Outdated
There was a problem hiding this comment.
Should we ask or alert before removing directories?
There was a problem hiding this comment.
This scares me too, but my intention here is to treat self.directory as a single filesystem object that pymc3 owns. In this context, it is not unusual to overwrite without asking/alerting. I think this is fine, but I could see something ugly happening if you call pm.save_trace(trace, './'), intending to save a file to the current directory.
My best "safe" alternative that is not overly onerous is to write .pymc.manifest to the top level of the directory, with a list of all the files that pymc is allowed to delete in the future. Then this deletion will be just running over all the files in the existing manifest and deleting those. In that case, I would throw an exception then if any path already exists after deleting files in the manifest.
There was a problem hiding this comment.
Even more explicitly, by default, there would be a file ./.pymc3.trace/.pymc.manifest, whose contents are something like
0/samples.npz
0/metadata.json
1/samples.npz
1/metadata.json
...
There was a problem hiding this comment.
Alternatively, we could do add overwrite=False flag that if False and the dir exists, we raise an exception that you have to set overwrite=True.
There was a problem hiding this comment.
that's reasonable, though I always appreciate programs where save "just works". what if i followed jupyter's lead and did .pymc3_1.trace, .pymc3_2.trace, etc. by default?
There was a problem hiding this comment.
I would imagine that this saving is done multiple times for a single model, so overwriting is probably a useful feature.
There was a problem hiding this comment.
sorry, i agree with that! I am suggesting that the signature be
def save_trace(trace, filename=None, overwrite=False):
if filename is None:
# find a unique filename to write to
else:
if filename exists:
if overwrite is False:
raise Exception
else:
# clear out directory
432fe3f to
d50e200
Compare
|
I think this is ready to go now. |
Added a test and a fix for a bug in
save/load_trace, and added a note to RELEASE_NOTES.md