Skip to content

Conversation

@pp-mo
Copy link
Member

@pp-mo pp-mo commented Oct 8, 2013

A highly simplified approach to the problems presented by automatic load pickling, (which was recently tried + abandoned for lack of agreement or insuperable problems ?)

The idea is to help the user manage their own results cacheing,
for example ...

if iris.util.file_is_newer(dump_filepath, my_source_specs):
  data = pickle.load(dump_filepath)
else:
  data = iris.load(my_source_specs, callback=twiddle_bits)

The key object of this approach is to have the user take responsibility for all the problems that make an automated approach too difficult and dodgy! Hopefully the docstrings explain the problems + caveats well enough for this.

@ghost ghost assigned rhattersley Oct 8, 2013
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The is overly complex. Neither of the other functions that use this function make use of the keys.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Whoops I hadn't spotted that.
It was used in the original caller before I split this out of it, but only for the error handling --which is now contained here.
Will fix...

@pp-mo
Copy link
Member Author

pp-mo commented Oct 9, 2013

Thanks @rhattersley. That really is a lot nicer all round.

@pp-mo
Copy link
Member Author

pp-mo commented Oct 9, 2013

NOTE: I just realised, this really needs a "what's new" entry.
I'll add one shortly...

@pp-mo
Copy link
Member Author

pp-mo commented Oct 9, 2013

needs a "what's new" entry.

Fixed. Think everything so far is addressed now.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you mind moving these tests to the new unit test structure, i.e. into: lib/iris/tests/unit/util/test_file_is_newer_than.py. At which point you reconsider adjusting the class/method groupings to take advantage of the new focus.

@rhattersley
Copy link
Member

I've made a couple of minor suggestions but essentially this is a top-quality submission. Thanks @pp-mo 😄

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would normally be done with the self.assertRaises context manager. See the unittest docs for an example which makes further assertions about the exception raised.

@rhattersley
Copy link
Member

Sorry @pp-mo - I didn't spot the modifications to test_load.py before. (Or it might not have been there.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As with the changes to test_load.py, this (and the other instances of this idiom in this file) would be better handled with the self.assertRaises context manager.

@pp-mo
Copy link
Member Author

pp-mo commented Oct 9, 2013

@rhattersley ... better handled with the self.assertRaises context manager.

Yes, I really should have tried harder with that: This way is much nicer.

rhattersley added a commit that referenced this pull request Oct 9, 2013
Provide utility function "file_is_newer_than" for results caching.
@rhattersley rhattersley merged commit ab874c9 into SciTools:master Oct 9, 2013
@pp-mo pp-mo deleted the pickle_helper branch October 9, 2013 17:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants