Feat/fingerprinting

[zilto]

Following up our design discussion on caching and fingerprinting. Here are the basic working mechanisms for caching as we discussed.

Composable parts:

• caching algorithm: decide when to compute or use cache
• fingerprint store (repository): how to store {execution context: fingerprint}
• result store (cache): how to store {fingerprint: result / data}
• serialization and deserialization (related to result store)

TODO

• figure out ideal module structure
• documentation
• standardize and agree on naming
• introspection mechanisms
• good logger for debugging
• special caching algorithm behavior
  • always recompute: does the recompute and produces a new fingerprint; this fingerprint affects the keys of downstream nodes
  • don't fingerprint: can be recomputed or read from cache; in the eye of the downstream nodes, the value / fingerprint is constant (related: Add ability to mark function outputs as unserializable  #743 )
  • read historical value: if a value exists, use it (i.e., the existing CachingGraphAdapter). It would require storing a node_name metadata in the fingerprint store. This is very easy with diskcache
• surface to user post-run caching explanations (visualization, rule-based text explanations)
• surface to user pre-run caching explanations; this will use node versions (must recompute), top-level inputs and overrides, and the state of the fingerprint store (if it's empty, we know we must recompute everything same if node_name metadata is absent)
• checkpointing is a subset of this cache feature. All you need to do is explicitly pass a mapping of fingerprints to SmartCache(fingerprints={...}) (which could be hidden beind a single run_id). Then the adapter will use these fingerprints to read from the result store instead of trying to compute these fingerprints or load them from the fingerprint store.
• improve the protocol definition for Store objects. get and set are required, but unclear if open() and close() are necessary for all (could be null operations)
• create a MaterializerStore where get() and set() are materializers

[skrawcz]

this is related to #940