Skip to content

Commit

Permalink
added basic fingerprinting support
Browse files Browse the repository at this point in the history
added docstring to base64 encode

fingerprinting notebook section

added caching code and example

dev checkpoint

added Fingerprint class

first-class caching v0.1

fixed tests

updated module_6 test case

updated docstrings

documentation WIP added

refactor file structure

add direct access to cache via Driver

remove

Support Parallelizable/Collect; removed Fingerprint

Refactored the logic of the CacheAdapter to have
explicit operations. Many challenges arise from
`Parallel` nodes that return sequence of elements,
but actually the data version of individual elements
is what matters. Also, `Collect` can have difficulties
access upstream data version if these were computed
in other threads/processes.

The `Fingerprint` construct was removed because it
obfuscated what information was relevant to pass
around. It's name is also less evocative than
"data version" and "code version".

made SQLMetadataStore threadsafe

added structured logging; move class to hamilton.lifecycle.caching

structured log; data saver; cache decorator

added from_string to CachingBehavior enum

added docs and mermaid graph support

fixed variable renaming issue

updated docs requirements; adapted typing to 3.8

reverted Sequence type to Sequence class for singledispatch

fixed typo

added deprecation warnings to other caching methods

fixed missing kwarg for recursive data versioning

output structured log to file

updated expand nodes handling; added ignore behavior

updated caching behaviors and admonitions

updated Sequence import for 3.8 support

fixed bugs: sentinel values, log printing, failed nodes

updated docs

updated docstring; fixed materialization with parallel

fixed 3.8 typing

guard against setting cache twice

registered separate function for versioning bytes

refactored adapter to use internal hook

replace HamiltonGraph by FunctionGraph

add _get_node_role() method

switched to internal hooks pre/post node

refactored to key on run_id; refactor result_store

refactored sqlitestore to hamilton.stores

fixed type annotations

fixed docs reference and docstrings

updated result_store tests

fixed materialization from result_store

added roadmap to docs

refactored to move to hamilton/caching

fingerprinting.set_max_depth() added

changed cache decorator to a class

updated all docstrings for SmartCacheAdapter

renamed context_key to cache_key

added deprecation warnings using logging

improved warning message

added TODO
  • Loading branch information
zilto authored and zilto committed Sep 25, 2024
1 parent 1365e88 commit dbe5a3c
Show file tree
Hide file tree
Showing 39 changed files with 4,335 additions and 13 deletions.
18 changes: 18 additions & 0 deletions docs/concepts/builder.rst
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,24 @@ Adds `DataSaver` and `DataLoader` nodes to your dataflow. This allows to visuali
:align: center


with_cache()
--------------

This enables Hamilton's caching feature, which allows to automatically store intermediary results and reuse them in subsequent executions to skip computations. Learn more in the :doc:`/concepts/caching` section.

.. code-block:: python
from hamilton import driver
import my_dataflow
dr = (
driver.Builder()
.with_modules(my_dataflow)
.with_cache()
.build()
)
with_adapters()
---------------

Expand Down
Loading

0 comments on commit dbe5a3c

Please sign in to comment.