Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: First-class caching #1104

Merged
merged 26 commits into from
Oct 3, 2024
Merged
Show file tree
Hide file tree
Changes from 25 commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
dbe5a3c
added basic fingerprinting support
Jul 18, 2024
998ca73
fixed tests and documentation build
Sep 25, 2024
d6cbcb6
fixed pre-commits
Sep 25, 2024
dc4a75c
fixed NoneType for <3.10
Sep 25, 2024
b968fba
fix DISABLE behavior and handle unhashable values
Sep 26, 2024
95d5e4f
fixed DISABLE behavior; fixed cache key name collision with extract_f…
Sep 26, 2024
1c2873f
fixed cache_key signature in test
Sep 26, 2024
9be419b
added caching tutorial notebook
Sep 27, 2024
ea74a27
fixed materializer bug
Sep 27, 2024
b68775d
rename kwarg to dependencies_data_versions
Sep 30, 2024
73cdf61
DEFAULT behavior can be specified at the Driver
Sep 30, 2024
ccc6c01
changed default cache path
Sep 30, 2024
82139db
renamed drivers and modules with descriptive names
Sep 30, 2024
963e5fc
updated tutorial notebook with feedback
Sep 30, 2024
6b0df62
renamed to HamiltonCacheAdapter, remove materializer RECOMPUTE behavior
Oct 1, 2024
571e4f7
fix default parameter and materializer; handle stdlib types
Oct 2, 2024
c4ffb9f
updated tutorial notebook; added materializer tutorial
Oct 2, 2024
eaca75c
fixed tests for dataloader tag change
Oct 2, 2024
39df87b
updated docs; added notebook rendering support
Oct 2, 2024
1a64444
fixed singledispatch type annotation for older Python versions
Oct 2, 2024
1356773
cleaned up docs; fixed type 3.8; remove docs dep
Oct 3, 2024
89388ea
fix docs config
Oct 3, 2024
eb6e277
remove unreliable test for 3.8
Oct 3, 2024
e27ee53
fixed recursion error for unordered maps
Oct 3, 2024
e778d51
add mention about cache keys stability
Oct 3, 2024
75cc9f3
fixed max_depth in recursive hashing test
Oct 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -152,3 +152,6 @@ examples/**/hamilton-env
**.pkl
**.lance
**.txn

# hamilton default caching directory
**/.hamilton_cache/
167 changes: 167 additions & 0 deletions docs/concepts/_caching/view_run_example.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
18 changes: 18 additions & 0 deletions docs/concepts/builder.rst
Original file line number Diff line number Diff line change
Expand Up @@ -179,6 +179,24 @@ Adds `DataSaver` and `DataLoader` nodes to your dataflow. This allows to visuali
:align: center


with_cache()
--------------

This enables Hamilton's caching feature, which allows to automatically store intermediary results and reuse them in subsequent executions to skip computations. Learn more in the :doc:`/concepts/caching` section.

.. code-block:: python

from hamilton import driver
import my_dataflow

dr = (
driver.Builder()
.with_modules(my_dataflow)
.with_cache()
.build()
)


with_adapters()
---------------

Expand Down
Loading