Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement the internal last cache API #25093

Closed
Tracked by #25091
hiltontj opened this issue Jun 25, 2024 · 0 comments · Fixed by #25133
Closed
Tracked by #25091

Implement the internal last cache API #25093

hiltontj opened this issue Jun 25, 2024 · 0 comments · Fixed by #25133
Assignees
Labels

Comments

@hiltontj
Copy link
Contributor

hiltontj commented Jun 25, 2024

The last cache can be stored as a map of database/table/cache (there can be multiple named caches per table).

Each cache has a set of key columns (this is described well in this comment: #25109 (review)), and itself is a nested map, with one level for each key column in the hash. The nesting is done to create a hierarchy from the key columns if the cache.

Writes to the last cache will be sent after they have been flushed to the WAL, i.e., here.

The last cache needs to have an age out, or time-to-live (TTL) so that cache entries beyond a given age (default to 4 hours) are evicted, and the cache isn't bloated with old data.

The last cache implementation needs to also implement Datafusion's TableProvider trait, but that work may make more sense along with #25095.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant