Skip to content

Conversation

@gante
Copy link
Member

@gante gante commented Feb 21, 2024

What does this PR do?

What does this PR do?

In #27931, where the static cache was introduced, we noticed it had the following hard requirements:

  1. The model instance holds the cache, as opposed to being a tensor passed around;
  2. Each layer has its own cache, as opposed to a single cache for all layers.

This contrasts with previous implementations (e.g. DynamicCache). Given the hard requirements of the static cache, and the resulting benefits, this PR migrates the interface for all cache classes so as to match the static cache. As a result, the modeling code becomes slightly simpler 🤗

@gante
Copy link
Member Author

gante commented Apr 18, 2024

Closing this PR and other cache PRs, as we want to move in the opposite direction (static cache behaving like the other caches)

@gante gante closed this Apr 18, 2024
@liangan1
Copy link

Closing this PR and other cache PRs, as we want to move in the opposite direction (static cache behaving like the other caches)

is there any related PR for this design?

@gante gante deleted the api_cache branch May 22, 2024 10:13
@gante
Copy link
Member Author

gante commented May 22, 2024

PR that standardized the interface: #30476
(cc @liangan1 )

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants