Is there a way to retrieve key-value cache values using onnxruntime-genai?

[Tabrizian]

Can we retrieve the key-value cache values using this package for re-use across different generations?

[natke]

Hi @Tabrizian, this is an API that we are looking at for another customer. I will move it into the Discussions forum where we can continue the conversation. If you have specific use cases, please feel free to add them there!

[Tabrizian]

Thanks @natke. The use-case I have is to be able to manually control which set of requests are included in every batch. Being able to obtain key-value cache parameters can help us avoid redundant computation when swapping sequences in and out of the request batch.