[Feature]: Manually inject Prefix KV Cache

[toilaluan]

🚀 The feature, motivation and pitch

We're having a good KV-cache compressor base on the paper: https://arxiv.org/abs/2401.03462
Example we can compress the KV-cache of 2048 tokens into the KV-cache of 512 special tokens, so we need a manual way to feed it into the generate function.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.