Skip to content

Conversation

@VJHack
Copy link
Collaborator

@VJHack VJHack commented Jan 5, 2025

Instead of displaying the original response metrics when the completion was pulled from cache, we want to display metrics related to the cache. The completion info is shown in the following format:

C: number of elements currently in the cache / cache size
t: total FIM time

 ... world\n");     | C: 3 / 250 | t: 0.66 ms

This is what the info message looks like when there is a cache hit.
Screen Shot 2025-01-05 at 2 42 25 AM

Changes in this PR:

  • Changed info message for when there is a cache hit
  • Optimized cache usage by only storing the content and not the other metrics. This helps reduce the size of the cache.

This suggestion was mentioned here: #18 (review)

Copy link
Member

@ggerganov ggerganov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The llama-server now has the option to filter response fields, so another idea for improvement it to use this feature to minimize the amount of information that is transported from the server to the client, by filtering just the fields that are needed by llama.vim:

ggml-org/llama.cpp#10819

@ggerganov ggerganov merged commit 3cc84b0 into ggml-org:master Jan 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants