server: improve user message detection and create checkpoints at every user message - #24176
Conversation
This comment has been minimized.
This comment has been minimized.
eb3f6bb to
68bcfa3
Compare
68bcfa3 to
fd324b1
Compare
|
Could you explain the use case where I could observe the behavior this PR fixes? I think my problem is that I implemented #22929 for my own use case, and I don't see any issues with the current implementation. So I assume this is either for a different kind of model (not Qwen 3.6 27B) or for different behavior (not agentic coding in pi). I remember ggerganov mentioned |
If you resume an older chat session, or restart the server, then using /tree after sending a message will trigger it. |
Yes, that's the main use case. |
|
I am testing and something I didn't realize before is that each tool response is actually a "user" message. So this results in a checkpoint created for each tool call. Instead, I was thinking about making the checkpoints only at the points where the user manually provides input. But I realize that maybe we cannot distinguish between the two? |
A tool response message in this case contains the user message preamble as a prefix, so we simply need to check for a tool response first. I can easily do this with the specialized parsers, but @pwilkin will need to assist in providing it for the autoparser variants. |
@aldehir An simpler alternative approach would be:
|
|
I'd tune it to 4096, I think 8192 is a bit too much with agentic sessions with large turnover, but yeah, definitely 256 is too little, the way it works currently is that if the agent calls a lot of small tools at once, then that immediately clobbers the checkpoints pool even at the large default size of 32. The way I see it, a rule of "always keep last user checkpoint, after that, respect the hard limit" would solve most of these problems. |
Sounds good, the latest commit should reflect that.
I set it 8192, can change if @ggerganov agrees. |
|
I rebased this PR onto the main branch—which includes #24746 —and put it through a full day of intensive testing. Everything seems to be working fine. I'll set it to 256; the harness I use trims old tool messages, which can improve the KV cache reuse rate. |
With 8192, on average we will be reprocessing half of the step (i.e. 4096) tokens when restoring a checkpoint. I think reprocessing 4096 tokens should be OK in majority of cases so default of 8192 is better IMO - we are more interested in sparing memory usage. |
ggerganov
left a comment
ggerganov
left a comment
There was a problem hiding this comment.
Restoring an old session now works correctly.
Note that when running a current session and we do many small tool calls one after the other, the logic creates a checkpoint for each one because it is always the last user message on each turn. To improve that, we should add a logic to remove the penultimate checkpoint if it does not satisfy the step-size condition. This should make the checkpoints to be both distributed sparsely and also respect the user boundaries. With this improvement I think we should be gucci. Can also do it in a follow-up PR.
|
@ggerganov let's do it in a follow up PR. |
…y user message (ggml-org#24176) * server : improve message span logic * cont : cast size_t to int32_t in comparisons * server : create checkpoints before every user msg * chat : remove \n in gemma4 delimiters * chat : merge msg delimiter structs into one * cont : reword comment * cont : initialize tokens in delimiter * cont : add server_tokens::get_raw_tokens() for mtmd * cont : move message finding to server_tokens and skip mtmd tokens * cont : update cohere2moe parser * cont : increase min-step to 8192 and always produce a chkpt for last user message
…y user message (ggml-org#24176) * server : improve message span logic * cont : cast size_t to int32_t in comparisons * server : create checkpoints before every user msg * chat : remove \n in gemma4 delimiters * chat : merge msg delimiter structs into one * cont : reword comment * cont : initialize tokens in delimiter * cont : add server_tokens::get_raw_tokens() for mtmd * cont : move message finding to server_tokens and skip mtmd tokens * cont : update cohere2moe parser * cont : increase min-step to 8192 and always produce a chkpt for last user message
Is this still working as expected? Currently, I only observe two checkpoints at the end when restoring a large session. |
@aldehir Can you please check and confirm whether this feature is still working as expected? It was working correctly after that this merge. However, now checkpoints are only being created at the end. There are dozens of user messages in here. |
|
@patrickzel I'm not aware of anything that would break this, but I'll run some tests. Do you happen to have an example payload I can compare against? |
I used the text completion API via SillyTavern and I copied the whole two prompts. I hope this helps. I think I've found the problem. Restoring the session doesn't like the second system message section. I think the restoration starts after the last system message. |
It's a text completion issue. The same chat generates the expected checkpoints via the chat completion API. |
@aldehir The checkpoint logic added around message_delimiters / message_spans assumes we are dealing with a chat prompt with identifiable user boundaries. In text completion requests, there is no real chat structure and no valid message spans to compute from, so the checkpoint placement logic becomes incorrect and breaks normal completion behavior. |
…y user message (ggml-org#24176) * server : improve message span logic * cont : cast size_t to int32_t in comparisons * server : create checkpoints before every user msg * chat : remove \n in gemma4 delimiters * chat : merge msg delimiter structs into one * cont : reword comment * cont : initialize tokens in delimiter * cont : add server_tokens::get_raw_tokens() for mtmd * cont : move message finding to server_tokens and skip mtmd tokens * cont : update cohere2moe parser * cont : increase min-step to 8192 and always produce a chkpt for last user message
6 participants
Overview
Scan for user message boundaries directly on the tokens and avoid doing a second pass that translates from byte offsets.
Create checkpoints at the start of every user message, as opposed to only the last message.
cc. @ggerganov @jacekpoplawski
Additional information
The checkpoint logic makes sense to me, but I haven't thoroughly tested it for edge cases.
Requirements