Context size

[aafsmarak]

Default n_ctx is 4096. Can I increase context size or will it affect quality? cli output says:
n_ctx_per_seq (4096) < n_ctx_train (131072) -- the full capacity of the model will not be utilized

[CISC]

It's perfectly fine to increase context length up to n_ctx_train without degradation, it will just consume more V/RAM.

[aafsmarak]

Thanks

[aafsmarak]

How to know context usage of a conversation?
I mean how many tokens used up.

[CISC]

For llama-cli I think that's only visible with --verbose (n_consumed), but that outputs a lot.

[aafsmarak]

What about llama-server?
And what happens when it runs out of context? Does it keep buffering in the webui of llama-server? I have seen such behaviour more than once.

[CISC]

If you use the OpenAI compatible endpoints there's a usage field in the response.

When you run out of context it will be shifted (unless disabled), meaning old context will be scrapped.