With chat templates, instructions shouldn't be prepended to system prompt

[Whadup]

lighteval/src/lighteval/few_shot_manager.py

Line 199 in df21407

examples[0]["content"] = instruction + examples[0]["content"]

If we use a chat_template, have a system prompt and an instruction, the instruction is prepended to the system prompt message. It should be prepended to the first user message instead.

[clefourrier]

Why not.
We are adding it with the system prompt so it plays better with the truncation options iirc.
We can provide both options btw, but I don't feel like there is "a single right way" of managing model prompts at the moment.

[Whadup]

Sure, there's not one single right way. Adding it in front of the system prompt feels weird to me tho, particularly for longer system prompts. In this fake example, I think the order of inputs is funky

The following are multiple choice questions on fish. You're a helpful assistant who only answers in rhyme. When you do not understand the question, politely ask for clarifications. Always remain respectful.

Idk, my intuition is that chat-models expect the prompt as the last instruction.

[clefourrier]

Oh true, got your point!
Yep that's a bug, should be examples[0]["content"] + instruction!

[Whadup]

Yeah, that seems like a good solution.
Alternatively, we could prepend to the first message with role="user".

[clefourrier]

We have both options actually, depending on whether there is a system prompt or not. As mentioned above, they will play differently with truncation