关于ceval的评测对于PPL计算结果一样的选项，是选择靠前面的答案吗？

[niexufei]

ceval的评测是对选择题的评测，看log的prediction的输出，是计算每个选项的PPL值，我的问题：
1、如果一道题目的多个选项的PPL值是一样的，prediction的选择是如何定的？是选择第一个吗？
例如：下面log是一道题的predictionlog输出，数据做了模糊化处理：这个题目中，C和D的PPL是一样的，此时答案选择C，是选择第一个最小的PPL吗？
2、关于对每个选项计算PPL的代码，能否告知位置？我想看下源码实现；
非常感谢。
"0": {
"in-context examples": "AAAA",
"label: A": {
"testing input": "XXXX",
"prompt": "YYYYY",
"PPL": 1.5621301775147929
},
"label: B": {
"testing input": "XXXX",
"prompt": "YYYYY",
"PPL": 1.5606508875739644
},
"label: C": {
"testing input": "XXXX",
"prompt": "YYYYY",
"PPL": 1.5591715976331362
},
"label: D": {
"testing input": "XXXX",
"prompt": "YYYYY",
"PPL": 1.5591715976331362
},
"prediction": "C"
},

[Leymore]

1. Yes, the first smallest PPL is taken
   https://github.com/InternLM/opencompass/blob/262ab794fb52084e4494c281210e652706ce1280/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py#L198-L201
2. The source code is here

[niexufei]

感觉这种PPL的方式，对于稍微复杂的选项，不靠谱；
我有一些题目，计算出来的PPL，好多选项都是一样的，那么预测的结果就很随机，这类复杂的选择题，评估框架怎么处理呢？有没其他数据的例子可以参考？多谢

[Leymore]

First, I want to confirm which model you're testing. If it's theoretically a more powerful model, and the PPL has equal items, then you might need to consider first whether there's a bug.
If the understanding / knowledge ability of the model you're testing is insufficient, then I would rather suggest looking at some relatively easy subsets, such as the humanities and social science subsets in C-Eval / mmlu. The STEM subsets may be of little value due to their resemblance to random guessing.
Finally, if you really need to observe, then you may consider incorporating the options into the prompt, for example: 'Answer: A. blablabla', instead of 'Answer: A'. During our large model testing, we found that this approach can lead to decreased accuracy, but we don't know what effect it will have in your case, because the problem you are encountering now is that the differences between the options are too small.

[niexufei]

感谢答复和建议。你帮我看下我的例子，测试的模型是baichuan-13b-base，对于这类题目是不是不擅长？可能是模型能力的问题？还是我们评估时使用的prompt不太合理的问题？
prediction的结果如下附件：
baichuan-13b-base-predication-ppl.txt

这五道题的标准答案应该是：DCACC

另外，你上面能说的可以将相应的选项也放到prompt中，是下面这样吗？就是testing input的最后，将选项内容加入？
"label: A": {
"testing input": "\n下面是一道单项选择题，请选出其中的正确答案。\n一件商品先提价10%，后又降价10%，则现在的价格于原来相比：\nA. 不变\nB. 提高了\nC. 降低了\nD. 0.1\n答案: \nA.不变",
"prompt": "下面是一道单项选择题，请选出其中的正确答案。\n如果香蕉对应黄色，西瓜对应什么？\nA. 黄色\nB. 红色\nC. 绿色\nD. 甜\n答案: \nC\n\n\n下面是一道单项选择题，请选出其中的正确答案。\n10000+300+5=？\nA. 10305\nB. 135\nC. 8\nD. 100003005\n答案: \nA\n\n\n下面是一道单项选择题，请选出其中的正确答案。\n有三件事：（1）穿袜子 （2）剪脚趾甲 （3）穿鞋，应该按照什么顺序处理？\nA. （1）→（2）→（3）\nB. （2）→（1）→（3）\nC. （3）→（2）→（1）\nD. （2）→（3）→（1）\n答案: \nB\n\n\n下面是一道单项选择题，请选出其中的正确答案。\n有10根点燃的蜡烛，吹灭3根，第二天还剩几根？\nA. 3\nB. 7\nC. 10\nD. 0\n答案: \nA\n\n\n下面是一道单项选择题，请选出其中的正确答案。\n桌子上有10个苹果，小苹吃了2个。小果拿来3个橘子并分给小明吃了1个。还剩几个苹果？\nA. 10\nB. 9\nC. 8\nD. 7\n答案: \nC\n\n\n\n\n下面是一道单项选择题，请选出其中的正确答案。\n一件商品先提价10%，后又降价10%，则现在的价格于原来相比：\nA. 不变\nB. 提高了\nC. 降低了\nD. 0.1\n答案: \nA. 不变",
"PPL": 1.025390625
},

[Leymore]

Looks like it is not C-Eval : ). The accuracy of baichuan-13b-base on C-Eval can reach 55%, which should be quite good. However, I am not familiar with the properties of this model, so I can't specify why it didn't perform well on your questions. As far as I understand, questions 1 / 2 are directly testing semantics, they are relatively simple questions, and there are in-context examples, theoretically, they should not be answered incorrectly. Question 3 involves a difficult mathematical calculation, the model is likely to guess the answer correctly by chance. Questions 4 / 5 involve some logical deduction, which should be quite difficult for the model. However, since the test volume is relatively limited, I can't pinpoint the exact issue.

Yes, just like that. In addition, both the in-context examples and the final answer should have the option incorporated, not only the final answer.