我解决了torchrun环境异常情况下的No predictions found问题. #1370

liuwake · 2024-07-28T05:27:30Z

liuwake
Jul 28, 2024

环境和配置

我在baidu AI studio V100 32G Ubuntu 20.04下安装了open-compass.由于baidu AI studio对于conda的限制,我选择在vs-code界面打开jupyter并使用pip安装.我安装了torch2.0.1和某些依赖,然后我git clone本repo然后执行pip install -e ..我然后用wget下载了相关测试数据并解压到指定位置.以上所有程序都很顺利.

执行出错

然而我在执行open-compass官方文档-快速开始的测试命令(python run.py --datasets ceval_gen --hf-type chat --hf-path /home/aistudio/data/internlm-chat-7b --debug for me)的时候,发现每次都是有问题的,表现为:

所有问题的结果都是No predictions found.如下是我截取了两条连续的示例.

07/28 13:08:26 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
07/28 13:08:26 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
07/28 13:08:26 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:08:26 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:08:28 - OpenCompass - DEBUG - Modules of opencompass's load_dataset registry have been automatically imported from opencompass.datasets
07/28 13:08:28 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:08:28 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:08:28 - OpenCompass - ERROR - /home/aistudio/opencompass/opencompass/tasks/openicl_eval.py - _score - 248 - Task [internlm-chat-7b_hf/ceval-computer_network]: No predictions found.
07/28 13:08:28 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:08:28 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:08:31 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:08:31 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:08:31 - OpenCompass - ERROR - /home/aistudio/opencompass/opencompass/tasks/openicl_eval.py - _score - 248 - Task [internlm-chat-7b_hf/ceval-operating_system]: No predictions found.

在所有问题都出现No predictions found.之后,且最终结果都为-.

07/28 13:10:47 - OpenCompass - DEBUG - An `DefaultSummarizer` instance is built from registry, and its implementation can be found in opencompass.summarizers.default
dataset                                         version    metric    mode    internlm-chat-7b_hf
----------------------------------------------  ---------  --------  ------  ---------------------
ceval-computer_network                          -          -         -       -
ceval-operating_system                          -          -         -       -
ceval-computer_architecture                     -          -         -       -
ceval-college_programming                       -          -         -       -
ceval-college_physics                           -          -         -       -
ceval-college_chemistry                         -          -         -       -
ceval-advanced_mathematics                      -          -         -       -
ceval-probability_and_statistics                -          -         -       -
ceval-discrete_mathematics                      -          -         -       -
ceval-electrical_engineer                       -          -         -       -
ceval-metrology_engineer                        -          -         -       -
ceval-high_school_mathematics                   -          -         -       -
ceval-high_school_physics                       -          -         -       -
ceval-high_school_chemistry                     -          -         -       -
ceval-high_school_biology                       -          -         -       -
ceval-middle_school_mathematics                 -          -         -       -
ceval-middle_school_biology                     -          -         -       -
ceval-middle_school_physics                     -          -         -       -
ceval-middle_school_chemistry                   -          -         -       -
ceval-veterinary_medicine                       -          -         -       -
ceval-college_economics                         -          -         -       -
ceval-business_administration                   -          -         -       -
ceval-marxism                                   -          -         -       -
ceval-mao_zedong_thought                        -          -         -       -
ceval-education_science                         -          -         -       -
ceval-teacher_qualification                     -          -         -       -
ceval-high_school_politics                      -          -         -       -
ceval-high_school_geography                     -          -         -       -
ceval-middle_school_politics                    -          -         -       -
ceval-middle_school_geography                   -          -         -       -
ceval-modern_chinese_history                    -          -         -       -
ceval-ideological_and_moral_cultivation         -          -         -       -
ceval-logic                                     -          -         -       -
ceval-law                                       -          -         -       -
ceval-chinese_language_and_literature           -          -         -       -
ceval-art_studies                               -          -         -       -
ceval-professional_tour_guide                   -          -         -       -
ceval-legal_professional                        -          -         -       -
ceval-high_school_chinese                       -          -         -       -
ceval-high_school_history                       -          -         -       -
ceval-middle_school_history                     -          -         -       -
ceval-civil_servant                             -          -         -       -
ceval-sports_science                            -          -         -       -
ceval-plant_protection                          -          -         -       -
ceval-basic_medicine                            -          -         -       -
ceval-clinical_medicine                         -          -         -       -
ceval-urban_and_rural_planner                   -          -         -       -
ceval-accountant                                -          -         -       -
ceval-fire_engineer                             -          -         -       -
ceval-environmental_impact_assessment_engineer  -          -         -       -
ceval-tax_accountant                            -          -         -       -
ceval-physician                                 -          -         -       -
07/28 13:10:47 - OpenCompass - INFO - write summary to /home/aistudio/opencompass/outputs/default/20240728_130820/summary/summary_20240728_130820.txt
07/28 13:10:47 - OpenCompass - INFO - write csv to /home/aistudio/opencompass/outputs/default/20240728_130820/summary/summary_20240728_130820.csv

分析问题

1.首先我对输出日志进行了观察,如下是开头:

07/28 13:08:20 - OpenCompass - INFO - Loading ceval_gen: configs/datasets/ceval/ceval_gen.py
07/28 13:08:20 - OpenCompass - DEBUG - Using model: {'type': 'opencompass.models.huggingface_above_v4_33.HuggingFacewithChatTemplate', 'abbr': 'internlm-chat-7b_hf', 'path': '/home/aistudio/data/internlm-chat-7b', 'model_kwargs': {}, 'tokenizer_path': None, 'tokenizer_kwargs': {}, 'generation_kwargs': {}, 'peft_path': None, 'peft_kwargs': {}, 'max_seq_len': None, 'max_out_len': 256, 'batch_size': 8, 'pad_token_id': None, 'stop_words': [], 'run_cfg': {'num_gpus': 1}}
07/28 13:08:20 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
07/28 13:08:20 - OpenCompass - INFO - Current exp folder: outputs/default/20240728_130820
07/28 13:08:20 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
07/28 13:08:20 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners
07/28 13:08:20 - OpenCompass - DEBUG - Get class `NumWorkerPartitioner` from "partitioner" registry in "opencompass"
07/28 13:08:20 - OpenCompass - DEBUG - An `NumWorkerPartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.num_worker
07/28 13:08:20 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
07/28 13:08:20 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
07/28 13:08:20 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
07/28 13:08:20 - OpenCompass - DEBUG - Additional config: {}
07/28 13:08:20 - OpenCompass - INFO - Partitioned into 1 tasks.
07/28 13:08:20 - OpenCompass - DEBUG - Task 0: [internlm-chat-7b_hf/ceval-computer_network,internlm-chat-7b_hf/ceval-operating_system,internlm-chat-7b_hf/ceval-computer_architecture,internlm-chat-7b_hf/ceval-college_programming,internlm-chat-7b_hf/ceval-college_physics,internlm-chat-7b_hf/ceval-college_chemistry,internlm-chat-7b_hf/ceval-advanced_mathematics,internlm-chat-7b_hf/ceval-probability_and_statistics,internlm-chat-7b_hf/ceval-discrete_mathematics,internlm-chat-7b_hf/ceval-electrical_engineer,internlm-chat-7b_hf/ceval-metrology_engineer,internlm-chat-7b_hf/ceval-high_school_mathematics,internlm-chat-7b_hf/ceval-high_school_physics,internlm-chat-7b_hf/ceval-high_school_chemistry,internlm-chat-7b_hf/ceval-high_school_biology,internlm-chat-7b_hf/ceval-middle_school_mathematics,internlm-chat-7b_hf/ceval-middle_school_biology,internlm-chat-7b_hf/ceval-middle_school_physics,internlm-chat-7b_hf/ceval-middle_school_chemistry,internlm-chat-7b_hf/ceval-veterinary_medicine,internlm-chat-7b_hf/ceval-college_economics,internlm-chat-7b_hf/ceval-business_administration,internlm-chat-7b_hf/ceval-marxism,internlm-chat-7b_hf/ceval-mao_zedong_thought,internlm-chat-7b_hf/ceval-education_science,internlm-chat-7b_hf/ceval-teacher_qualification,internlm-chat-7b_hf/ceval-high_school_politics,internlm-chat-7b_hf/ceval-high_school_geography,internlm-chat-7b_hf/ceval-middle_school_politics,internlm-chat-7b_hf/ceval-middle_school_geography,internlm-chat-7b_hf/ceval-modern_chinese_history,internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation,internlm-chat-7b_hf/ceval-logic,internlm-chat-7b_hf/ceval-law,internlm-chat-7b_hf/ceval-chinese_language_and_literature,internlm-chat-7b_hf/ceval-art_studies,internlm-chat-7b_hf/ceval-professional_tour_guide,internlm-chat-7b_hf/ceval-legal_professional,internlm-chat-7b_hf/ceval-high_school_chinese,internlm-chat-7b_hf/ceval-high_school_history,internlm-chat-7b_hf/ceval-middle_school_history,internlm-chat-7b_hf/ceval-civil_servant,internlm-chat-7b_hf/ceval-sports_science,internlm-chat-7b_hf/ceval-plant_protection,internlm-chat-7b_hf/ceval-basic_medicine,internlm-chat-7b_hf/ceval-clinical_medicine,internlm-chat-7b_hf/ceval-urban_and_rural_planner,internlm-chat-7b_hf/ceval-accountant,internlm-chat-7b_hf/ceval-fire_engineer,internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer,internlm-chat-7b_hf/ceval-tax_accountant,internlm-chat-7b_hf/ceval-physician]
07/28 13:08:20 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners
07/28 13:08:20 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
07/28 13:08:20 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
07/28 13:08:20 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks
07/28 13:08:20 - OpenCompass - DEBUG - Get class `OpenICLInferTask` from "task" registry in "opencompass"
07/28 13:08:20 - OpenCompass - DEBUG - An `OpenICLInferTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer
/bin/sh: 1: torchrun: not found
07/28 13:08:26 - OpenCompass - DEBUG - Get class `NaivePartitioner` from "partitioner" registry in "opencompass"
07/28 13:08:26 - OpenCompass - DEBUG - An `NaivePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.naive
07/28 13:08:26 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
07/28 13:08:26 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
07/28 13:08:26 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
07/28 13:08:26 - OpenCompass - DEBUG - Additional config: {'eval': {'runner': {'task': {}}}}
07/28 13:08:26 - OpenCompass - INFO - Partitioned into 52 tasks.

然后日志展示了问题名字和编号.
2. 我注意到了这一句

07/28 13:08:20 - OpenCompass - DEBUG - An `OpenICLInferTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer
/bin/sh: 1: torchrun: not found

这里出现/bin/bash字样说明open-compass调用了bash执行某些命令.而根据 OpenICLInferTask和torchrun我认为这句输出可能跟该问题强关联.而且,我还注意到运行期间__VRAM__和 GPU 占用都是 0!所以这更加强化了我的观点.
3. 我用vscode全文搜索了torchrun,发现多个结果.其中很多结果是注释,但是其中一个结果吸引了我.是位于opencompass/opencompass/tasks/openicl_infer.py的get_command(self, cfg_path, template)函数中.(当然opencompass/opencompass/tasks/目录下也有openicl_eval.py,openicl_attack.py,其中代码大致思路雷同,而我们只需关注推理的代码也就是openicl_infer.py.)

def get_command(self, cfg_path, template):
        """Get the command template for the task.

        Args:
            cfg_path (str): The path to the config file of the task.
            template (str): The template which have '{task_cmd}' to format
                the command.
        """
        sys.path.append(os.getcwd())
        script_path = __file__
        backend_keys = ['VLLM', 'Lmdeploy']
        use_backend = any(
            key in str(self.model_cfgs[0].get('type', ''))
            or key in str(self.model_cfgs[0].get('llm', {}).get('type', ''))
            for key in backend_keys)
        if self.num_gpus > 0 and not use_backend:
            port = random.randint(12000, 32000)
            command = (f'torchrun --master_port={port} '
                       f'--nproc_per_node {self.num_procs} '
                       f'{script_path} {cfg_path}')
        else:
            python = sys.executable
            command = f'{python} {script_path} {cfg_path}'

        return template.format(task_cmd=command)

这里torchrun实质是shell的命令,在不是很久的torch版本中已经支持负责替代以前的分布式计算命令.我在conda环境下输入which torchrun发现没有返回结果.然后我查找了下:

(python35-paddle120-env) aistudio@jupyter-145988-8195042:~$ find / -name torchrun 2>/dev/null
/home/aistudio/.local/bin/torchrun

然后:

(python35-paddle120-env) aistudio@jupyter-145988-8195042:~/opencompass$ export PATH=$PATH:/home/aistudio/.local/bin/

最后重新运行命令,成功开始从hf下载数据,载入模型,并占用VRAM到30G,占用GPU到70%以上,开始了推理.日志也展示了相关滚动条!

liuwake · 2024-07-28T05:28:29Z

liuwake
Jul 28, 2024
Author

这里是成功的日志全文:

Click to expand/collapse

(python35-paddle120-env) aistudio@jupyter-145988-8195042:~/opencompass$ export PATH=$PATH:/home/aistudio/.local/bin/
(python35-paddle120-env) aistudio@jupyter-145988-8195042:~/opencompass$ python run.py     --datasets ceval_gen     --hf-type chat     --hf-path /home/aistudio/data/internlm-chat-7b     --debug
07/28 12:34:45 - OpenCompass - INFO - Loading ceval_gen: configs/datasets/ceval/ceval_gen.py
07/28 12:34:45 - OpenCompass - DEBUG - Using model: {'type': 'opencompass.models.huggingface_above_v4_33.HuggingFacewithChatTemplate', 'abbr': 'internlm-chat-7b_hf', 'path': '/home/aistudio/data/internlm-chat-7b', 'model_kwargs': {}, 'tokenizer_path': None, 'tokenizer_kwargs': {}, 'generation_kwargs': {}, 'peft_path': None, 'peft_kwargs': {}, 'max_seq_len': None, 'max_out_len': 256, 'batch_size': 8, 'pad_token_id': None, 'stop_words': [], 'run_cfg': {'num_gpus': 1}}
07/28 12:34:45 - OpenCompass - INFO - Loading example: configs/summarizers/example.py
07/28 12:34:45 - OpenCompass - INFO - Current exp folder: outputs/default/20240728_123445
07/28 12:34:45 - OpenCompass - WARNING - SlurmRunner is not used, so the partition argument is ignored.
07/28 12:34:45 - OpenCompass - DEBUG - Modules of opencompass's partitioner registry have been automatically imported from opencompass.partitioners
07/28 12:34:45 - OpenCompass - DEBUG - Get class `NumWorkerPartitioner` from "partitioner" registry in "opencompass"
07/28 12:34:45 - OpenCompass - DEBUG - An `NumWorkerPartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.num_worker
07/28 12:34:45 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
07/28 12:34:45 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
07/28 12:34:45 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
07/28 12:34:45 - OpenCompass - DEBUG - Additional config: {}
07/28 12:34:45 - OpenCompass - INFO - Partitioned into 1 tasks.
07/28 12:34:45 - OpenCompass - DEBUG - Task 0: [internlm-chat-7b_hf/ceval-computer_network,internlm-chat-7b_hf/ceval-operating_system,internlm-chat-7b_hf/ceval-computer_architecture,internlm-chat-7b_hf/ceval-college_programming,internlm-chat-7b_hf/ceval-college_physics,internlm-chat-7b_hf/ceval-college_chemistry,internlm-chat-7b_hf/ceval-advanced_mathematics,internlm-chat-7b_hf/ceval-probability_and_statistics,internlm-chat-7b_hf/ceval-discrete_mathematics,internlm-chat-7b_hf/ceval-electrical_engineer,internlm-chat-7b_hf/ceval-metrology_engineer,internlm-chat-7b_hf/ceval-high_school_mathematics,internlm-chat-7b_hf/ceval-high_school_physics,internlm-chat-7b_hf/ceval-high_school_chemistry,internlm-chat-7b_hf/ceval-high_school_biology,internlm-chat-7b_hf/ceval-middle_school_mathematics,internlm-chat-7b_hf/ceval-middle_school_biology,internlm-chat-7b_hf/ceval-middle_school_physics,internlm-chat-7b_hf/ceval-middle_school_chemistry,internlm-chat-7b_hf/ceval-veterinary_medicine,internlm-chat-7b_hf/ceval-college_economics,internlm-chat-7b_hf/ceval-business_administration,internlm-chat-7b_hf/ceval-marxism,internlm-chat-7b_hf/ceval-mao_zedong_thought,internlm-chat-7b_hf/ceval-education_science,internlm-chat-7b_hf/ceval-teacher_qualification,internlm-chat-7b_hf/ceval-high_school_politics,internlm-chat-7b_hf/ceval-high_school_geography,internlm-chat-7b_hf/ceval-middle_school_politics,internlm-chat-7b_hf/ceval-middle_school_geography,internlm-chat-7b_hf/ceval-modern_chinese_history,internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation,internlm-chat-7b_hf/ceval-logic,internlm-chat-7b_hf/ceval-law,internlm-chat-7b_hf/ceval-chinese_language_and_literature,internlm-chat-7b_hf/ceval-art_studies,internlm-chat-7b_hf/ceval-professional_tour_guide,internlm-chat-7b_hf/ceval-legal_professional,internlm-chat-7b_hf/ceval-high_school_chinese,internlm-chat-7b_hf/ceval-high_school_history,internlm-chat-7b_hf/ceval-middle_school_history,internlm-chat-7b_hf/ceval-civil_servant,internlm-chat-7b_hf/ceval-sports_science,internlm-chat-7b_hf/ceval-plant_protection,internlm-chat-7b_hf/ceval-basic_medicine,internlm-chat-7b_hf/ceval-clinical_medicine,internlm-chat-7b_hf/ceval-urban_and_rural_planner,internlm-chat-7b_hf/ceval-accountant,internlm-chat-7b_hf/ceval-fire_engineer,internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer,internlm-chat-7b_hf/ceval-tax_accountant,internlm-chat-7b_hf/ceval-physician]
07/28 12:34:45 - OpenCompass - DEBUG - Modules of opencompass's runner registry have been automatically imported from opencompass.runners
07/28 12:34:45 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
07/28 12:34:45 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
07/28 12:34:45 - OpenCompass - DEBUG - Modules of opencompass's task registry have been automatically imported from opencompass.tasks
07/28 12:34:45 - OpenCompass - DEBUG - Get class `OpenICLInferTask` from "task" registry in "opencompass"
07/28 12:34:45 - OpenCompass - DEBUG - An `OpenICLInferTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_infer
07/28 12:35:03 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-computer_network,internlm-chat-7b_hf/ceval-operating_system,internlm-chat-7b_hf/ceval-computer_architecture,internlm-chat-7b_hf/ceval-college_programming,internlm-chat-7b_hf/ceval-college_physics,internlm-chat-7b_hf/ceval-college_chemistry,internlm-chat-7b_hf/ceval-advanced_mathematics,internlm-chat-7b_hf/ceval-probability_and_statistics,internlm-chat-7b_hf/ceval-discrete_mathematics,internlm-chat-7b_hf/ceval-electrical_engineer,internlm-chat-7b_hf/ceval-metrology_engineer,internlm-chat-7b_hf/ceval-high_school_mathematics,internlm-chat-7b_hf/ceval-high_school_physics,internlm-chat-7b_hf/ceval-high_school_chemistry,internlm-chat-7b_hf/ceval-high_school_biology,internlm-chat-7b_hf/ceval-middle_school_mathematics,internlm-chat-7b_hf/ceval-middle_school_biology,internlm-chat-7b_hf/ceval-middle_school_physics,internlm-chat-7b_hf/ceval-middle_school_chemistry,internlm-chat-7b_hf/ceval-veterinary_medicine,internlm-chat-7b_hf/ceval-college_economics,internlm-chat-7b_hf/ceval-business_administration,internlm-chat-7b_hf/ceval-marxism,internlm-chat-7b_hf/ceval-mao_zedong_thought,internlm-chat-7b_hf/ceval-education_science,internlm-chat-7b_hf/ceval-teacher_qualification,internlm-chat-7b_hf/ceval-high_school_politics,internlm-chat-7b_hf/ceval-high_school_geography,internlm-chat-7b_hf/ceval-middle_school_politics,internlm-chat-7b_hf/ceval-middle_school_geography,internlm-chat-7b_hf/ceval-modern_chinese_history,internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation,internlm-chat-7b_hf/ceval-logic,internlm-chat-7b_hf/ceval-law,internlm-chat-7b_hf/ceval-chinese_language_and_literature,internlm-chat-7b_hf/ceval-art_studies,internlm-chat-7b_hf/ceval-professional_tour_guide,internlm-chat-7b_hf/ceval-legal_professional,internlm-chat-7b_hf/ceval-high_school_chinese,internlm-chat-7b_hf/ceval-high_school_history,internlm-chat-7b_hf/ceval-middle_school_history,internlm-chat-7b_hf/ceval-civil_servant,internlm-chat-7b_hf/ceval-sports_science,internlm-chat-7b_hf/ceval-plant_protection,internlm-chat-7b_hf/ceval-basic_medicine,internlm-chat-7b_hf/ceval-clinical_medicine,internlm-chat-7b_hf/ceval-urban_and_rural_planner,internlm-chat-7b_hf/ceval-accountant,internlm-chat-7b_hf/ceval-fire_engineer,internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer,internlm-chat-7b_hf/ceval-tax_accountant,internlm-chat-7b_hf/ceval-physician]
Loading checkpoint shards:   0%|                                                                            | 0/8 [00:00<?, ?it/s]/home/aistudio/.local/lib/python3.10/site-packages/torch/_utils.py:776: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
  return self.fget.__get__(instance, owner)()
Loading checkpoint shards: 100%|████████████████████████████████████████████████████████████████████| 8/8 [00:25<00:00,  3.16s/it]
07/28 12:35:35 - OpenCompass - INFO - using stop words: ['</s>']
07/28 12:35:37 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-computer_network]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 372391.48it/s]
[2024-07-28 12:35:37,642] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
  0%|                                                                                                       | 0/3 [00:00<?, ?it/s]No chat template is set for this tokenizer, falling back to a default class-level template. This is very error-prone, because models are often trained with templates different from the class default! Default chat templates are a legacy feature and will be removed in Transformers v4.43, at which point any code depending on them will stop working. We recommend setting a valid chat template before then to ensure that this model continues working without issues.
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:20<00:00,  6.90s/it]
07/28 12:35:58 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-operating_system]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 205391.18it/s]
[2024-07-28 12:35:58,460] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:28<00:00,  9.46s/it]
07/28 12:36:26 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-computer_architecture]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 564617.85it/s]
[2024-07-28 12:36:26,914] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:25<00:00,  8.38s/it]
07/28 12:36:52 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-college_programming]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:00<00:00, 403088.96it/s]
[2024-07-28 12:36:52,212] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [01:20<00:00, 16.04s/it]
07/28 12:38:12 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-college_physics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 527760.11it/s]
[2024-07-28 12:38:12,486] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:25<00:00,  8.54s/it]
07/28 12:38:38 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-college_chemistry]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 599186.29it/s]
[2024-07-28 12:38:38,171] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:20<00:00,  6.69s/it]
07/28 12:38:58 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-advanced_mathematics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 527760.11it/s]
[2024-07-28 12:38:58,309] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:42<00:00, 14.26s/it]
07/28 12:39:41 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-probability_and_statistics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:00<00:00, 426539.39it/s]
[2024-07-28 12:39:41,151] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:57<00:00, 19.31s/it]
07/28 12:40:39 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-discrete_mathematics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 16/16 [00:00<00:00, 459649.75it/s]
[2024-07-28 12:40:39,138] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:24<00:00, 12.23s/it]
07/28 12:41:03 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-electrical_engineer]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 37/37 [00:00<00:00, 902263.07it/s]
[2024-07-28 12:41:03,694] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:28<00:00,  5.74s/it]
07/28 12:41:32 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-metrology_engineer]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 424739.65it/s]
[2024-07-28 12:41:32,440] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:44<00:00, 14.98s/it]
07/28 12:42:17 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_mathematics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 18/18 [00:00<00:00, 480875.62it/s]
[2024-07-28 12:42:17,438] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:43<00:00, 14.49s/it]
07/28 12:43:00 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_physics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 545834.08it/s]
[2024-07-28 12:43:00,959] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:23<00:00,  7.71s/it]
07/28 12:43:24 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_chemistry]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 524288.00it/s]
[2024-07-28 12:43:24,159] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:15<00:00,  5.25s/it]
07/28 12:43:39 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_biology]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 501206.14it/s]
[2024-07-28 12:43:39,971] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00,  1.62s/it]
07/28 12:43:44 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_mathematics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 450236.02it/s]
[2024-07-28 12:43:44,896] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:40<00:00, 13.56s/it]
07/28 12:44:25 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_biology]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 550502.40it/s]
[2024-07-28 12:44:25,646] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:23<00:00,  7.89s/it]
07/28 12:44:49 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_physics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 520861.28it/s]
[2024-07-28 12:44:49,388] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:14<00:00,  4.74s/it]
07/28 12:45:03 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_chemistry]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 524288.00it/s]
[2024-07-28 12:45:03,673] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.93s/it]
07/28 12:45:09 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-veterinary_medicine]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 570822.44it/s]
[2024-07-28 12:45:09,549] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:19<00:00,  6.44s/it]
07/28 12:45:28 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-college_economics]
100%|████████████████████████████████████████████████████████████████████████████████████████| 55/55 [00:00<00:00, 1109070.77it/s]
[2024-07-28 12:45:29,000] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:57<00:00,  8.15s/it]
07/28 12:46:26 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-business_administration]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:00<00:00, 823881.14it/s]
[2024-07-28 12:46:26,140] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:12<00:00,  2.40s/it]
07/28 12:46:38 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-marxism]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 542120.93it/s]
[2024-07-28 12:46:38,228] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:24<00:00,  8.09s/it]
07/28 12:47:02 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-mao_zedong_thought]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 625237.86it/s]
[2024-07-28 12:47:02,574] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:25<00:00,  8.37s/it]
07/28 12:47:27 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-education_science]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:00<00:00, 691106.91it/s]
[2024-07-28 12:47:27,759] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:28<00:00,  7.19s/it]
07/28 12:47:56 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-teacher_qualification]
100%|████████████████████████████████████████████████████████████████████████████████████████| 44/44 [00:00<00:00, 1036794.25it/s]
[2024-07-28 12:47:56,623] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:40<00:00,  6.72s/it]
07/28 12:48:36 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_politics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 520861.28it/s]
[2024-07-28 12:48:36,982] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:43<00:00, 14.53s/it]
07/28 12:49:20 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_geography]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 514140.49it/s]
[2024-07-28 12:49:20,634] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:14<00:00,  4.86s/it]
07/28 12:49:35 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_politics]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 21/21 [00:00<00:00, 611669.33it/s]
[2024-07-28 12:49:35,268] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:20<00:00,  6.91s/it]
07/28 12:49:56 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_geography]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 12/12 [00:00<00:00, 396312.19it/s]
[2024-07-28 12:49:56,030] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 2/2 [00:16<00:00,  8.36s/it]
07/28 12:50:12 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-modern_chinese_history]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 527152.96it/s]
[2024-07-28 12:50:12,824] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:17<00:00,  5.93s/it]
07/28 12:50:30 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 534844.13it/s]
[2024-07-28 12:50:30,689] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:27<00:00,  9.03s/it]
07/28 12:50:57 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-logic]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:00<00:00, 623477.62it/s]
[2024-07-28 12:50:57,831] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:49<00:00, 16.53s/it]
07/28 12:51:47 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-law]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 24/24 [00:00<00:00, 588674.25it/s]
[2024-07-28 12:51:47,505] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:17<00:00,  5.86s/it]
07/28 12:52:05 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-chinese_language_and_literature]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 588225.56it/s]
[2024-07-28 12:52:05,155] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:20<00:00,  6.97s/it]
07/28 12:52:26 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-art_studies]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 33/33 [00:00<00:00, 777595.69it/s]
[2024-07-28 12:52:26,147] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:26<00:00,  5.22s/it]
07/28 12:52:52 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-professional_tour_guide]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 29/29 [00:00<00:00, 715498.92it/s]
[2024-07-28 12:52:52,330] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:25<00:00,  6.46s/it]
07/28 12:53:18 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-legal_professional]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 23/23 [00:00<00:00, 442518.31it/s]
[2024-07-28 12:53:18,250] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:39<00:00, 13.05s/it]
07/28 12:53:57 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_chinese]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 517479.06it/s]
[2024-07-28 12:53:57,459] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:38<00:00, 12.98s/it]
07/28 12:54:36 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-high_school_history]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 20/20 [00:00<00:00, 521031.55it/s]
[2024-07-28 12:54:36,459] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:05<00:00,  1.69s/it]
07/28 12:54:41 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-middle_school_history]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:00<00:00, 566102.38it/s]
[2024-07-28 12:54:41,584] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:04<00:00,  1.55s/it]
07/28 12:54:46 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-civil_servant]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 47/47 [00:00<00:00, 887983.28it/s]
[2024-07-28 12:54:46,354] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [01:20<00:00, 13.41s/it]
07/28 12:56:06 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-sports_science]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 520861.28it/s]
[2024-07-28 12:56:06,906] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:18<00:00,  6.13s/it]
07/28 12:56:25 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-plant_protection]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:00<00:00, 580343.95it/s]
[2024-07-28 12:56:25,349] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:19<00:00,  6.64s/it]
07/28 12:56:45 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-basic_medicine]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 19/19 [00:00<00:00, 573322.13it/s]
[2024-07-28 12:56:45,325] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:16<00:00,  5.41s/it]
07/28 12:57:01 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-clinical_medicine]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 22/22 [00:00<00:00, 470789.22it/s]
[2024-07-28 12:57:01,618] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:26<00:00,  8.70s/it]
07/28 12:57:27 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-urban_and_rural_planner]
100%|████████████████████████████████████████████████████████████████████████████████████████| 46/46 [00:00<00:00, 1042908.02it/s]
[2024-07-28 12:57:27,827] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:34<00:00,  5.73s/it]
07/28 12:58:02 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-accountant]
100%|████████████████████████████████████████████████████████████████████████████████████████| 49/49 [00:00<00:00, 1093196.26it/s]
[2024-07-28 12:58:02,310] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:20<00:00,  2.99s/it]
07/28 12:58:23 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-fire_engineer]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 769369.37it/s]
[2024-07-28 12:58:23,330] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:18<00:00,  4.59s/it]
07/28 12:58:41 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer]
100%|█████████████████████████████████████████████████████████████████████████████████████████| 31/31 [00:00<00:00, 769369.37it/s]
[2024-07-28 12:58:41,786] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:14<00:00,  3.51s/it]
07/28 12:58:55 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-tax_accountant]
100%|████████████████████████████████████████████████████████████████████████████████████████| 49/49 [00:00<00:00, 1093196.26it/s]
[2024-07-28 12:58:56,222] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:14<00:00,  2.02s/it]
07/28 12:59:10 - OpenCompass - INFO - Start inferencing [internlm-chat-7b_hf/ceval-physician]
100%|████████████████████████████████████████████████████████████████████████████████████████| 49/49 [00:00<00:00, 1059386.06it/s]
[2024-07-28 12:59:10,483] [opencompass.openicl.icl_inferencer.icl_gen_inferencer] [INFO] Starting inference process...
100%|███████████████████████████████████████████████████████████████████████████████████████████████| 7/7 [00:23<00:00,  3.42s/it]
07/28 12:59:34 - OpenCompass - INFO - time elapsed: 1471.07s
07/28 12:59:37 - OpenCompass - DEBUG - Get class `NaivePartitioner` from "partitioner" registry in "opencompass"
07/28 12:59:37 - OpenCompass - DEBUG - An `NaivePartitioner` instance is built from registry, and its implementation can be found in opencompass.partitioners.naive
07/28 12:59:37 - OpenCompass - DEBUG - Key eval.runner.task.judge_cfg not found in config, ignored.
07/28 12:59:37 - OpenCompass - DEBUG - Key eval.runner.task.dump_details not found in config, ignored.
07/28 12:59:37 - OpenCompass - DEBUG - Key eval.given_pred not found in config, ignored.
07/28 12:59:37 - OpenCompass - DEBUG - Additional config: {'eval': {'runner': {'task': {}}}}
07/28 12:59:37 - OpenCompass - INFO - Partitioned into 52 tasks.
07/28 12:59:37 - OpenCompass - DEBUG - Task 0: [internlm-chat-7b_hf/ceval-computer_network]
07/28 12:59:37 - OpenCompass - DEBUG - Task 1: [internlm-chat-7b_hf/ceval-operating_system]
07/28 12:59:37 - OpenCompass - DEBUG - Task 2: [internlm-chat-7b_hf/ceval-computer_architecture]
07/28 12:59:37 - OpenCompass - DEBUG - Task 3: [internlm-chat-7b_hf/ceval-college_programming]
07/28 12:59:37 - OpenCompass - DEBUG - Task 4: [internlm-chat-7b_hf/ceval-college_physics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 5: [internlm-chat-7b_hf/ceval-college_chemistry]
07/28 12:59:37 - OpenCompass - DEBUG - Task 6: [internlm-chat-7b_hf/ceval-advanced_mathematics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 7: [internlm-chat-7b_hf/ceval-probability_and_statistics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 8: [internlm-chat-7b_hf/ceval-discrete_mathematics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 9: [internlm-chat-7b_hf/ceval-electrical_engineer]
07/28 12:59:37 - OpenCompass - DEBUG - Task 10: [internlm-chat-7b_hf/ceval-metrology_engineer]
07/28 12:59:37 - OpenCompass - DEBUG - Task 11: [internlm-chat-7b_hf/ceval-high_school_mathematics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 12: [internlm-chat-7b_hf/ceval-high_school_physics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 13: [internlm-chat-7b_hf/ceval-high_school_chemistry]
07/28 12:59:37 - OpenCompass - DEBUG - Task 14: [internlm-chat-7b_hf/ceval-high_school_biology]
07/28 12:59:37 - OpenCompass - DEBUG - Task 15: [internlm-chat-7b_hf/ceval-middle_school_mathematics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 16: [internlm-chat-7b_hf/ceval-middle_school_biology]
07/28 12:59:37 - OpenCompass - DEBUG - Task 17: [internlm-chat-7b_hf/ceval-middle_school_physics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 18: [internlm-chat-7b_hf/ceval-middle_school_chemistry]
07/28 12:59:37 - OpenCompass - DEBUG - Task 19: [internlm-chat-7b_hf/ceval-veterinary_medicine]
07/28 12:59:37 - OpenCompass - DEBUG - Task 20: [internlm-chat-7b_hf/ceval-college_economics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 21: [internlm-chat-7b_hf/ceval-business_administration]
07/28 12:59:37 - OpenCompass - DEBUG - Task 22: [internlm-chat-7b_hf/ceval-marxism]
07/28 12:59:37 - OpenCompass - DEBUG - Task 23: [internlm-chat-7b_hf/ceval-mao_zedong_thought]
07/28 12:59:37 - OpenCompass - DEBUG - Task 24: [internlm-chat-7b_hf/ceval-education_science]
07/28 12:59:37 - OpenCompass - DEBUG - Task 25: [internlm-chat-7b_hf/ceval-teacher_qualification]
07/28 12:59:37 - OpenCompass - DEBUG - Task 26: [internlm-chat-7b_hf/ceval-high_school_politics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 27: [internlm-chat-7b_hf/ceval-high_school_geography]
07/28 12:59:37 - OpenCompass - DEBUG - Task 28: [internlm-chat-7b_hf/ceval-middle_school_politics]
07/28 12:59:37 - OpenCompass - DEBUG - Task 29: [internlm-chat-7b_hf/ceval-middle_school_geography]
07/28 12:59:37 - OpenCompass - DEBUG - Task 30: [internlm-chat-7b_hf/ceval-modern_chinese_history]
07/28 12:59:37 - OpenCompass - DEBUG - Task 31: [internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation]
07/28 12:59:37 - OpenCompass - DEBUG - Task 32: [internlm-chat-7b_hf/ceval-logic]
07/28 12:59:37 - OpenCompass - DEBUG - Task 33: [internlm-chat-7b_hf/ceval-law]
07/28 12:59:37 - OpenCompass - DEBUG - Task 34: [internlm-chat-7b_hf/ceval-chinese_language_and_literature]
07/28 12:59:37 - OpenCompass - DEBUG - Task 35: [internlm-chat-7b_hf/ceval-art_studies]
07/28 12:59:37 - OpenCompass - DEBUG - Task 36: [internlm-chat-7b_hf/ceval-professional_tour_guide]
07/28 12:59:37 - OpenCompass - DEBUG - Task 37: [internlm-chat-7b_hf/ceval-legal_professional]
07/28 12:59:37 - OpenCompass - DEBUG - Task 38: [internlm-chat-7b_hf/ceval-high_school_chinese]
07/28 12:59:37 - OpenCompass - DEBUG - Task 39: [internlm-chat-7b_hf/ceval-high_school_history]
07/28 12:59:37 - OpenCompass - DEBUG - Task 40: [internlm-chat-7b_hf/ceval-middle_school_history]
07/28 12:59:37 - OpenCompass - DEBUG - Task 41: [internlm-chat-7b_hf/ceval-civil_servant]
07/28 12:59:37 - OpenCompass - DEBUG - Task 42: [internlm-chat-7b_hf/ceval-sports_science]
07/28 12:59:37 - OpenCompass - DEBUG - Task 43: [internlm-chat-7b_hf/ceval-plant_protection]
07/28 12:59:37 - OpenCompass - DEBUG - Task 44: [internlm-chat-7b_hf/ceval-basic_medicine]
07/28 12:59:37 - OpenCompass - DEBUG - Task 45: [internlm-chat-7b_hf/ceval-clinical_medicine]
07/28 12:59:37 - OpenCompass - DEBUG - Task 46: [internlm-chat-7b_hf/ceval-urban_and_rural_planner]
07/28 12:59:37 - OpenCompass - DEBUG - Task 47: [internlm-chat-7b_hf/ceval-accountant]
07/28 12:59:37 - OpenCompass - DEBUG - Task 48: [internlm-chat-7b_hf/ceval-fire_engineer]
07/28 12:59:37 - OpenCompass - DEBUG - Task 49: [internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer]
07/28 12:59:37 - OpenCompass - DEBUG - Task 50: [internlm-chat-7b_hf/ceval-tax_accountant]
07/28 12:59:37 - OpenCompass - DEBUG - Task 51: [internlm-chat-7b_hf/ceval-physician]
07/28 12:59:37 - OpenCompass - DEBUG - Get class `LocalRunner` from "runner" registry in "opencompass"
07/28 12:59:37 - OpenCompass - DEBUG - An `LocalRunner` instance is built from registry, and its implementation can be found in opencompass.runners.local
07/28 12:59:37 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:37 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:40 - OpenCompass - DEBUG - Modules of opencompass's load_dataset registry have been automatically imported from opencompass.datasets
07/28 12:59:40 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:40 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:40 - OpenCompass - DEBUG - Modules of opencompass's text_postprocessors registry have been automatically imported from opencompass.utils.text_postprocessors
07/28 12:59:40 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:40 - OpenCompass - DEBUG - Modules of opencompass's icl_evaluators registry have been automatically imported from opencompass.openicl.icl_evaluator
07/28 12:59:40 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:40 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:40 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-computer_network]: {'accuracy': 42.10526315789473}
07/28 12:59:40 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:40 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:43 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:43 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:43 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:43 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:43 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:43 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-operating_system]: {'accuracy': 36.84210526315789}
07/28 12:59:43 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:43 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:46 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:46 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:46 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:46 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:46 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:46 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-computer_architecture]: {'accuracy': 42.857142857142854}
07/28 12:59:46 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:46 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:48 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:48 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:48 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:48 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:48 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:48 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-college_programming]: {'accuracy': 43.24324324324324}
07/28 12:59:48 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:48 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:51 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:51 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:51 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:51 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:51 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:51 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-college_physics]: {'accuracy': 42.10526315789473}
07/28 12:59:51 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:51 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:54 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:54 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:54 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:54 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:54 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:54 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-college_chemistry]: {'accuracy': 16.666666666666664}
07/28 12:59:54 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:54 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:57 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:57 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:57 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:57 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:57 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:57 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-advanced_mathematics]: {'accuracy': 26.31578947368421}
07/28 12:59:57 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:57 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 12:59:59 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 12:59:59 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 12:59:59 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 12:59:59 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 12:59:59 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 12:59:59 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-probability_and_statistics]: {'accuracy': 38.88888888888889}
07/28 12:59:59 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 12:59:59 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:02 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:02 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:02 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:02 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:02 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:02 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-discrete_mathematics]: {'accuracy': 18.75}
07/28 13:00:02 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:02 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:05 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:05 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:05 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:05 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:05 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:05 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-electrical_engineer]: {'accuracy': 43.24324324324324}
07/28 13:00:05 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:05 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:07 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:08 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:08 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:08 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:08 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:08 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-metrology_engineer]: {'accuracy': 37.5}
07/28 13:00:08 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:08 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:10 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:10 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:10 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:10 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:10 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:10 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_mathematics]: {'accuracy': 27.77777777777778}
07/28 13:00:10 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:10 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:13 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:13 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:13 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:13 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:13 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:13 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_physics]: {'accuracy': 42.10526315789473}
07/28 13:00:13 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:13 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:16 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:16 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:16 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:16 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:16 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:16 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_chemistry]: {'accuracy': 63.1578947368421}
07/28 13:00:16 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:16 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:19 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:19 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:19 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:19 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:19 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:19 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_biology]: {'accuracy': 42.10526315789473}
07/28 13:00:19 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:19 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:21 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:22 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:22 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:22 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:22 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:22 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_mathematics]: {'accuracy': 31.57894736842105}
07/28 13:00:22 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:22 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:24 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:24 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:24 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:24 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:24 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:24 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_biology]: {'accuracy': 66.66666666666666}
07/28 13:00:24 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:24 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:27 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:27 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:27 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:27 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:27 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:27 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_physics]: {'accuracy': 68.42105263157895}
07/28 13:00:27 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:27 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:30 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:30 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:30 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:30 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:30 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:30 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_chemistry]: {'accuracy': 70.0}
07/28 13:00:30 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:30 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:33 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:33 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:33 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:33 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:33 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:33 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-veterinary_medicine]: {'accuracy': 47.82608695652174}
07/28 13:00:33 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:33 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:36 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:36 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:36 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:36 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:36 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:36 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-college_economics]: {'accuracy': 38.18181818181819}
07/28 13:00:36 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:36 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:38 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:38 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:38 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:38 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:38 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:38 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-business_administration]: {'accuracy': 42.42424242424242}
07/28 13:00:38 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:38 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:41 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:41 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:41 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:41 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:41 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:41 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-marxism]: {'accuracy': 73.68421052631578}
07/28 13:00:41 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:41 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:44 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:44 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:44 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:44 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:44 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:44 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-mao_zedong_thought]: {'accuracy': 70.83333333333334}
07/28 13:00:44 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:44 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:46 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:46 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:46 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:46 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:46 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:46 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-education_science]: {'accuracy': 62.06896551724138}
07/28 13:00:46 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:46 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:49 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:49 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:49 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:49 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:49 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:49 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-teacher_qualification]: {'accuracy': 68.18181818181817}
07/28 13:00:49 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:49 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:52 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:52 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:52 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:52 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:52 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:52 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_politics]: {'accuracy': 94.73684210526315}
07/28 13:00:52 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:52 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:55 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:55 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:55 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:55 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:55 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:55 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_geography]: {'accuracy': 52.63157894736842}
07/28 13:00:55 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:55 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:00:58 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:00:58 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:00:58 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:00:58 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:00:58 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:00:58 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_politics]: {'accuracy': 76.19047619047619}
07/28 13:00:58 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:00:58 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:00 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:00 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:00 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:00 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:00 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:01 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_geography]: {'accuracy': 83.33333333333334}
07/28 13:01:01 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:01 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:03 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:03 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:03 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:03 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:03 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:03 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-modern_chinese_history]: {'accuracy': 78.26086956521739}
07/28 13:01:03 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:03 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:06 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:06 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:06 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:06 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:06 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:06 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-ideological_and_moral_cultivation]: {'accuracy': 68.42105263157895}
07/28 13:01:06 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:06 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:09 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:09 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:09 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:09 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:09 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:09 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-logic]: {'accuracy': 50.0}
07/28 13:01:09 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:09 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:12 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:12 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:12 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:12 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:12 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:12 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-law]: {'accuracy': 20.833333333333336}
07/28 13:01:12 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:12 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:15 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:15 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:15 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:15 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:15 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:15 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-chinese_language_and_literature]: {'accuracy': 43.47826086956522}
07/28 13:01:15 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:15 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:17 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:17 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:17 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:17 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:17 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:17 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-art_studies]: {'accuracy': 57.57575757575758}
07/28 13:01:17 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:17 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:20 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:20 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:20 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:20 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:20 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:20 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-professional_tour_guide]: {'accuracy': 58.620689655172406}
07/28 13:01:20 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:20 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:23 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:23 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:23 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:23 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:23 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:23 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-legal_professional]: {'accuracy': 47.82608695652174}
07/28 13:01:23 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:23 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:26 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:26 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:26 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:26 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:26 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:26 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_chinese]: {'accuracy': 57.89473684210527}
07/28 13:01:26 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:26 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:28 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:28 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:28 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:28 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:28 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:29 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-high_school_history]: {'accuracy': 70.0}
07/28 13:01:29 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:29 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:31 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:31 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:31 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:31 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:31 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:31 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-middle_school_history]: {'accuracy': 77.27272727272727}
07/28 13:01:31 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:31 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:34 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:34 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:34 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:34 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:34 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:34 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-civil_servant]: {'accuracy': 55.319148936170215}
07/28 13:01:34 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:34 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:37 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:37 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:37 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:37 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:37 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:37 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-sports_science]: {'accuracy': 68.42105263157895}
07/28 13:01:37 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:37 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:40 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:40 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:40 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:40 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:40 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:40 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-plant_protection]: {'accuracy': 63.63636363636363}
07/28 13:01:40 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:40 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:43 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:43 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:43 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:43 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:43 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:43 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-basic_medicine]: {'accuracy': 57.89473684210527}
07/28 13:01:43 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:43 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:45 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:45 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:45 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:45 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:45 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:45 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-clinical_medicine]: {'accuracy': 63.63636363636363}
07/28 13:01:45 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:45 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:48 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:48 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:48 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:48 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:48 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:48 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-urban_and_rural_planner]: {'accuracy': 50.0}
07/28 13:01:48 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:48 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:51 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:51 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:51 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:51 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:51 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:51 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-accountant]: {'accuracy': 34.69387755102041}
07/28 13:01:51 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:51 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:54 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:54 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:54 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:54 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:54 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:54 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-fire_engineer]: {'accuracy': 41.935483870967744}
07/28 13:01:54 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:54 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:56 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:56 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:56 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:56 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:56 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:56 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-environmental_impact_assessment_engineer]: {'accuracy': 48.38709677419355}
07/28 13:01:56 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:56 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:01:59 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:01:59 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:01:59 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:01:59 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:01:59 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:01:59 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-tax_accountant]: {'accuracy': 53.06122448979592}
07/28 13:01:59 - OpenCompass - DEBUG - Get class `OpenICLEvalTask` from "task" registry in "opencompass"
07/28 13:01:59 - OpenCompass - DEBUG - An `OpenICLEvalTask` instance is built from registry, and its implementation can be found in opencompass.tasks.openicl_eval
07/28 13:02:02 - OpenCompass - DEBUG - Get class `CEvalDataset` from "load_dataset" registry in "opencompass"
07/28 13:02:02 - OpenCompass - DEBUG - An `CEvalDataset` instance is built from registry, and its implementation can be found in opencompass.datasets.ceval
07/28 13:02:02 - OpenCompass - DEBUG - Get class `first_capital_postprocess` from "text_postprocessors" registry in "opencompass"
07/28 13:02:02 - OpenCompass - DEBUG - Get class `AccEvaluator` from "icl_evaluators" registry in "opencompass"
07/28 13:02:02 - OpenCompass - DEBUG - An `AccEvaluator` instance is built from registry, and its implementation can be found in opencompass.openicl.icl_evaluator.icl_hf_evaluator
07/28 13:02:02 - OpenCompass - INFO - Task [internlm-chat-7b_hf/ceval-physician]: {'accuracy': 48.97959183673469}
07/28 13:02:02 - OpenCompass - DEBUG - An `DefaultSummarizer` instance is built from registry, and its implementation can be found in opencompass.summarizers.default
dataset                                         version    metric         mode      internlm-chat-7b_hf
----------------------------------------------  ---------  -------------  ------  ---------------------
ceval-computer_network                          db9ce2     accuracy       gen                     42.11
ceval-operating_system                          1c2571     accuracy       gen                     36.84
ceval-computer_architecture                     a74dad     accuracy       gen                     42.86
ceval-college_programming                       4ca32a     accuracy       gen                     43.24
ceval-college_physics                           963fa8     accuracy       gen                     42.11
ceval-college_chemistry                         e78857     accuracy       gen                     16.67
ceval-advanced_mathematics                      ce03e2     accuracy       gen                     26.32
ceval-probability_and_statistics                65e812     accuracy       gen                     38.89
ceval-discrete_mathematics                      e894ae     accuracy       gen                     18.75
ceval-electrical_engineer                       ae42b9     accuracy       gen                     43.24
ceval-metrology_engineer                        ee34ea     accuracy       gen                     37.50
ceval-high_school_mathematics                   1dc5bf     accuracy       gen                     27.78
ceval-high_school_physics                       adf25f     accuracy       gen                     42.11
ceval-high_school_chemistry                     2ed27f     accuracy       gen                     63.16
ceval-high_school_biology                       8e2b9a     accuracy       gen                     42.11
ceval-middle_school_mathematics                 bee8d5     accuracy       gen                     31.58
ceval-middle_school_biology                     86817c     accuracy       gen                     66.67
ceval-middle_school_physics                     8accf6     accuracy       gen                     68.42
ceval-middle_school_chemistry                   167a15     accuracy       gen                     70.00
ceval-veterinary_medicine                       b4e08d     accuracy       gen                     47.83
ceval-college_economics                         f3f4e6     accuracy       gen                     38.18
ceval-business_administration                   c1614e     accuracy       gen                     42.42
ceval-marxism                                   cf874c     accuracy       gen                     73.68
ceval-mao_zedong_thought                        51c7a4     accuracy       gen                     70.83
ceval-education_science                         591fee     accuracy       gen                     62.07
ceval-teacher_qualification                     4e4ced     accuracy       gen                     68.18
ceval-high_school_politics                      5c0de2     accuracy       gen                     94.74
ceval-high_school_geography                     865461     accuracy       gen                     52.63
ceval-middle_school_politics                    5be3e7     accuracy       gen                     76.19
ceval-middle_school_geography                   8a63be     accuracy       gen                     83.33
ceval-modern_chinese_history                    fc01af     accuracy       gen                     78.26
ceval-ideological_and_moral_cultivation         a2aa4a     accuracy       gen                     68.42
ceval-logic                                     f5b022     accuracy       gen                     50.00
ceval-law                                       a110a1     accuracy       gen                     20.83
ceval-chinese_language_and_literature           0f8b68     accuracy       gen                     43.48
ceval-art_studies                               2a1300     accuracy       gen                     57.58
ceval-professional_tour_guide                   4e673e     accuracy       gen                     58.62
ceval-legal_professional                        ce8787     accuracy       gen                     47.83
ceval-high_school_chinese                       315705     accuracy       gen                     57.89
ceval-high_school_history                       7eb30a     accuracy       gen                     70.00
ceval-middle_school_history                     48ab4a     accuracy       gen                     77.27
ceval-civil_servant                             87d061     accuracy       gen                     55.32
ceval-sports_science                            70f27b     accuracy       gen                     68.42
ceval-plant_protection                          8941f9     accuracy       gen                     63.64
ceval-basic_medicine                            c409d6     accuracy       gen                     57.89
ceval-clinical_medicine                         49e82d     accuracy       gen                     63.64
ceval-urban_and_rural_planner                   95b885     accuracy       gen                     50.00
ceval-accountant                                002837     accuracy       gen                     34.69
ceval-fire_engineer                             bc23f5     accuracy       gen                     41.94
ceval-environmental_impact_assessment_engineer  c64e2d     accuracy       gen                     48.39
ceval-tax_accountant                            3a5e3c     accuracy       gen                     53.06
ceval-physician                                 6e277d     accuracy       gen                     48.98
ceval-stem                                      -          naive_average  gen                     42.41
ceval-social-science                            -          naive_average  gen                     66.23
ceval-humanities                                -          naive_average  gen                     57.29
ceval-other                                     -          naive_average  gen                     53.27
ceval-hard                                      -          naive_average  gen                     34.47
ceval                                           -          naive_average  gen                     52.43
07/28 13:02:02 - OpenCompass - INFO - write summary to /home/aistudio/opencompass/outputs/default/20240728_123445/summary/summary_20240728_123445.txt
07/28 13:02:02 - OpenCompass - INFO - write csv to /home/aistudio/opencompass/outputs/default/20240728_123445/summary/summary_20240728_123445.csv

0 replies

liuwake · 2024-07-28T05:40:13Z

liuwake
Jul 28, 2024
Author

我也搜索了有关本问题的其他相关讨论,但是很多对于报错语焉不详,输出日志也不全,我不确定是不是跟我反馈的问题是一样的.
也看了另外一些出现No predictions found的issue,能大体确认与我的情况不相同:主要表现在
- 他们的日志出现了滚动条有推理过程,但是最后没有结果,输出No predictions found.
- 或者直接用windows,而不兼容
- etc.
我用的目前最新的代码20240726的[Feature] Update CHARM Memeorziation .我简单阅读了代码,发现实现推理和评测都是用shell异步调用torchrun.这会导致一个黑箱的问题,也就是bash返回的错误用户即便开启--debug可能也无法获得到———最终只能看到No predictions found字样.这会导致debug困难的提升.不知道有什么好办法可以捕获到bash输出的所有错误.

0 replies

tonysy · 2024-07-28T12:41:25Z

tonysy
Jul 28, 2024
Maintainer

Great. Would you like to try this solution?(for opencompass/runner/local.py)

                        tmp_logs = f'tmp/{os.getpid()}_debug.log'
                        get_logger().info(
                            f'Debug mode, log will be saved to {tmp_logs}')
                        with open(tmp_logs, 'w') as log_file:
                            subprocess.run(cmd, shell=True, text=True, stdout=log_file, stderr=subprocess.STDOUT)

1 reply

liuwake Jul 30, 2024
Author

Thank you. It helps.I added these codes and reproduced the bug.One line that the console gived attracted me: Debug mode, log will be saved to tmp/22331_debug.log. Then I

aistudio@jupyter-145988-8195042:~/opencompass/tmp$ cat 22331_debug.log 
/bin/sh: 1: torchrun: not found

That made the debugging more easier.

tonysy · 2024-07-31T16:48:06Z

tonysy
Jul 31, 2024
Maintainer

We have merged this feature into the latest version. Feel free to re-open if needed. Thanks again.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

我解决了torchrun环境异常情况下的No predictions found问题. #1370

{{title}}

Replies: 4 comments 1 reply

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

我解决了torchrun环境异常情况下的No predictions found问题. #1370

liuwake Jul 28, 2024

环境和配置

执行出错

分析问题

Replies: 4 comments · 1 reply

liuwake Jul 28, 2024 Author

liuwake Jul 28, 2024 Author

tonysy Jul 28, 2024 Maintainer

liuwake Jul 30, 2024 Author

tonysy Jul 31, 2024 Maintainer

liuwake
Jul 28, 2024

Replies: 4 comments 1 reply

liuwake
Jul 28, 2024
Author

liuwake
Jul 28, 2024
Author

tonysy
Jul 28, 2024
Maintainer

liuwake Jul 30, 2024
Author

tonysy
Jul 31, 2024
Maintainer