ADD GPT-4 as Judge

[philschmid]

No description provided.

[clefourrier]

LGTM, ideally you'd also want to add it in metrics

[philschmid]

ideally you'd also want to add it in metrics

Where?

[clefourrier]

src/lighteval/metrics/metrics.py, like these 2 atm

    llm_judge_multi_turn_openai = SampleLevelMetricGrouping(
        metric=["single_turn", "multi_turn"],
        higher_is_better=True,
        category=MetricCategory.LLM_AS_JUDGE_MULTI_TURN,
        use_case=MetricUseCase.SUMMARIZATION,
        sample_level_fn=JudgeLLM(
            judge_model_name="gpt-3.5-turbo",
            template_path=os.path.join(os.path.dirname(__file__), "judge_prompts.jsonl"),
            multi_turn=True,
        ).compute,
        corpus_level_fn={
            "single_turn": np.mean,
            "multi_turn": np.mean,
        },
    )
    llm_judge_openai = SampleLevelMetricGrouping(
        metric=["judge_score"],
        higher_is_better=True,
        category=MetricCategory.LLM_AS_JUDGE,
        use_case=MetricUseCase.SUMMARIZATION,
        sample_level_fn=JudgeLLM(
            judge_model_name="gpt-3.5-turbo",
            template_path=os.path.join(os.path.dirname(__file__), "", "judge_prompts.jsonl"),
            multi_turn=False,
        ).compute,
        corpus_level_fn={
            "judge_score": np.mean,
        },
    )

(name might be slightly different)

[clefourrier]

Basically it allows the judges you defined to be in the general metrics available to users