Use the programmatic interface using an already in memory loaded model

[clefourrier]

To test the PR

import lighteval
import os
from datetime import timedelta
from transformers import AutoModelForCausalLM

from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.logging.hierarchical_logger import hlog_warn, htrack
from lighteval.models.model_config import create_model_config
from lighteval.pipeline import EnvConfig, ParallelismManager, Pipeline, PipelineParameters
from lighteval.utils.imports import is_accelerate_available, is_tgi_available

TOKEN = os.getenv("HF_TOKEN")
    
model = AutoModelForCausalLM.from_pretrained("trl-lib/Qwen2-0.5B-DPO")

env_config = EnvConfig(token=TOKEN, cache_dir="~/tmp")
evaluation_tracker = EvaluationTracker(
    output_dir="~/tmp",
    save_details=False,
    push_to_hub=False,
    push_to_tensorboard=False,
    public=False,
    hub_results_org=False,
)
pipeline_params = PipelineParameters(
    launcher_type=ParallelismManager.ACCELERATE,
    env_config=env_config,
    job_id=1,
    override_batch_size=1,
    num_fewshot_seeds=0,
    max_samples=10,
    use_chat_template=False,
)

pipeline = Pipeline(
    tasks="leaderboard|arc:challenge|0|0",
    pipeline_parameters=pipeline_params,
    evaluation_tracker=evaluation_tracker,
    model=model
)

pipeline.evaluate()

pipeline.show_results()

[clefourrier]

Ideally, we would want to migrate a bunch of params from the ModelParams to the PipelineParams:

• multichoice_continuations_start_space
• pairwise_tokenization
• add_special_tokens
• use_chat_template
• trust_remote_code
• batch_size

Would simplify model config code. Wdyt @NathanHB ? (Should probably be another PR)

[NathanHB]

Tested it locally and looks great !!

[clefourrier]

Cool! Waiting for some last feedback from Lewis and will merge afterwards