Adding long context benchmark MRCR

[fayejf]

OpenAI MRCR (Multi-round co-reference resolution): Long context multiple needle in a haystack benchmark

1. Prepare data
   By dafault it prepares all 2400 samples up to 1M tokens.

ns prepare_data \
    --data_dir=/workspace/ns-data \
    --cluster=fei-ord \
    mrcr

Or you can prepare subset.

ns prepare_data \
    --data_dir=/workspace/ns-data \
    --cluster=fei-ord \
    mrcr --max_context_window 131072 --needles_subset 2 --setup needle2_128k

2. Run evaluation
   Specific eval split or use what saved in __init__.py (default is all)

model=Meta-Llama-3.1-8B-Instruct
split=needle2_64k
ns eval \
    --cluster=fei-ord \
    --data_dir=/workspace/ns-data \
    --server_type=vllm \
    --model=/hf_models/$model \
    --server_gpus=8 \
    --benchmarks=mrcr:0 \
    --split=$split \
    --output_dir=/workspace/results/mrcr/split/$model 

[Kipok]

Thanks! Just a few small changes are needed

[Kipok]

please move it inside the function where it's needed. Otherwise this is going to run on every import even when the script isn't called

[fayejf]

Good point! changed.

[Kipok]

best not to override init dynamically here. Users can always provide --split argument to change this, so no need to change defaults

[fayejf]

Got it. Changed!

[Kipok]

Or is it supposed to use that messages list directly (so complete the last turn of a large multi-turn generation)? In that case you should put it as a list in "messages" key and then set ++prompt_format=openai in generation args

[fayejf]

Or is it supposed to use that messages list directly (so complete the last turn of a large multi-turn generation)? In that case you should put it as a list in "messages" key and then set ++prompt_format=openai in generation args

Wait do we support that? I wanted it but I didn't know.
I think people follow this way complete the last turn of a large multi-turn generation. But does this support for all models in nemo-skills?

messages = json.loads(row["prompt"])
completion = client.chat.completions.create(
    model=MODEL,
    messages=messages,
)
response = completion.choices[0].message.content

[Kipok]

yes, that should be supported with the parameters I shared

[Kipok]

Looks great, thanks!