add 0 shot requests dumps for olmes

[gituser768]

This is a PR to merge into ot-oe-eval-requests (not main). I think this is the right way to be adding these tasks, let me know if i missed something @OyvindTafjord

[OyvindTafjord]

Thanks, this should be fine. We should be careful about not calling this "OLMES" though, since that should be specifically used about the 5-shot prompt. The 0-shot version is more to match what's already in-loop for some peace of mind. For that purpose it might be good to also add 0-shot versions of sciq and copa.

Then in addition can add 5-shot "OLMES" versions of these task (validation sets), both rc and mc (although might not activate all of these in most training runs), maybe with names like arc_easy_val_rc_olmes and arc_easy_val_mc_olmes.

[gituser768]

Ah ok, so your comment is specifically about {"regimes": ["OLMES-v0.1"]} in the metadata? I can just remove that I suppose (along with any "description"s that mention olmes specifically)

[OyvindTafjord]

No, it was more about the "olmes" in the PR title :) Not too worried about what's buried in the config.json files, fine to leave that as is I think.

[OyvindTafjord]

I'll merge this in and do some refactoring of the directories (also adding a script for easily adding these tasks to the label_to_task_map in OLMo). I'll also add the 5-shot versions at the same time.

[gituser768]

@OyvindTafjord if it's useful at all, here is the script I used to package up the request dumps: https://gist.github.com/dmh43/04973f9e2b5b4556d36416e63770da4c

might require some tweaking depending on the naming convention