Adds config tempaltes #295
Conversation
There was a problem hiding this comment.
The templates folder mixes up a lot of items of different logic level, we'll have to disuss the structure IRL.
Do we really need "NLITaskConfig, CopaTaskConfig, BoolQATaskConfig", etc? I think it would be better to aim for a general structure instead of having lots of small ones
I did remove the BoolQATask, for the rest I think they have their place as they have some logic in them. Secondly I decided that I won't create a question-answer template as one can just use multichoice with cf to achieve just that |
hynky1999
force-pushed
the
config_templates
branch
from
78f3518 to
22eeddb
Compare
There was a problem hiding this comment.
Thanks for the PR!
Some comments:
- avoid too many layers of nested function: in
continuation, you nest 3 functions in one other function over about 100 lines of code, it's not super clear - in general, you also need to be careful to not change the naming conventions of the lib (for example, we use
choicesand you usedoptions), it would be best if you could rename to fit our naming convention. - other comments are mostly nits/need for doc/comment suggestions/...
| instruction: NotRequired[str] | ||
|
|
||
|
|
||
| # Python too dumb to do fancy inference :( |
There was a problem hiding this comment.
Remove or convert to actionable todo
| translation_literals = TRANSLATION_LITERALS[language] | ||
| adapter_fn = create_adapter_from_dict(adapter) if isinstance(adapter, dict) else adapter # type: ignore | ||
|
|
||
| def nli_natural_prompt(line: dict, task_name: str): |
There was a problem hiding this comment.
add docstring explaining the diff between the 2 prompt functions
There was a problem hiding this comment.
It should be clear from the format. The second function uses the natural function for CF formulation
| elif label == "contradiction": | ||
| return translation_literals.no | ||
| elif label == "neutral": | ||
| return translation_literals.also |
There was a problem hiding this comment.
That can't happen as long as you are respecting the typing, you don't that's on you
There was a problem hiding this comment.
I would not underestimate the creativity of some users ^^ and add an error message - but program will fail later anyway so up to you
There was a problem hiding this comment.
Well it's like calling load_datset(False), it's undefine behaviour, so it's user's problem.
| from lighteval.utils.utils import as_list | ||
|
|
||
|
|
||
| CONTINUATION_QUERY_CF = "{instruction}{context}" |
There was a problem hiding this comment.
Would be good to explain a bit what the diff is between continuation and multichoice
There was a problem hiding this comment.
I did so in the function docstring
Co-authored-by: Clémentine Fourrier <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>
Co-authored-by: Clémentine Fourrier <[email protected]>
This reverts commit b014b50.
3 participants
This is PR, which will bring easier way to task prompts, which are language-aware.
Following templates are create