Transformers model as Judge #174
Conversation
|
Hi! Thanks for this PR! Can you fix your PR so that tests are passing? |
|
Yes, I'll fix on a short time but Run tests gives error for
Is it about huggingface hub? |
|
Hm, let me re-run your tests, maybe you committed when the hub was down |
|
Thanks, I fixed code quality and pushed. We can re-run whenever you want |
|
cc @NathanHB if you have the time to do a more in depth review |
| self.generation_args = { | ||
| "max_new_tokens": 500, | ||
| "return_full_text": False, | ||
| "temperature": temperature, |
There was a problem hiding this comment.
temperature is not needed if do_sample is set to False
| def __init__(self, judge_model_name: str, template_path: str, multi_turn: bool = False): | ||
| if judge_model_name not in self.available_models: | ||
| raise ValueError(f"{judge_model_name} not in available models for llm as a judge metric") | ||
| judge_type = "openai" |
There was a problem hiding this comment.
this wouldn't work, if we pass in gpt-12 for example, it will set the judge type to openai and continue, only to fail later because gpt-12 does not exist in the openai api
|
hey ! thanks for the fix, I will have the bandwitdh to test next week and will merge asap :) |
|
Hi ! Closing for inactivity. But this PR was used as the base for #223 thanks ! |
#153
Transformers library added to llm as judge.
Only need is when the JudgeLLM class called, change judge model with transformers.
Ex.