what should I do if I want to improve the performance of hellaswag?

[mathCrazyy]

I want to find some dataset , for example OpenO1, KD 14B to 3B， or use lora, but I have a bad result:

the result of KD only reach 96.8% of the ori 3B Qwen2.5 model
what should I do? Thanks.

[joecummings]

Did you fine-tune the 14B model on your desired dataset first? That's an important pre-step to knowledge distillation.

[mathCrazyy]

Sorry I didn't, I mistakenly thought it was not important.

[joecummings]

Sorry I didn't, I mistakenly thought it was not important.

All good - give that a go and LMK how it works after re-evaluating