Ability to acquire loss for next token generation

[ramkrishna2910]

The generate function from huggingface returns language modelling loss for next token generation. This loss helps calculating perplexity of the model.
Is there a known way to acquire loss from the generate function in onnxruntime_genai?

[yufenglee]

Will add an api to support this.

[ramkrishna2910]

That would be great! Thanks :)

[ramkrishna2910]

Thanks for making this a discussion.
Having onnxruntime_genai's generate function mirror the huggingface generate function would greatly improve ease of use.
For example, options like max_new_tokens are very useful.

[anutkk]

It's been implemented, although I'm not sure of how exactly this should be used:
#522