[AI Evaluation] Rationalize IEvaluationTokenCounter interface

[shyamnamboodiripad]

Currently IEvaluationTokenCounter has a single int CountTokens(string message) function that takes a string and returns a token count. Ideally, this would also support other modalities like images, audio etc. We may also want to make the function asynchronous (to support cases where IO may be needed).

The proposal is to change the function to CountTokensAsync(AIContent content, CancellationToken cancellationToken) with extension method overloads for counting tokens in IEnumerable<AIContent> and ChatMessage.

AIContent already models all supported modalities so it seems like an ideal candidate.

[shyamnamboodiripad]

FYI @peterwald @luisquintanilla @stephentoub

The main question that remains is whether it would make sense to generalize the above proposed token counter interface to use cases beyond the evaluation libraries, and move it to the MEAI.Abstractions layer. @luisquintanilla was going to follow up on this. If the answer is no, I can go ahead and make the proposed changes to IEvaluationTokenCounter before we GA the libraries.

[stephentoub]

cc: @SteveSandersonMS

[SteveSandersonMS]

I expect token counting is something we'd be open to including in MEAI in the longer term. Assuming MEAI usage grows, we'll reach a point where people demand ways of managing conversation history length, and token counting will be a prereq for that. I don't know exactly how that will look, but whatever you learn from IEvaluationTokenCounter could help with that.

However, unless we need this to be in MEAI right away (e.g., because introducing it later would be breaking), then I'd lean towards avoiding introducing new feature areas before GA and prioritize stabilization.

[shyamnamboodiripad]

Yes, the reason we had raised this a couple of months ago is because this could be a breaking change for MEAI.Eval if we end up introducing a better (more complete) token counting abstraction in MEAI (or as part of the ML.Tokenizers library) later😊

However, if the new MEAI abstraction we end up introducing (at some point post GA) happens to be close enough to what we end up GA-ing for IEvaluationTokenCounter, perhaps it would not be a huge problem. We can potentially provide some helpers to go back and forth between IEvaluationTokenCounter and the new interface at that point...