is there any plan to support Tiktoken?

[Jasonsey]

I am trying to make mlc to support Qwen, but the model's tokenizer use tiktoken developed by openai, which is not supported by this repo now? So is there any plan for this feature?

[tqchen]

As of now we don't have planned effort yet, contribution is welcomed

[gaokao123]

@Jasonsey for tiktoken of qwen, Is there a solution now?

[Jasonsey]

@Jasonsey for tiktoken of qwen, Is there a solution now?

Currently, we can use the mlc chat to convert Qwen'tiktoken to this repo's tokenizer format.