SentencePieceProcessor vs C [Enhancement]

[kris-jusiak]

To do the inference with just c and without sentence piece processor one easy way would be to save the id to token in the model.bin?

tokenizer = SentencePieceProcessor(tokenizer_model)
vocab = [tokenizer.id_to_piece(id) for id in range(tokenizer.get_piece_size())]

and then just an array to get the proper token from id

auto decode(auto id) {
  return vocab[id];
}

That would allow not to use the run_wrap.py and it would be in pure C (kinda).

[trholding]

Ref: https://github.com/google/sentencepiece/blob/master/src/spm_decode_main.cc

[karpathy]

Closing old issue. This has been implemented in run.c for a while with ASCII, and UTF-8 support is about to merge soon in #226 .