MCore dataset compatibility for tokenizers#8390

Merged

pablo-garay merged 3 commits intoNVIDIA:r1.23.0from vysarge:vsarge/tokenizer_unique_identifiers

Feb 11, 2024

+58-8

Commits on Feb 9, 2024

Add unique_identifiers for all tokenizers and eod for SentencePieceTokenizer
vysarge
committed
Add generalized token aliases to TokenizerSpec to conform with MegatronTokenizer's interface. Remove now-redundant individual fixes from AutoTokenizer and SentencePieceTokenizer.
vysarge
committed

Commits on Feb 10, 2024

Merge branch 'r1.23.0' into vsarge/tokenizer_unique_identifiers
pablo-garay
authored