adding special_tokens from tokenizer config for transformer-lm model

[clumsy]

What does this PR do ?

special_tokens are currently ignored when constructing a tokenizer for transformer_lm_model using the following example:

NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml

Line 51 in 5cb76a5

special_tokens: # only necessary for adding transformer/bert-specific special tokens to tokenizer if the tokenizer does not already have these inherently.

Collection: NLP/language-modeling

Changelog

• transformer_lm_model: use special_tokens from config when creating the tokenizer, making it backward-compatible with previous behavior (when special_tokens are not provided we proceed with None).

Usage

• NeMo/examples/nlp/language_modeling/conf/bert_pretraining_from_text_config.yaml

  Line 51 in 5cb76a5

  special_tokens: # only necessary for adding transformer/bert-specific special tokens to tokenizer if the tokenizer does not already have these inherently.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
• Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

If you haven't finished some of the above items you can still open "Draft" PR.

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

Additional Information

• Related to # (issue)

[wdykas]

@ericharper for visibility when you get the chance fix for using special tokens

[clumsy]

@ericharper , @wdykas is there anything I can do from my end to unblock this PR?

[clumsy]

Please let me know if there's anything I can do to get this PR approved, @wdykas @ericharper. Thanks!

[aklife97]

this PR should be good to go but we need to ensure we do not do this special token passing for sentencepiece (its fine for other tokenizer types). We know it causes different tokenization as compared to a standard pre-trained sentencepiece model and can cause hidden issues

The recommended way to add special tokens to sentencepiece is to directly add it to the tokenizer by editing it using something like: https://github.com/NVIDIA/NeMo/blob/main/scripts/tokenizers/add_special_tokens_to_sentencepiece.py

[ericharper]

jenkins

[ericharper]

@aklife97 are you recommending that we add a check in this PR?

[aklife97]

I think it should be good to merge, we need to think of a more general solution for this...