Fix finetuner attention masking.

[Eta0]

Attention Masks

All training configurations except for training a model in mixed precision mode ("fp16") with DeepSpeed enabled were broken, because the tokenized dataset loader was outputting its accompanying attention masks as all zeroes, meaning "ignore all training data."
A bug fixed in version 4.21.0 of the transformers library had allowed models with sufficiently extreme parameters to ignore the attention mask, which was the only way this code was previously able to function. Updating that dependency revealed this bug.

DeepSpeed's mixed precision mode seemingly reintroduced the bug that had previously allowed this code to function, which let it continue to sort of work under select circumstances even with the updated transformers library.

This PR fixes the attention masks for training and inference such that they mask away only the padding token ID.

Special Tokens

Attention masking is intended to prevent accidentally training the model on padding tokens. To generate correct attention masks at training time, it is necessary to know the pad_token ID used in the tokenizer.
The current implementation of the dataset tokenizer chooses the first available of:

1. An explicitly specified padding token string,
2. The model's default padding token, from the Hugging Face model config, or
3. <|endoftext|>

The default values for eos_token and pad_token in this code are now resolved in a matching way.

Additionally, the command line arguments --eot '' and --pad '' (the Argo Workflow's defaults) now refer to the aforementioned default-picking algorithm rather than being taken literally. Before, they were resolving to whatever special token the tokenizer interpreted '' to be.

Code Cleanup

Highlights:

• Explicitly casting models to half-precision as model.half() was removed, since it isn't right for mixed-precision training.
• no_init() was changed to a context manager that covers more steps of model instantiation.
• Logging was changed to consistently use units of mebibytes ("MiB").
  • It used to be three different (but similar) units in different parts of the code all labeled as "mb" ($10^{6}$ bytes, $2^{10} \times 10^{3}$ bytes, and $2^{20}$ bytes).
• Replaced usages of deprecated aliases in ds_config.json and in the logic to disable the WandB integration.

[wbrown]

Very nicely done. 👍