Have datasets account for tokenizers which incorrectly define PAD - #2017

Merged

tdene merged 16 commits into

NVIDIA:mainfrom

tdene:tde/tokenizer_pad_warning

Oct 29, 2025

Commits on Sep 23, 2025

Add warning if PAD == EOS
tdene
committed
0bb064a
Minor change to restart pipeline
tdene
committed
f107b0d

Commits on Sep 24, 2025

Reimplement warnings/errors as discussed
tdene
committed
d2f318c
Fix typo
tdene
committed
bbb503b
Fix unnecessary import
tdene
committed
8e0d19a
Fix variable name overload
tdene
committed
a9eaa00
Only ignore pads if there is an actual collision
tdene
committed
86c71a1
Rename ignore_pad_collision_in_dataset
tdene
committed
59712c1
Update docstring
tdene
committed
0ea26f2
Rename argument to mask_ambiguous_pad_tokens
tdene
committed
5d1c1e3

Commits on Sep 25, 2025

Change error to warning
tdene
committed
7af4ac1

Commits on Sep 29, 2025

Rename argument to allow_ambiguous_pad_tokens
tdene
committed
8d25df5

Commits on Oct 15, 2025

Catch NotImplementedError
tdene
committed
3d2f077
Merge remote-tracking branch 'remotes/lm/main' into tde/tokenizer_pad_warning
tdene
committed
062fa08

Commits on Oct 18, 2025

Account for bad pad tokens in bert and t5 dataset
tdene
committed
0ff4679

Commits on Oct 29, 2025

Merge remote-tracking branch 'gh/main' into tde/tokenizer_pad_warning
tdene
committed
e3c08fb