Allow add_tokens for ESM#28535

Merged

Rocketknight1 merged 5 commits intomainfrom

allow_esm_add_tokens

Jan 19, 2024

Commits on Jan 16, 2024

Commits on Jan 17, 2024

Revert changes to id_to_token and token_to_id
Rocketknight1
committed

Commits on Jan 18, 2024

Update the ESM tokenizer to be a bit more standardized
Rocketknight1
committed

Commits on Jan 19, 2024

Update src/transformers/models/esm/tokenization_esm.py

Rocketknight1
and
ArthurZucker
authored