add a test checking the format of convert_tokens_to_string's output#16540

SaulLu merged 2 commits intohuggingface:mainhuggingface/transformers:mainfrom

SaulLu:LS/add_test_format_convert_tokens_to_stringSaulLu/transformers:LS/add_test_format_convert_tokens_to_stringCopy head branch name to clipboard