Skip to content

Commit cc7aa33

Browse files
Muqi1029bvantuan
authored andcommitted
Explicitly setting encoding in tokenization_utils_base.py (huggingface#38553)
Update tokenization_utils_base.py Add encoding explicitly
1 parent 6e4892f commit cc7aa33

File tree

1 file changed

+1
-1
lines changed

1 file changed

+1
-1
lines changed

src/transformers/tokenization_utils_base.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2086,7 +2086,7 @@ def _from_pretrained(
20862086
chat_template_file = resolved_vocab_files.pop("chat_template_file", None)
20872087
extra_chat_templates = [key for key in resolved_vocab_files if key.startswith("chat_template_")]
20882088
if chat_template_file is not None:
2089-
with open(chat_template_file) as chat_template_handle:
2089+
with open(chat_template_file, encoding="utf-8") as chat_template_handle:
20902090
chat_templates["default"] = chat_template_handle.read()
20912091
for extra_chat_template in extra_chat_templates:
20922092
template_file = resolved_vocab_files.pop(extra_chat_template, None)

0 commit comments

Comments
 (0)