feat(tokenization): add encode_message to tokenize messages one by one#39507

Merged

ArthurZucker merged 12 commits intohuggingface:mainfrom

pco111:fix/issue-39417-incremental-tokenization

Jul 31, 2025

Commits on Jul 28, 2025

feat(tokenization): add encode_message to tokenize messages one by one

pco111
authored and
Rocketknight1
committed
Fix the `encode_message` method, remove the `add_generation_prompt` parameter and add the corresponding error handling. Update the document to reflect this change and verify the error handling in t…

pco111
authored and
Rocketknight1
committed
Optimize the `encode_message` method, improve the processing logic of the empty dialogue history, and ensure that the chat template can be applied correctly when the dialogue history is empty. Upda…

pco111
authored and
Rocketknight1
committed
The `_encode_message` method is deleted, the message coding logic is simplified, and the functional integrity of the `encode_message` method is ensured. Update the document to reflect these changes.

pco111
authored and
Rocketknight1
committed
Docs fix

pco111
authored and
Rocketknight1
committed
Revert changes in docstring of pad()

pco111
authored and
Rocketknight1
committed
Revert changes in docstring

pco111
authored and
Rocketknight1
committed

Commits on Jul 29, 2025

Commits on Jul 30, 2025

Merge branch 'main' into fix/issue-39417-incremental-tokenization
pco111
authored