Add special tokens flag to C API tokenizer#940

Merged

sayanshaw24 merged 7 commits intomainfrom

sayanshaw/add-special-tokens

May 2, 2025

Commits on May 1, 2025

add special tokens flag to C API tokenizer
Sayan Shaw
committed
move to separate function for backward compatibility
Sayan Shaw
committed
add relevant python changes and unit test
Sayan Shaw
committed
more python changes
Sayan Shaw
committed
Merge branch 'main' of https://github.com/microsoft/onnxruntime-extensions into sayanshaw/add-special-tokens
Sayan Shaw
committed

Commits on May 2, 2025

final changes
Sayan Shaw
committed
trigger CLA check
Sayan Shaw
committed