Add backwards-compatible support for multiple EOS tokens#305

Merged

hudson-ai merged 37 commits into

guidance-ai:mainfrom

hudson-ai:multi_eos

Mar 18, 2026

Commits on Mar 6, 2026

Commits on Mar 10, 2026

Commits on Mar 11, 2026

Use std::vector instead of new[]/delete[] in c_sample

hudson-ai
and
Copilot
committed
Take a single eos_tokens vector in create_tokenizer_v2

hudson-ai
and
Copilot
committed
Remove stale commented-out v2 snippet from create_tokenizer

hudson-ai
and
Copilot
committed

Commits on Mar 17, 2026