v0.1.1

What's Changed

• Add npm package by @tqchen in #3
• Update CMakeLists.txt by @songkq in #2
• Fix readme typo by @erjanmx in #4
• Fix windows build by linking against ntdll by @junrushao in #5
• Update CMakeLists.txt by @luiyen in #6
• Update rust to support latest tokenizers by @tqchen in #11
• add rwkv world tokenizer by @BBuf in #14
• msgpack as 3rdparty library instead of fetch_context by @BBuf in #15
• [FIX] Remove boost dependency from submodule msgpack by @Hzfengsy in #16
• Update to reduce memory realloc by @tqchen in #18
• fix rwkv world tokenzier by @BBuf in #19
• Update build.sh to make web tokenizer work with chrome extension by @manuongithub in #20
• Add support for querying vocabulary from tokenizer by @Ubospica in #22
• [Web] Expose getVocabSize and idToToken to web, bump version to 0.1.3 by @CharlieFRuan in #25
• Allow parameters related to special tokens for HFTokenizer by @Abhishek8394 in #27
• Set value of TOKENIZERS_CPP_CARGO_TARGET based on ANDROID_ABI by @yuxuanchiadm in #28
• Added EncodeBatch interface by @vinx13 in #33
• Remove incorrect final for non-virtual func by @vinx13 in #34
• [Rust] Bump HuggingFace tokenizer version by @MasterJH5574 in #36
• [CMake] Support disable SentencePiece tokenizer by @MasterJH5574 in #38
• [Web] Set TOKENIZERS_PARALLELISM to false for HFTokenizer by @CharlieFRuan in #42
• [CMake] Enable SentencePiece tokenizer by default by @MasterJH5574 in #47
• [Rust] Bump huggingface tokenizer to 0.20.0 by @MasterJH5574 in #49
• Bump CMake minimum to v3.19 by @afuller-TT in #59
• Fix for loading tokenizers using non-utf8 strings by @ThomasProg in #55
• updating tokenizer version from 0.20.0 to 0.21.0 by @NaveenMittal0 in #62
• Add a way to force the macos target in case it is set by @ykhrustalev in #52
• Fix CMake Configuration for MacOS Support by @Darce-One in #65
• [Web] Bump web-tokenizer to 0.1.6 by @CharlieFRuan in #67
• Fixed compilation for aarch64 by @NaveenMittal0 in #66

New Contributors

• @tqchen made their first contribution in #3
• @songkq made their first contribution in #2
• @erjanmx made their first contribution in #4
• @junrushao made their first contribution in #5
• @luiyen made their first contribution in #6
• @BBuf made their first contribution in #14
• @Hzfengsy made their first contribution in #16
• @manuongithub made their first contribution in #20
• @Ubospica made their first contribution in #22
• @CharlieFRuan made their first contribution in #25
• @Abhishek8394 made their first contribution in #27
• @yuxuanchiadm made their first contribution in #28
• @vinx13 made their first contribution in #33
• @MasterJH5574 made their first contribution in #36
• @afuller-TT made their first contribution in #59
• @ThomasProg made their first contribution in #55
• @NaveenMittal0 made their first contribution in #62
• @ykhrustalev made their first contribution in #52
• @Darce-One made their first contribution in #65

Full Changelog: v0.1.0...v0.1.1

v0.1.0

Add examples