Code

[WIP][text] add tokens#2201

Code

Closed

Mddct wants to merge 6 commits intomainwenet-e2e/wenet:mainfrom Mddct-tokenizer-add_special_tokenswenet-e2e/wenet:Mddct-tokenizer-add_special_tokensCopy head branch name to clipboard

+83-270

Commits

Commits on Dec 7, 2023

• [text] add tokens

  Mddct

  committed
  4a6c862

  Copy full SHA for 4a6c862

Commits on Dec 8, 2023

• [text] add tries for bpe part

  Mddct

  committed
  36af934

  Copy full SHA for 36af934

• [text] add tokens

  Mddct

  committed
  6bfeb52

  Copy full SHA for 6bfeb52

• [text] bpe add tokens work

  Mddct

  committed
  5086569

  Copy full SHA for 5086569

• [text] bpe fix right

  Mddct

  committed
  89c3f8a

  Copy full SHA for 89c3f8a

Commits on Dec 11, 2023

• [text] rm wenet tokenizer && add add tokens for bpe by protobuf way

  Mddct

  committed
  772e8a0

  Copy full SHA for 772e8a0