Code

[text] refine tokenizer#2165

Code

Merged

robin1001 merged 20 commits intomainwenet-e2e/wenet:mainfrom Mddct-refine-tokenzierwenet-e2e/wenet:Mddct-refine-tokenzierCopy head branch name to clipboardNov 28, 2023

+1,131-242

Commits

Commits on Nov 25, 2023

• [text] refine tokenizer

  Mddct

  committed
  8144a2d

  Copy full SHA for 8144a2d

• [text] fix flake8

  Mddct

  committed
  99cf7d7

  Copy full SHA for 99cf7d7

• [text] fix lint

  Mddct

  committed
  5694565

  Copy full SHA for 5694565

• [text] fix unit

  Mddct

  committed
  2418d79

  Copy full SHA for 2418d79

Commits on Nov 26, 2023

• [text] add bpe tokenizer and char tokenizer

  Mddct

  committed
  3552b94

  Copy full SHA for 3552b94

• [text] add char tokenizer unit test

  Mddct

  committed
  9912df9

  Copy full SHA for 9912df9

• [text] add bpe tokenizer unit test

  Mddct

  committed
  266a4fa

  Copy full SHA for 266a4fa

Commits on Nov 27, 2023

• [text] add WhisperTokenizer for test_whisper.py

  Mddct

  committed
  c2ecc7c

  Copy full SHA for c2ecc7c

• [text] revert wenet/utils/file_utils.py

  Mddct

  committed
  55da48a

  Copy full SHA for 55da48a

• [text] add consistency for char and bpe unit

  Mddct

  committed
  50422fc

  Copy full SHA for 50422fc

• [text] merge main

  Mddct

  committed
  a479de7

  Copy full SHA for a479de7

• [text] merge main

  Mddct

  committed
  cf754ff

  Copy full SHA for cf754ff

• [text] add symbol table

  Mddct

  committed
  bd24277

  Copy full SHA for bd24277

• [text] add init_tokenizer unit test

  Mddct

  committed
  49994bf

  Copy full SHA for 49994bf

• [text] uncomment

  Mddct

  committed
  301af9e

  Copy full SHA for 301af9e

• [text] fix bpe model in multiprocess env

  Mddct

  committed
  a13dedf

  Copy full SHA for a13dedf

Commits on Nov 28, 2023

• [text] fix whisper tokenzier in multiprocess env

  Mddct

  committed
  ec2d838

  Copy full SHA for ec2d838

• [text] add test unit parallel for bpe and whisper

  Mddct

  committed
  51a10fa

  Copy full SHA for 51a10fa

• [text] fix none type in test_whisper.py

  Mddct

  committed
  1dc2d79

  Copy full SHA for 1dc2d79

• [text] all work

  Mddct

  committed
  f1099a8

  Copy full SHA for f1099a8