Add g2pW to Chinese frontend #2230

BarryKCL · 2022-08-07T04:02:36Z

PR types

Performance optimization

PR changes

APIs

Describe

add g2pW onnxruntime

CLAassistant · 2022-08-07T04:02:41Z

All committers have signed the CLA.

yt605155624 · 2022-08-08T05:27:42Z

paddlespeech/t2s/frontend/zh_frontend.py

-                initials.append(sub_initials)
-                finals.append(sub_finals)
-                # assert len(sub_initials) == len(sub_finals) == len(word)
+            if self.g2p_model == "g2pW":


为保持一致，if self.g2p_model == "g2pW" 的判断和执行逻辑，是否也放到 _get_initials_finals，此处保持不变？

_get_initials_finals是基于分词后的word去转拼音，对多音词效果不佳。因此我在使用g2pW改成了整句预测拼音，再映射回分词后的word。如果要将if self.g2p_model == "g2pW"放到_get_initials_finals里面，那么分词前的句子也要传到_get_initials_finals里面。两个地方至少要改一个，现在这样其实也差不多？

好的，明白这里细节的区别了，那这里可以先保持这样，但是你可以在代码里面加一下注释，说明是为了多音字更好 g2pw 用了分词前的句子作为输入
但是不知道如果 pypinyin 直接输入分词前的句子多音字效果是否会变好？我猜还是不行？

pypinyin直接输入句子的效果我在你之前发的badcase展示过，那三个对比的结果都是整句预测的。

yt605155624 · 2022-08-08T05:37:17Z

paddlespeech/t2s/frontend/zh_frontend.py

@@ -53,9 +56,24 @@ def insert_after_character(lst, item):
    return result


+class Polyphonic():


如果这个类是 G2PW 独用的，是否重命名为 G2PWPolyphonic()，如果也可以用在 G2PM 上，也可以加到 G2PM 的逻辑里（现在 pypinyin 的写法应该是暂时加不上这个修正逻辑）

这个是我根据G2PW预测的一些badcase添加的，理论上也可以用在G2PM上。但是G2PM的badcase与G2PW的可能不需要共用一个polyphonic.yaml，可以根据自己的实际情况决定吧。

ok，这块等这个 pr 合入后我们考虑修改

yt605155624 · 2022-08-08T05:48:56Z

paddlespeech/t2s/frontend/g2pw/utils.py

+import sys
+
+
+class RunningAverage:


这个类如果没有用到可以删掉

yt605155624 · 2022-08-08T05:50:19Z

paddlespeech/t2s/frontend/g2pw/onnx_api.py

+
+from paddlenlp.transformers import BertTokenizer
+
+from paddlespeech.t2s.frontend.g2pw.dataset import prepare_data, prepare_onnx_input, get_phoneme_labels, get_char_phoneme_labels


此处分多行 import，最好用 pre-commit 刷一下代码格式

yt605155624 · 2022-08-08T05:52:46Z

paddlespeech/t2s/frontend/g2pw/utils.py

@@ -0,0 +1,161 @@
+import re


如果参考了 g2pw 的代码，请添加 copyright 或者 "
This code is copied/modified from {g2pw} 的链接 "

lym0302 · 2022-08-08T06:30:37Z

paddlespeech/t2s/frontend/g2pw/dataset.py

+        }
+        return outputs
+
+def _truncate_texts(window_size, texts, query_ids):


这个文件的函数可以增加一点注释，会更易懂一些~~~

GitYCC · 2022-08-10T03:37:47Z

@BarryKCL Thanks for your effort! And please reference GitYCC/g2pW in
https://github.com/PaddlePaddle/PaddleSpeech/blob/develop/docs/source/reference.md
Thank you!

yt605155624 · 2022-08-11T06:26:56Z

original wer of g2p:
The avg WER of g2p is: 0.026014352515701198

     ,--------------------------------------------------------------------.
     |        | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.3    2.7    0.0    0.0    2.7   52.2 |
     `--------------------------------------------------------------------'

wer after add g2pw:
The avg WER of g2p is: 0.028952373312476395

     ,--------------------------------------------------------------------.
     |                         ./exp/g2p/text.g2p                         |
     |--------------------------------------------------------------------|
     | SPKR   | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.2    2.8    0.0    0.1    2.9   53.3 |
     `--------------------------------------------------------------------'

yt605155624 · 2022-08-11T07:29:01Z

need to solve this issue: GitYCC/g2pW#9

sixyang · 2022-09-01T09:18:20Z

original wer of g2p: The avg WER of g2p is: 0.026014352515701198

     ,--------------------------------------------------------------------.
     |        | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.3    2.7    0.0    0.0    2.7   52.2 |
     `--------------------------------------------------------------------'

wer after add g2pw: The avg WER of g2p is: 0.028952373312476395

     ,--------------------------------------------------------------------.
     |                         ./exp/g2p/text.g2p                         |
     |--------------------------------------------------------------------|
     | SPKR   | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.2    2.8    0.0    0.1    2.9   53.3 |
     `--------------------------------------------------------------------'

这个表示 g2pw 相比 g2p 并没有多少提升吗？

yt605155624 · 2022-09-08T03:26:50Z

original wer of g2p: The avg WER of g2p is: 0.026014352515701198

     ,--------------------------------------------------------------------.
     |        | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.3    2.7    0.0    0.0    2.7   52.2 |
     `--------------------------------------------------------------------'

wer after add g2pw: The avg WER of g2p is: 0.028952373312476395

     ,--------------------------------------------------------------------.
     |                         ./exp/g2p/text.g2p                         |
     |--------------------------------------------------------------------|
     | SPKR   | # Snt    # Wrd  | Corr    Sub    Del    Ins    Err  S.Err |
     |--------+-----------------+-----------------------------------------|
     | Sum/Avg|  9996   299181  | 97.2    2.8    0.0    0.1    2.9   53.3 |
     `--------------------------------------------------------------------'

这个表示 g2pw 相比 g2p 并没有多少提升吗？

可以去 example/other/g2p 查看最新的数据，目前是 0.024

Add g2pW to Chinese frontend

ab2a121

mergify bot added T2S Documentation Installation labels Aug 7, 2022

BarryKCL added 3 commits August 7, 2022 16:49

Rollback "get_input_ids"

e60a63f

change transformers to paddlenlp.transformers

7b0f2a7

add MANIFEST.in

d17b2ee

yt605155624 reviewed Aug 8, 2022

View reviewed changes

yt605155624 assigned BarryKCL Aug 8, 2022

yt605155624 requested a review from lym0302 August 8, 2022 06:30

lym0302 reviewed Aug 8, 2022

View reviewed changes

add comment

744ea44

mergify bot added the README label Aug 8, 2022

yt605155624 mentioned this pull request Aug 8, 2022

[tts] 基于 BERT 实现语音合成文本前端的多音字预测 #1283

Closed

BarryKCL added 2 commits August 8, 2022 21:29

change G2PWModel download

de0f991

update

61dd92e

yt605155624 mentioned this pull request Aug 9, 2022

Include g2pW into PaddleSpeech TTS GitYCC/g2pW#5

Closed

yt605155624 added this to the r1.1.0 milestone Aug 9, 2022

add onnxruntime sess_options

aecf8fd

BarryKCL added 2 commits August 10, 2022 18:58

Fix a bug in g2pW

5e63ac1

set window_size None

6593c24

update g2pW dict

a84b40e

yt605155624 approved these changes Aug 15, 2022

View reviewed changes

yt605155624 merged commit a75b2a5 into PaddlePaddle:develop Aug 15, 2022

This was referenced Aug 16, 2022

睡得着觉 #2206

Closed

🔥 r1.1.0 release note #2271

Closed

yt605155624 added the contributor label Aug 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add g2pW to Chinese frontend #2230

Add g2pW to Chinese frontend #2230

BarryKCL commented Aug 7, 2022

CLAassistant commented Aug 7, 2022 •

edited

Loading

yt605155624 Aug 8, 2022

BarryKCL Aug 8, 2022

yt605155624 Aug 8, 2022

BarryKCL Aug 8, 2022

yt605155624 Aug 8, 2022

BarryKCL Aug 8, 2022

yt605155624 Aug 8, 2022

yt605155624 Aug 8, 2022

BarryKCL Aug 8, 2022

yt605155624 Aug 8, 2022

yt605155624 Aug 8, 2022

BarryKCL Aug 8, 2022

lym0302 Aug 8, 2022

GitYCC commented Aug 10, 2022 •

edited

Loading

yt605155624 commented Aug 11, 2022 •

edited

Loading

yt605155624 commented Aug 11, 2022

sixyang commented Sep 1, 2022

yt605155624 commented Sep 8, 2022

		@@ -53,9 +56,24 @@ def insert_after_character(lst, item):
		return result


		class Polyphonic():


		from paddlenlp.transformers import BertTokenizer

		from paddlespeech.t2s.frontend.g2pw.dataset import prepare_data, prepare_onnx_input, get_phoneme_labels, get_char_phoneme_labels

Add g2pW to Chinese frontend #2230

Add g2pW to Chinese frontend #2230

Conversation

BarryKCL commented Aug 7, 2022

PR types

PR changes

Describe

CLAassistant commented Aug 7, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

GitYCC commented Aug 10, 2022 • edited Loading

yt605155624 commented Aug 11, 2022 • edited Loading

yt605155624 commented Aug 11, 2022

sixyang commented Sep 1, 2022

yt605155624 commented Sep 8, 2022

CLAassistant commented Aug 7, 2022 •

edited

Loading

GitYCC commented Aug 10, 2022 •

edited

Loading

yt605155624 commented Aug 11, 2022 •

edited

Loading