[vie] Enhancements

[kylebgorman]

I notice a few obvious problems with Vietnamese:

1. As far as I can tell, nearly all Vietnamese words have three phonetic pronunciations: "Hà Nội", "Huế, and "Hồ Chí Minh City". We should just add dialects to languages.json
2. As is well-known, the Vietnamese Roman orthography puts a space between every syllable. So our data set is really only monosyllabic Vietnamese words! We could solve this by adding a flag (--no-skip-space) here, or we could make a language specific-extractor, I suppose.

[lfashby]

I like the idea of adding a --no-skip-space flag. Even if it is just useful for one language for the time being. Without reworking how we skip words a Vietnamese extraction function wouldn't do the trick.

Relatedly, we have this somewhat crude way of not skipping spaces in Chinese transcriptions. Perhaps we should we have a --no-skip-space-ortho flag and a --no-skip-space-pron flag?

[kylebgorman]

I should mention it looks like the Vietnamese data has spaces in the IPA transcriptions between each syllable too. (Have to check whether segments can handle that sensibly---by default we should just pass through.)

Good idea re: two flags.

[lfashby]

I believe segments handles it a bit awkwardly.

[yeonju123]

2. As is well-known, the Vietnamese Roman orthography puts a space between every syllable. So our data set is really only monosyllabic Vietnamese words! We could solve this by adding a flag (--no-skip-space)

Why were we skipping data consisting of multi words?
The code bit here.

def _skip_word(word: str) -> bool: # Skips multiword examples. if " " in word: return True

We must have talked about this before, but I cannot recall. Removing this code will help this issue.
If we added this code in order to avoid too repetitive types of compound words (e.g., living room, bath room, small room etc.), at least in English Wiktionary, their pronunciations are not provided. So these cases will not be scraped automatically without adding the skipping code. If it works the same for other languages, would it be ok to remove that code?

[kylebgorman]

I guess some reasons to skip them include:

1. There's no general guidance for why multiword expressions might be entered in Wiktionary in the first place. Sometimes they're common compounds or idioms, but occasionally there are even entire phrases.
2. There's no reason to collect multiword expressions for pronunciation purposes, which is our goal here.

You're right that that function controls it, but I think we probably want it to be scriptable so users can control whether they skip, and we can set it on a per-language basis for the big scrape. So I think the obvious thing to do is to add a flag (in cli.py) and then make sure it's passed to _skip_word; then, disable that for Chinese and Vietnamese.

Potentially we could also make the rules about what gets skipped more expressive---maybe you could pass in a regular expression and if it's matched, the word is skipped---but that's probably premature.

[lfashby]

Any thoughts on what we should do with transcriptions containing spaces between syllables? segments does the following:

import segments
tokenizer = segments.Tokenizer()       

print(tokenizer("ʔaːm˧˦ hiəw˧˨ʔ", ipa=True)) # Will print: ʔ aː m ˧˦ # h i ə w ˧˨ ʔ

Additionally I've found that some of our Persian data and some of our Tibetan data already contain "#". These transcriptions utilize the non-breaking space character, which we don't check for and therefore don't skip.

[kylebgorman]

We should remove the "#" as a general post-processing step after tokenization) We also should remove the non-breaking space (before tokenization, presumably). K

…

On Sat, Mar 28, 2020 at 12:31 PM Lucas Ashby ***@***.***> wrote: Any thoughts on what we should do with transcriptions containing spaces between syllables? segments does the following: import segments tokenizer = segments.Tokenizer() print(tokenizer("ʔaːm˧˦ hiəw˧˨ʔ", ipa=True)) # Will print: ʔ aː m ˧˦ # h i ə w ˧˨ ʔ Additionally I've found that some of our Persian <https://github.com/kylebgorman/wikipron/blob/master/languages/wikipron/tsv/per_phonemic.tsv#L388> data and some of our Tibetan <https://github.com/kylebgorman/wikipron/blob/master/languages/wikipron/tsv/tib_phonemic.tsv#L304> data already contain "#". These transcriptions utilize the non-breaking space <https://en.wikipedia.org/wiki/Non-breaking_space> character, which we don't check for and therefore don't skip. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/kylebgorman/wikipron/issues/133#issuecomment-605485017>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABG4ONMEBK7GZJL55FZM33RJYQ5JANCNFSM4LTZSFKA> .

[kylebgorman]

Could you give us example words with three vs. four dialects? Then I'll try to replicate. FWIW, I hadn't seen those other dialect specifications for Vietnamese.

…

On Sat, Mar 28, 2020 at 10:26 PM yeonju123 ***@***.***> wrote: I have some issue. When a word has three dialects, nothing is scraped. When there are four dialects including Vinh, Thanh Chương, Hà Tĩnh dialect, only this dialect is scraped. I tried scraping with a dialect specified as 'hanoi', but it still does not work. Because of this issue, I cannot test whether my code for fixing the space is working. Does anyone have the same issue? — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <https://github.com/kylebgorman/wikipron/issues/133#issuecomment-605548515>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABG4OP25WMMWDYELZCQLIDRJ2WXFANCNFSM4LTZSFKA> .

[yeonju123]

Ah, don't worry about that issue. I set the phonemic parameter wrong. But as for the 4th dialect,
Examples are like cầm. I just found that this word cháy has five dialects. Interestingly, the other dialects that are not noted before in our program are using /phonemic/ representation, unlike Hanoi, Ho chi minh, and Hue dialects that use [phonect] representation.

[kylebgorman]

Thanks for those two examples; we should add those dialects to the list, I suppose, and see if they get over threshold.

[lfashby]

Scraping for a Vietnamese dialect from the three that Kyle added in #135 returns results for all three current dialects (when phonetic=True). Scraping for one of the additional dialects that Yeonju has added returns the same results as scraping for one of the first three dialects added in #135 (when phonetic=True).
As far as I can tell this is due to how we define _DIALECT_XPATH_SELECTOR_TEMPLATE in config.py and the inconsistent HTML used on pages like cầm.
It looks like Vietnamese dialect information (for the three dialects covered in #135 ) is stored within <i> elments, while the dialect information for the dialects that Yeonju has added is stored as text in <span class="ib-content qualifier-content"> elements - but not as text in the <a> elements within those span elements that we usually target (which is why we get nothing when scraping for one of the dialects Yeonju found while phonetic=False).