Availability of using other BERT alternatives for current langauges

[gordon0414]

Hello, I was wondering if we can replace the BERT for current supported models

    Languages.JP: BASE_DIR / "bert" / "deberta-v2-large-japanese-char-wwm",
    Languages.EN: BASE_DIR / "bert" / "deberta-v3-large",
    Languages.ZH: BASE_DIR / "bert" / "chinese-roberta-wwm-ext-large",

Since the current models are mostly large variants of the models, but I think smaller BERT models will still be capable for our use case.

I mean by smaller variants such as
- English: microsoft/deberta-v3-small
- Japanese: ku-nlp/deberta-v2-base-japanese-char-wwm

I suppose these variants won't cause any issues since the tokenizers are identical to the original ones

My hypothesis is that if we fine-tune our model with the new BERT models, it will seamlessly incorporate with the current system.

[litagin02]

I haven't checked, but maybe as you guess we can, and you can try it.

[gordon0414]

okay, maybe I should try it this weekend!

[gordon0414]

The dimensional discrepancy between the base and large models (768 vs. 1024) seems to be a significant problem. We need to come up with techniques such as knowledge distillation or other methods.