May I ask you a question about the "scale_emb_or_prj" parameter?

[aitch25]

May I ask you a question about the "scale_emb_or_prj" parameter?

As far as I know, the authors of "Attention is all you need" used both 'prj' and 'emb' together in their model. But, you made your code to select one between the two ('prj' or 'emb'), or 'none.' I want to ask why you made this difference from the original paper.

I believe you disigned it because of some reasons, such as performance or something.

Or, if I misunderstood the original paper, please kindly let me know.

I always thank you for your wonderful project.

=== in "transformer.Models.py" ============================================

    self.src_pad_idx, self.trg_pad_idx = src_pad_idx, trg_pad_idx

    # In section 3.4 of paper "Attention Is All You Need", there is such detail:
    # "In our model, we share the same weight matrix between the two
    # embedding layers and the pre-softmax linear transformation...
    # In the embedding layers, we multiply those weights by \sqrt{d_model}".
    #
    # Options here:
    #   'emb': multiply \sqrt{d_model} to embedding output
    #   'prj': multiply (\sqrt{d_model} ^ -1) to linear projection output
    #   'none': no multiplication

    assert scale_emb_or_prj in ['emb', 'prj', 'none']
    scale_emb = (scale_emb_or_prj == 'emb') if trg_emb_prj_weight_sharing else False
    self.scale_prj = (scale_emb_or_prj == 'prj') if trg_emb_prj_weight_sharing else False
    self.d_model = d_model

=================================================================