Conformer Modules

[mmz33]

Fix #54.

This is a draft now. It still requires the implementation of MultiHeadSelfAttention from #52.

[mmz33]

The parameters names still need to be changed to be consistent with #53, #55.

[albertz]

See also my other comments (they don't show up in the "Files changed" view because they got outdated due to the changed indent). Mark them as "resolved" when you resolved them.

[albertz]

Btw, as usual, also see failing tests.

[albertz]

Merged now. Let's further improve this in later PRs or commits.

[albertz]

A question on the ConformerConvSubsample: I think this is not exactly described in the original paper, right? So where is it exactly described? Where did you base this on?

[mmz33]

Yes, in the original paper it is not described. The one implemented here was based on ESPNet code. See here:

• https://github.com/espnet/espnet/blob/master/espnet2/asr/encoder/conformer_encoder.py#L151
• https://github.com/espnet/espnet/blob/f6a2522ad0035451e1edf69b7c92cba272b25871/espnet/nets/pytorch_backend/transformer/subsampling.py#L44

[albertz]

But in the original paper, there were some references to other paper they referred to on the preprocessing. I think it is explained in those other papers, or not?

Also, in your implementation here, you use maxpooling as far as I see. But this is different to ESPnet. In ESPnet, they don't use any pooling but just striding instead. Where do you have the pooling from?

[albertz]

Also, in ESPnet there are some other variants, including VGG2L (and I think VGG-style was also mentioned in the Conformer paper, or the one it refers to?). VGG2L looks also similar to @christophmluscher 's hybrid baseline?

[albertz]

Further, in ESPnet (ConformerEncoder), there is pos encoding at the end of the preprocessing block: here

This is usually RelPositionalEncoding (e.g. in train_asr_conformer10_hop_length160).

Then, for the self-attention, it usually uses RelPositionMultiHeadedAttention.

[albertz]

Also related: espnet/espnet#2816, espnet/espnet#2684

[mmz33]

Where do you have the pooling from?

Ok then it is not exactly the same. you are right they use striding instead.