Add BGE-M3 Encoder

[andreaschari]

Adds BGE-M3 Encoder support and documentation.

[seanmacavaney]

Great, thanks! I think it's mostly there, but it needs a few changes. Mostly stuff to bring it in line with the existing conventions in the package so that it doesn't need to be treated differently.

Also, can you add unit tests?

[cmacdonald]

realised my review wasnt submitted (and its substantially the same as Seans)

[cmacdonald]

This looks better now Andreas. You need some unit tests though.

@seanmacavaney should we align the column names for the multi-vec with the pyt_colbert?:

• query_toks [list of token ids]
• query_embs [tensor of shape padded query len [32] x dimension size]
• doc_toks [list of token ids]
• doc_embs [tensor of shape doc len x dimension size]

[seanmacavaney]

@cmacdonald I largely agree the above. But it looks like query_toks is an unfortunate name conflict between LSR stuff (type: Dict[str, float|int]) and the ColBERT stuff (type: List[int]). And to make matters worse, these directly conflict in this setting :/.

[seanmacavaney]

@andreaschari chatting with @cmacdonald, I think we can use *_embs_toks for the tokens from the multivec model, to avoid conflicting the names with the LSR model.

[seanmacavaney]

Looks great, thanks @andreaschari!