Add support for BERT embedding models

[iamlemec]

Following discussion in #2872, adds support for BERT model architecture. Built on top of various contributions from @skeskinen, @xyzhang626, and @cebtenzzre. Includes:

• New WordPiece tokenizer llm_tokenize_wpm. Needed for slightly different behavior from SentencePiece. On conversion, vocab is mapped from ## subword scheme to ▁ prefix scheme to allow for unified vocab mappings.
• New model fields bert.attention.causal that controls whether attention mask is causal or not (default is true). Also tokenizer.ggml.token_type_count which accounts for token type info, though these are tpyically ignored in actual computations.
• Addition of build_bert for graph construction. This is fairly standard. The only difference is the pooling layer at the end. Currently it will pool the entire batch. Ideally, it could be made to pool only within sequence.

In terms of which models actually work, the main limitation is tokenization. I have tested with all-MiniLM-L6-v2 and BAAI/bge-*-*-v1.5 (small, base, and large plus en and zh) and they seem to work and the embedding numbers look similar to Huggingface implementations. The newer BAAI/bge-m3 uses a SentencePiece tokenizer, so it should be doable but I haven't tested it.

[cebtenzzre]

Note to self: resolve this before merge

[iacore]

have you... have you forgotten about this...

[ggerganov]

In terms of which models actually work, the main limitation is tokenization.

When I was playing with bert.cpp the other day, I noticed some potential problems with the tokenization when using a bge-base model. For example:

./build/bin/main -m models/bge-base-en-v1.5/ggml-model-f16.gguf -p "This is a ggml"

Tokenizes to:

101 -> [CLS]
2023 -> this
2003 -> is
1037 -> a
2290 -> ##g
19968 -> ##ml
102 -> [SEP]

Seems like a g is missing and also there is an extra concat ## in the 2290 token. So the tokenization might need some more work, but this can be improved later

[iamlemec]

When I was playing with bert.cpp the other day, I noticed some potential problems with the tokenization when using a bge-base model.

Ah yeah, that was a bug in bert.cpp that was fixed a few days ago. It's correct in this PR.

[iamlemec]

I have batched embedding working now (bert-batched). Basically just matmul an [n_tokens, n_tokens] pooling matrix at the end. It would make more sense for it to be [n_tokens, n_seq_max], but we don't actually know n_seq_max, so this is a worst case scenario. Embeddings can be fetched by seq_id just like with logits using get_embeddings_ith. Updated the embeddings example to split by lines and embed as separate sequences in one batch.

Should I push this to this PR or wait until this goes through and start a new one?

[ggerganov]

Using ggml_graph_get_tensor is not recommended here because it will do a strcmp with the entire graph which can become noticeable in terms of speed. For now, we should be "poking" at the last few tensors to find what we need - not great, but will improve in the future

[ggerganov]

Let's fix the ggml_graph_get_tensor comment and merge. After that, we can look into batching support in separate PR

[astrowonk]

This is very exciting. I was able to convert all-MiniLM-L6-v2 but alas notsentence-transformers/all-mpnet-base-v2 which is what I had been using for sentence-transformers lately.

I get a NotImplementedError: Architecture "MPNetForMaskedLM" not supported!

I'll try all-MiniLM-L6-v2 for now and mabye nomic which I have tried previously, but if we could add support to convert all-mpnet-base-v2 that would be amazing. (I can open another issue if that's helpful.)

EDIT: I was able to convert all-MiniLM-L6-v2 with a fork of bert.cpp but the resulting gguf model doesn't load in llama.cpp.

EDIT OF THE EDIT: Sorry I meant that I converted all-mpnet-base-v2 with the bert.cpp fork but yeah, it's not working as others have explained subsequently.

[ditsuke]

EDIT: I was able to convert all-MiniLM-L6-v2 with a fork of bert.cpp but the resulting gguf model doesn't load in llama.cpp.

You have to use the hf-to-gguf script that ships with this repo, bert.cpp's conversion script doesn't produce a llama.cpp-compatible version.

[astrowonk]

EDIT: I was able to convert all-MiniLM-L6-v2 with a fork of bert.cpp but the resulting gguf model doesn't load in llama.cpp.

You have to use the hf-to-gguf script that ships with this repo, bert.cpp's conversion script doesn't produce a llama.cpp-compatible version.

Yes, that's what I'm asking for: for the hf-to-gguf script here to support converting the all-mpnet-base-v2 model. It crashes at the moment with the NotImplemented error I describe in my earlier post.

[ditsuke]

EDIT: I was able to convert all-MiniLM-L6-v2 with a fork of bert.cpp but the resulting gguf model doesn't load in llama.cpp.

You have to use the hf-to-gguf script that ships with this repo, bert.cpp's conversion script doesn't produce a llama.cpp-compatible version.

Yes, that's what I'm asking for: for the hf-to-gguf script here to support converting the all-mpnet-base-v2 model. It crashes at the moment with the NotImplemented error I describe in my earlier post.

Oh, I see. I think your last statement was meant for all-mpnet then, you should open a new issue.

[iamlemec]

It looks like all-mpnet has T5-style relative position embeddings. I don't think those are supported here yet.

[mofanke]

i tried BAAI/bge-m3 , but i does not work by now. because the model architectures is XLMRobertaModel not Bert , and "tokenizer_class": "XLMRobertaTokenizer"

[cebtenzzre]

i tried BAAI/bge-m3 , but i does not work by now. because the model architectures is XLMRobertaModel not Bert , and "tokenizer_class": "XLMRobertaTokenizer"

You could open a feature request if you haven't already.

[mofanke]

#6007 already done

[hiepxanh]

#6007 already done

What do you mean, I think the PR is not support yet?

I try convert to day an see this one?

(llama.cpp) E:\pre-built\llama.cpp>python convert-hf-to-gguf.py models/multilingual-e5-large/
Loading model: multilingual-e5-large
Traceback (most recent call last):
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2073, in <module>
    main()
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2053, in main
    model_class = Model.from_model_architecture(hparams["architectures"][0])
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 204, in from_model_architecture
    raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'XLMRobertaModel' not supported!

[mofanke]

#6007 already done

What do you mean, I think the PR is not support yet?

I try convert to day an see this one?

(llama.cpp) E:\pre-built\llama.cpp>python convert-hf-to-gguf.py models/multilingual-e5-large/
Loading model: multilingual-e5-large
Traceback (most recent call last):
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2073, in <module>
    main()
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2053, in main
    model_class = Model.from_model_architecture(hparams["architectures"][0])
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 204, in from_model_architecture
    raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'XLMRobertaModel' not supported!

aha， I'm sorry for causing you confusion， i just mean i opened a feature request

[jkgenser]

Just tried google bert uncased and raised NotImplementedError: Architecture "BertForMaskedLM" not supported! I probably miss something here.
another model NotImplementedError: Architecture "BertForSequenceClassification" not supported!

These are BERT models that have been pretrained to create embeddings for individual words, but this PR is for BERT models that have been trained to generate embeddings for entire sentences and paragraphs, and those will not produce good results here.

The keyword you are looking for is "SBert", or Sentence Transformers in general (paper, website, HF).

nomic-embed-text-v1 is a good model to start with. Disclosure: I work for Nomic.

bge-base-en-v1.5 is another BERT of similar size.

So if I finetune as bert model for classification task, it would not work to convert it to GGML? I've been watching this work and really excited to be able to deploy my fine-tuned BERT models on llama.cpp

[beyondskyway]

#6007 already done

What do you mean, I think the PR is not support yet?
I try convert to day an see this one?

(llama.cpp) E:\pre-built\llama.cpp>python convert-hf-to-gguf.py models/multilingual-e5-large/
Loading model: multilingual-e5-large
Traceback (most recent call last):
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2073, in <module>
    main()
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 2053, in main
    model_class = Model.from_model_architecture(hparams["architectures"][0])
  File "E:\pre-built\llama.cpp\convert-hf-to-gguf.py", line 204, in from_model_architecture
    raise NotImplementedError(f'Architecture {arch!r} not supported!') from None
NotImplementedError: Architecture 'XLMRobertaModel' not supported!

aha， I'm sorry for causing you confusion， i just mean i opened a feature request

same with convert https://huggingface.co/maidalun1020/bce-embedding-base_v1/tree/main

[ggerganov]

Where is the reference implementation of XLMRobertaModel for models such as https://huggingface.co/intfloat/multilingual-e5-base/tree/main? Would like to add support for these

[iamlemec]

I think the original is here at fairseq: https://github.com/facebookresearch/fairseq/blob/main/fairseq/models/roberta. There's also an implementation in transformers: https://github.com/huggingface/transformers/tree/main/src/transformers/models/xlm_roberta. I've actually been looking into XLMRobertaModel to run BAAI/bge-m3, and comparing the transformers implementations, I think the model side is actually identical to BERT.

But there are differences in the tokenization that have driven me slightly mad trying to understand. The model file is called sentencepiece.bpe.model, but it appears to be an actual SentencePiece (unigram) style model, not BPE. Even then the way it handles spaces with metaspaces looks to be a little different from the way SPM works in the current llama.cpp implementation.

[ggerganov]

Thanks!

Even then the way it handles spaces with metaspaces looks to be a little different from the way SPM works in the current llama.cpp implementation.

Maybe the "clean_up_tokenization_spaces" parameter is controlling this behaviour?

https://huggingface.co/BAAI/bge-m3/blob/main/tokenizer_config.json#L3

[sragrawal]

Hi All, is there any plan to support XLMRobertaModel? https://huggingface.co/intfloat/multilingual-e5-small works very well for multilingual embeddings for its size (https://huggingface.co/spaces/mteb/leaderboard). Please let me know if there if I should open a new issue for this.

[iamlemec]

@sragrawal I believe that Unigram support from #8089 will get us most of the way there on the XLMRoberta tokenizer (which is featured in this and others such as BAAI/bge-m3). The main thing is loading and using the trie structure stored in precompiled_charsmap. There may be some additional pretokenization stuff, but that should be easier to handle.