From 09b5ece87f2e6f3b2d8639bcab0dfe6bc8778e2f Mon Sep 17 00:00:00 2001 From: raushan Date: Wed, 23 Oct 2024 09:19:32 +0200 Subject: [PATCH 1/3] update docs --- docs/source/en/model_doc/mllama.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) diff --git a/docs/source/en/model_doc/mllama.md b/docs/source/en/model_doc/mllama.md index 9cb038ed2e34..7c7289bf65ba 100644 --- a/docs/source/en/model_doc/mllama.md +++ b/docs/source/en/model_doc/mllama.md @@ -30,6 +30,32 @@ The Llama 3.2-Vision collection of multimodal large language models (LLMs) is a - The text passed to the processor should have the `"<|image|>"` tokens where the images should be inserted. - The processor has its own `apply_chat_template` method to convert chat messages to text that can then be passed as text to the processor. + + + +Mllama has an extra token used as a placeholder for image positions in the text. It means that input ids and an input embedding layer will have an extra token. But since the weights for input and output embeddings are not tied, the `lm_head` layer has one less token and will fail if you want to calculate loss on image tokens or apply some logit processors. In case you are training, make sure to mask out special `"<|image|>"` tokens as the model should not be trained on predicting them. + +Otherwise if you see CUDA-side index erros when generating, use the below code to expand the `lm_head` by one more token. + + +```python +pre_expansion_embeddings = model.language_model.lm_head.weight.data +mu = torch.mean(pre_expansion_embeddings, dim=0).float() +n = pre_expansion_embeddings.size()[0] +sigma = ((pre_expansion_embeddings - mu).T @ (pre_expansion_embeddings - mu)) / n +dist = torch.distributions.multivariate_normal.MultivariateNormal(mu, covariance_matrix=1e-5 * sigma) + + +num_new_tokens = 1 # 1 for the `"<|image|>"` token +lm_head_weights = model.language_model.lm_head.weight + +new_token_embedding = torch.stack(tuple(dist.sample() for _ in range(num_new_tokens)), dim=0).to(device=lm_head_weights.device, dtype=lm_head_weights.dtype) +lm_head_weights.data = torch.cat([lm_head_weights.data, new_token_embedding], dim=0) +lm_head_weights.num_embeddings = lm_head_weights.data.shape[0] +``` + + + ## Usage Example #### Instruct model From a823c5362b4d787b13704386b44afee1cb3a29c7 Mon Sep 17 00:00:00 2001 From: raushan Date: Wed, 23 Oct 2024 09:21:52 +0200 Subject: [PATCH 2/3] be more explicit --- docs/source/en/model_doc/mllama.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/source/en/model_doc/mllama.md b/docs/source/en/model_doc/mllama.md index 7c7289bf65ba..61efb8ba2aaa 100644 --- a/docs/source/en/model_doc/mllama.md +++ b/docs/source/en/model_doc/mllama.md @@ -33,7 +33,7 @@ The Llama 3.2-Vision collection of multimodal large language models (LLMs) is a -Mllama has an extra token used as a placeholder for image positions in the text. It means that input ids and an input embedding layer will have an extra token. But since the weights for input and output embeddings are not tied, the `lm_head` layer has one less token and will fail if you want to calculate loss on image tokens or apply some logit processors. In case you are training, make sure to mask out special `"<|image|>"` tokens as the model should not be trained on predicting them. +Mllama has an extra token used as a placeholder for image positions in the text. It means that input ids and an input embedding layer will have an extra token. But since the weights for input and output embeddings are not tied, the `lm_head` layer has one less token and will fail if you want to calculate loss on image tokens or apply some logit processors. In case you are training, make sure to mask out special `"<|image|>"` tokens in the `labels` as the model should not be trained on predicting them. Otherwise if you see CUDA-side index erros when generating, use the below code to expand the `lm_head` by one more token. From 9899363eee56b734a3a703b017abf55bb617d7fd Mon Sep 17 00:00:00 2001 From: raushan Date: Tue, 29 Oct 2024 16:31:35 +0100 Subject: [PATCH 3/3] use avaialble methods --- docs/source/en/model_doc/mllama.md | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-) diff --git a/docs/source/en/model_doc/mllama.md b/docs/source/en/model_doc/mllama.md index 61efb8ba2aaa..4a6080ea2ce0 100644 --- a/docs/source/en/model_doc/mllama.md +++ b/docs/source/en/model_doc/mllama.md @@ -39,19 +39,12 @@ Otherwise if you see CUDA-side index erros when generating, use the below code t ```python -pre_expansion_embeddings = model.language_model.lm_head.weight.data -mu = torch.mean(pre_expansion_embeddings, dim=0).float() -n = pre_expansion_embeddings.size()[0] -sigma = ((pre_expansion_embeddings - mu).T @ (pre_expansion_embeddings - mu)) / n -dist = torch.distributions.multivariate_normal.MultivariateNormal(mu, covariance_matrix=1e-5 * sigma) +old_embeddings = model.get_output_embeddings() - -num_new_tokens = 1 # 1 for the `"<|image|>"` token -lm_head_weights = model.language_model.lm_head.weight - -new_token_embedding = torch.stack(tuple(dist.sample() for _ in range(num_new_tokens)), dim=0).to(device=lm_head_weights.device, dtype=lm_head_weights.dtype) -lm_head_weights.data = torch.cat([lm_head_weights.data, new_token_embedding], dim=0) -lm_head_weights.num_embeddings = lm_head_weights.data.shape[0] +num_tokens = model.vocab_size + 1 +resized_embeddings = model._get_resized_lm_head(old_embeddings, new_num_tokens=num_tokens, mean_resizing=True) +resized_embeddings.requires_grad_(old_embeddings.weight.requires_grad) +model.set_output_embeddings(resized_embeddings) ```