🚨 Leftover processors - #47924
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
run-slow: ernie4_5_vl_moe, fuyu, qwen2_audio |
|
This comment contains models: ["models/ernie4_5_vl_moe", "models/fuyu", "models/qwen2_audio"] |
CI ResultsCommit Info
Model CI Report❌ 3 new failed tests from this PR 😭
|
| image_patches_indices (`torch.LongTensor` of shape `(batch_size, sequence_length)`, *optional*): | ||
| Tensor of indices of the image patches in the input_ids tensor. |
There was a problem hiding this comment.
unused by model I dunno since when, so deleted here so it is just consumed by kwargs. Also deleted from processor outputs
| # IMPORTANT: here comes the custom part with tokenization, do not change it! | ||
| if not images_replacements: | ||
| text_inputs = self.tokenizer(text, **merged_kwargs["text_kwargs"]) | ||
| else: | ||
| # encode the text and the placeholders separately, then always pad on the left | ||
| batch_input_ids, batch_attention_mask = [], [] | ||
| for sample in text: | ||
| split_sample = re.split(r"(?<=<s>)", sample, maxsplit=1) | ||
| prompt_inputs = self.tokenizer(split_sample[-1], **merged_kwargs["text_kwargs"]) | ||
|
|
||
| if len(split_sample) == 2: | ||
| # strip off the underscore which is always prepended before special image tokens | ||
| # we are guaranteed that the output are torch tensors | ||
| placeholder_inputs = self.tokenizer(split_sample[0], **merged_kwargs["text_kwargs"]) | ||
| batch_input_ids.append( | ||
| torch.cat( | ||
| [placeholder_inputs["input_ids"][..., 1:], prompt_inputs["input_ids"]], dim=-1 |
There was a problem hiding this comment.
very ugly but there is no way around with the saved tokenizer's quirks. We can't suppress the prefix underscore always, it should be kept in text-only samples and dropped only when added before image tokens
|
run-slow: emu3, ernie4_5_vl_moe, fuyu, qwen2_audio |
|
This comment contains models: ["models/emu3", "models/ernie4_5_vl_moe", "models/fuyu", "models/qwen2_audio"] |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
potentially a new rule for the mlinter? 👀
|
Maybe... We might have a few reaaaly old ones that I don't want to even touch, they will have to go on allow list 😆 |
|
[For maintainers] Suggested jobs to run (before merge) run-slow: emu3, ernie4_5_vl_moe, fuyu, qwen2_audio |
CI recapDashboard: View test results in Grafana |
5068901
3 participants
What does this PR do?
As per title + fix emu's BOS token addition. Fuyu is hard, so I am checking how to do it with minimal changes
relate to vllm: vllm-project/vllm#51827
Note: this is breaking because we don't return an unused
image_patch_indicesanymore from FuyuProcessor. The model has not been using them for a while thus the input has been dropped from processors as well