[Bugfix] handle alignment of encoder_seq_lens in mllama.py

[tjohnson31415]

Fix for the crash repro reported in this comment. The bug and fix are pretty similar to #12347 in that the problem arises in a batch of mixed text and image requests leading to the attn metadata having some lists with an element for each sequence and others with an element only for each sequence-with-images.

Another fix that came out of #10648

FIX #10648

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[tjohnson31415]

input_processor_for_mllama has been removed and I don't see our use of the Transformers processor doing this same trick to only process the last image group.

I'm working to undersatnd this better, but I think we either should re-implement the "cheat" or remove the extra complexity (i.e. this check comparing to num tiles and remove kv_range_for_decode).

[heheda12345]

I'm checking with the author of PR #11427 which removes the previous input processor. See the discussion here.
https://vllm-dev.slack.com/archives/C07QCGVDNUF/p1743001000770969

[heheda12345]

This cheat is implementing here #15564

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @tjohnson31415.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[heheda12345]

In general this PR looks good. Thanks for the fix. Can you sync this PR with main to get in recent bug fixes for mllama #15564 and #14883 and test it again?

[tjohnson31415]

@heheda12345 I rebased and updated the branch.
tests/models/encoder_decoder/vision_language/test_mllama.py tests all pass for me.

Ready for another look!

[heheda12345]

LGTM in general. Only some nits.

[tjohnson31415]

@heheda12345 Thanks for the review! I pushed the suggested changes (the linter didn't like l as a var name so used x instead). PTAL

[heheda12345]

Thanks for the bug fix!