[CLEANUP]: Remove `batch_token_ids` and replace `token_ids` with nested `Vec` for cleaner processing

[ishandhanani]

Feature request

As per #1626 (comment) - we want to just check if token_ids is a batch or a single. Changes are required in preprocessor.rs and the sglang workers

Describe the problem you're encountering

n/a

Describe alternatives you've tried

No response

[Medyan-Naser]

Hi!
I’d like to work on this issue (#1648). Could you please assign it to me? I’ll follow the contributing guidelines and submit a PR soon. Thanks!

[dzier]

@Medyan-Naser, the issue is yours to work on, if you want. Assigning Ishan as well to follow up when you have PR ready to go.

[Medyan-Naser]

@Medyan-Naser, the issue is yours to work on, if you want. Assigning Ishan as well to follow up when you have PR ready to go.

Thanks! I’ll work on this issue following a similar approach to my previous PR and will update once it’s ready.

[zhixiongli2011]

Hello, might I help with this issue?

[ishandhanani]

#3482