Fix apply_bitmask logit for both CPU and triton versions when shape and stride doesn't match

[Jialin]

Motivation

In vLLM, in order to address Issue #19493, PR #19565 served as a workaround to mitigate the issue by switching to use torch.compile version instead of triton version.

Root cause

With some investigation, we found that the error would happen when logits.stride()[0] != logits.shape[-1].

Changes

• Fix CPU and Triton-version apply_grammar_bitmask for this scenario
  • use stride[0] instead of shape[-1] when access rows
• Add a unit test to reproduce the errors

Tests

We've verified

• New unit tests would failed on trunk
• New unit tests passed with the change (and no new failure, however, there're a lot of failing tests in trunk, 47 failed in tests/python/test_token_bitmask_operations.py)

Followup

We will follow up with a benchmark comparison to see if we should bring back triton-version or use the new cuda version on vLLM side.

[Ubospica]

Thank you so much for your contribution!

strides are not considered previously. It's good that this PR fixes this issue.

In the future we will further use nanobind's ArrayList to simplify the logic of passing torch tensor to C++. See https://github.com/mlc-ai/xgrammar/blob/main/cpp/nanobind/nanobind.cc#L45. This can avoid the complex logic of passing shape, stride, and dtype, etc. separately to C++.

[Jialin]

@Ubospica The precommit failure is addressed. And also we tried to fix cuda implementation in #394

[Ubospica]

@Jialin Thank you so much for your contribution