Improve nvtext::tokenize_with_vocabulary performance

[davidwendt]

Description

Part of refactoring split/tokenize common code and improvements.
This improves performance of the vocabulary-tokenizer mostly for smaller strings (<=128 bytes) by using an intermediate buffer to hold the token-counts before computing offsets.

Checklist

• I am familiar with the Contributing Guidelines.
• New or existing tests cover these changes.
• The documentation is up to date with these changes.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[davidwendt]

/ok to test

[davidwendt]

Benchmarks show performance improvement for the specialized logic for smaller strings

## [0] NVIDIA RTX A6000

| max_width | num_rows |   Ref Time |   Cmp Time |         Diff |   %Diff |
|-----------|----------|------------|------------|--------------|---------|
|    32     |  32768   | 194.769 us | 168.589 us |   -26.181 us | -13.44% |
|    64     |  32768   | 305.154 us | 245.225 us |   -59.929 us | -19.64% |
|    128    |  32768   | 573.142 us | 425.584 us |  -147.559 us | -25.75% |
|    256    |  32768   | 337.069 us | 338.036 us |     0.967 us |   0.29% |
|    32     |  262144  | 311.163 us | 300.832 us |   -10.331 us |  -3.32% |
|    64     |  262144  | 524.604 us | 498.134 us |   -26.469 us |  -5.05% |
|    128    |  262144  |   1.080 ms | 919.996 us |  -160.119 us | -14.82% |
|    256    |  262144  |   1.776 ms |   1.774 ms |    -2.098 us |  -0.12% |
|    32     | 2097152  |   1.471 ms |   1.424 ms |   -46.918 us |  -3.19% |
|    64     | 2097152  |   2.840 ms |   2.631 ms |  -208.705 us |  -7.35% |
|    128    | 2097152  |   6.575 ms |   5.292 ms | -1282.198 us | -19.50% |
|    256    | 2097152  |  13.569 ms |  13.521 ms |   -47.806 us |  -0.35% |

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[davidwendt]

/ok to test

[davidwendt]

/merge