server : Add option to return token pieces in /tokenize endpoint

[mathijshenquet]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Description

This PR enhances the /tokenize endpoint by adding an option to return token pieces along with their IDs. This feature allows users to easily understand what each token represents without needing to make additional API calls or perform client-side lookups.

Motivation

Often, users want to know not just the token IDs but also what these tokens represent in the original text. Previously, this required either inefficient additional API calls or complex client-side logic. This change makes it much easier and more efficient to get this information in a single request.

Changes

1. Added a new with_pieces boolean parameter to the /tokenize endpoint.
2. When with_pieces is true, the response includes both token IDs and their corresponding pieces.
3. Updated the endpoint documentation to reflect the new parameter and response format.
4. The change is backward compatible: if with_pieces is not specified or is false, the endpoint behaves as before.

Testing

• Added unit tests to cover the new functionality.
• Manually tested the endpoint with various inputs and parameter combinations.

Documentation

Updated the API documentation to include the new with_pieces parameter and to show example responses for both cases (with and without pieces).

NB: I wasn't able to run the CI pipeline locally as I'm currently on Windows.

[ngxson]

Also, this will fail if one of the pieces has incomplete unicode bytes. For example:

{"content": "媽", "with_pieces": true}

Response:

{
    "error": {
        "code": 500,
        "message": "[json.exception.type_error.316] incomplete UTF-8 string; last byte: 0xE5",
        "type": "server_error"
    }
}

[mathijshenquet]

Also, this will fail if one of the pieces has incomplete unicode bytes. For example:

For the case that a piece is invalid utf8 I have added a fallback where a list of bytes will be sent instead.

[mathijshenquet]

ping @ngxson

[ngxson]

LGTM. Thanks. Let's merge after the CI passes

Small suggestion: we actually already had a version of is_valid_utf8 in process_token, would be nice if we can deduplicate the code by reusing that. It can be done in a follow-up PR though.

[ggerganov]

@mathijshenquet Sorry for the delay. Let's resolve the conflict and merge.

[ngxson]

I merged with upstream master. That should fix the CI problem. (Let's merge once CI passes)