[Model] Add Reasoning Parser for Granite Models

[alex-jw-brooks]

This PR adds a reasoning parser for Granite 3.2 models! These models have an optional chat template kwarg thinking that changes the system prompt to enable reasoning. 😄

The format of the text is expected to be:

Here is my thought process: <reasoning_content> Here is my response: <content>

There have been reports of quantized versions of the model using Here's instead of Here is though, so this PR matches on both.

Examples

Start the server with a granite (3.2) language model that has reasoning and the granite parser:

python vllm/entrypoints/openai/api_server.py \
    --device cuda \
    --model ibm-granite/granite-3.2-8b-instruct \
    --tokenizer ibm-granite/granite-3.2-8b-instruct \
    --enable-reasoning \
    --reasoning-parser granite

Snippets are copied from the docs, with the only change being adding chat_template_kwargs with thinking=True. Without this, reasoning will be disabled, and it'll generally parse everything into content.

No streaming:

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

# Round 1
messages = [
    {
        "role": "user",
        "content": "9.11 and 9.8, which is greater?"
    }
]
response = client.chat.completions.create(model=model, messages=messages, extra_body={"chat_template_kwargs": {"thinking": True}})

reasoning_content = response.choices[0].message.reasoning_content
content = response.choices[0].message.content

print("reasoning_content:", reasoning_content)
print("content:", content)

With streaming:

import json

import requests

# Modify OpenAI's API key and API base to use vLLM's API server.
openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

models = requests.get(
    f"{openai_api_base}/models",
    headers={
        "Authorization": f"Bearer {openai_api_key}"
    },
).json()
model = models["data"][0]["id"]

# Streaming chat completions
messages = [{"role": "user", "content": "9.11 and 9.8, which is greater?"}]

response = requests.post(
    f"{openai_api_base}/chat/completions",
    headers={"Authorization": f"Bearer {openai_api_key}"},
    json={
        "model": model,
        "messages": messages,
        "chat_template_kwargs": {"thinking": True},
        "stream": True
    },
)



print("client: Start streaming chat completions...")
printed_reasoning_content = False
printed_content = False
# Make the streaming request
if response.status_code == 200:
    # Process the streaming response
    for line in response.iter_lines():
        if line:  # Filter out keep-alive new lines
            # Decode the line and parse the JSON
            decoded_line = line.decode("utf-8")
            if decoded_line.startswith("data:"):
                data = decoded_line[5:].strip()  # Remove "data:" prefix
                if data == "[DONE]":  # End of stream
                    print("\nclient: Stream completed.")
                    break
                try:
                    # Parse the JSON data
                    chunk = json.loads(data)
                    reasoning_content = chunk["choices"][0]["delta"].get(
                        "reasoning_content", "")
                    content = chunk["choices"][0]["delta"].get("content", "")

                    if reasoning_content:
                        if not printed_reasoning_content:
                            printed_reasoning_content = True
                            print("reasoning_content:", end="", flush=True)
                        print(reasoning_content, end="", flush=True)
                    elif content:
                        if not printed_content:
                            printed_content = True
                            print("\ncontent:", end="", flush=True)
                        # Extract and print the content
                        print(content, end="", flush=True)
                except json.JSONDecodeError:
                    print("Error decoding JSON:", decoded_line)
else:
    print(f"Error: {response.status_code} - {response.text}")

Example output (run from the streaming snippet above)

reasoning_content:
This is a straightforward comparison of two numbers. The task is to determine which is larger: 9.11 or 9.8. 

I need to recall the value of these decimal numbers and compare them. Given both are very close, it requires precise comprehension to understand which has the larger value—specifically focusing on the tenths and hundredths places.


content:

9.8 is greater than 9.11. 

Let's break down the comparison:

- Both numbers are above 9, so we're comparing the decimal parts.
- 9.11 has a '11' in the hundredths place.
- 9.8 has an '80' in the hundredths place, which is larger (even if it's ten times, 80 > 11).

Therefore, 9.8 > 9.11.
client: Stream completed.

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[mgoin]

Nice use of this new feature! Will try in a bit cc @gaocegege

[gaocegege]

Thanks for the contribution!

Could you please rebase the upstream? In a previous PR to support reasoning outputs in structured outputs https://github.com/vllm-project/vllm/pull/12955/files#diff-ea8b8ff63961713ccb62d78e53e96404b587b7828cb9fee08a9e5576bf563673R1065, we moved the CLI argument --reasoning-parser to https://github.com/vllm-project/vllm/blob/main/vllm/engine/arg_utils.py#L1076

Thus you may need to add a new choice there.

[gaocegege]

Hi, I updated the docs in this PR #14114

Maybe you should rebase the docs too. Just FYI

[alex-jw-brooks]

Adding a warning for now since this is already a large PR, but I think adding a GraniteReasoner for guided decoding could be a follow-up later?

[gaocegege]

SGTM!

[aarnphm]

We can just fail for now.

[alex-jw-brooks]

@aarnphm This will behave the same way as models with no reasoner. The intention here was mostly to clarify that there isn't a reasoning backend for granite in case users conflate it with --enable-reasoning / --reasoning-parser granite being supported for these models

[aarnphm]

wfm.

[alex-jw-brooks]

Awesome thanks @gaocegege! It's been rebased 😄

[gaocegege]

Thanks for your contribution! 🎉 👍

[gaocegege]

@mgoin Please give it another review, thanks!

[b8zhong]

Suggested change

	response_content = current_text[current_chunk_end + 1:]
	response_content = current_text[current_chunk_end + 1:]
	parsed_content = True

parsed_content flag doesn't seem to be updated, so maybe helpful to set it?
Very minor suggestion, totally optional

[alex-jw-brooks]

Hey @b8zhong, thanks for the suggestion! For now, I'd prefer to keep it as is since it returns immediately after parsing the response content. I.e., once this condition is met, there is no need to keep going, so updating the flag won't do anything 🙂

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @alex-jw-brooks.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[gaocegege]

@alex-jw-brooks Hi could you please resolve the conflicts

[aarnphm]

tiny comments,otherwise lgtm.

[aarnphm]

Can we rever this to reduce code change? thanks.

[alex-jw-brooks]

Sure, done!

[aarnphm]

We can just fail for now.

[alex-jw-brooks]

Thanks @aarnphm - it's ready for another look when you have a moment 🙂

[aarnphm]

great work!

[alex-jw-brooks]

Hi @mgoin, can you please take a look at this PR when you have a moment?

[gaocegege]

@mgoin @simon-mo Could you please take a look at this?

[DarkLight1337]

Since this is basically coming from the model vendor, I'll just stamp it. The code looks reasonable to me

[DarkLight1337]

Can you fix the merge conflicts?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @alex-jw-brooks.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[alex-jw-brooks]

Thanks for the review @DarkLight1337! Sure should be resolved now 🤞