fix: Fix deprecated max_tokens param in openai ChatCompletionRequest

[mickqian]

Motivation

Address #3098

Modifications

1. replace the deprecated max_tokens param with a newer one: max_completion_tokens, according to here

Checklist

• Format your code according to the Code Formatting with Pre-Commit.
• Add unit tests as outlined in the Running Unit Tests.
• Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
• Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling.

[merrymercy]

Can you only update the adapter and not change other parts?

[ywang96]

Much thanks for addressing #3098 so quickly and FYI I've tested this branch and it does resolve the issue!

[mickqian]

Can you only update the adapter and not change other parts?

Updated. Removed the frontend part

[zhaochenyang20]

@mickqian mick I will take a look on this. Thanks!

[zhaochenyang20]

wondering what's the definition of max_tokens in non-chat model? And why we do not keep max_tokens for non-chat models in the backend?

[mickqian]

1. the def of max_tokens in completion models is tokens that can be generated in the completion, pretty much the same as max_completion_tokens
2. Yes we should keep both params, updated

[zhaochenyang20]

Hey. For embedding model, “max_tokens” means the max sequence length that can be processed. What if it exceeds the length? Should the sequence be truncated or throw an error? The chat model also. And, I personally think we should call it generation model and embedding model. That's what we typically call these models.

[mickqian]

1. for embedding models, I think both ways will do, depending on the situation. Providing an option sounds good too
2. Yes, completion model and chat completion model fall into the category of generation model, I thought you were referring to completion model. The is_chat_model is used to distinguish different generation models, if I'm correct. For sglang.lang, does it involve embedding models(or did I miss something?)? If not, probably is_chat_model would suffice for generation models in backend

[zhaochenyang20]

Okay. This should be sound to me. But, in my ideal:

def to_openai_kwargs(self, is_chat_model):

You mean is_chat_model is an element of class OpenAI(BaseBackend) in python/sglang/lang/backend/openai.py rather than an element of class SglSamplingParams in python/sglang/lang/ir.py ?

So this function is not:

def to_openai_kwargs(self):

Right? That make sense. But I prefer the latter one if we can.

[mickqian]

yes it's supposed to be an internal property of a BaseBackend( as it directly describes the model ), and passed to SglSamplingParams for generating openai-compatible request.
While adding this field to SglSamplingParams do sounds good in some cases, I personally reckon SglSamplingParams is meant to be a model-unaware data, which can be sent to different backends, and let the actual backend decides the final openai request? Feel free to correct me.

[zhaochenyang20]

I agree! Nice work.

[zhaochenyang20]

1. I think we should comment on the protocal.py or anywhere regarding the definition of max_completion_tokens and the difference before previous max_tokens.

2. Could you run the docs CI locally, just make compile is enough. Current docs CI is closed due to long queue time to compile on CI. But we should run it locally.

[zhaochenyang20]

Okay. This should be sound to me. But, in my ideal:

def to_openai_kwargs(self, is_chat_model):

You mean is_chat_model is an element of class OpenAI(BaseBackend) in python/sglang/lang/backend/openai.py rather than an element of class SglSamplingParams in python/sglang/lang/ir.py ?

So this function is not:

def to_openai_kwargs(self):

Right? That make sense. But I prefer the latter one if we can.

[mickqian]

1. I think we should comment on the protocal.py or anywhere regarding the definition of max_completion_tokens and the difference before previous max_tokens.
2. Could you run the docs CI locally, just make compile is enough. Current docs CI is closed due to long queue time to compile on CI. But we should run it locally.

make compile failed even in main branch:

$ jupyter nbconvert --to notebook --execute --inplace ./backend/openai_api_completions.ipynb \
                                --ExecutePreprocessor.timeout=600 \
                                --ExecutePreprocessor.kernel_name=python3 || exit 1; 
                                
...
AttributeError                            Traceback (most recent call last)
Cell In[9], line 58
     52 batch_details = client.batches.retrieve(batch_id=batch_job.id)
     54 print_highlight(
     55     f"Batch job details (check {i+1} / {max_checks}) // ID: {batch_details.id} // Status: {batch_details.status} // Created at: {batch_details.created_at} // Input file ID: {batch_details.input_file_id} // Output file ID: {batch_details.output_file_id}"
     56 )
     57 print_highlight(
---> 58     f"<strong>Request counts: Total: {batch_details.request_counts.total} // Completed: {batch_details.request_counts.completed} // Failed: {batch_details.request_counts.failed}</strong>"
     59 )
     61 time.sleep(3)

AttributeError: 'NoneType' object has no attribute 'total'

Is it a known issue, or the error is from my side?

[mickqian]

1. I think we should comment on the protocal.py or anywhere regarding the definition of max_completion_tokens and the difference before previous max_tokens.
2. Could you run the docs CI locally, just make compile is enough. Current docs CI is closed due to long queue time to compile on CI. But we should run it locally.

make compile failed even in main branch:

$ jupyter nbconvert --to notebook --execute --inplace ./backend/openai_api_completions.ipynb \
                                --ExecutePreprocessor.timeout=600 \
                                --ExecutePreprocessor.kernel_name=python3 || exit 1; 
                                
...
AttributeError                            Traceback (most recent call last)
Cell In[9], line 58
     52 batch_details = client.batches.retrieve(batch_id=batch_job.id)
     54 print_highlight(
     55     f"Batch job details (check {i+1} / {max_checks}) // ID: {batch_details.id} // Status: {batch_details.status} // Created at: {batch_details.created_at} // Input file ID: {batch_details.input_file_id} // Output file ID: {batch_details.output_file_id}"
     56 )
     57 print_highlight(
---> 58     f"<strong>Request counts: Total: {batch_details.request_counts.total} // Completed: {batch_details.request_counts.completed} // Failed: {batch_details.request_counts.failed}</strong>"
     59 )
     61 time.sleep(3)

AttributeError: 'NoneType' object has no attribute 'total'

Is it a known issue, or the error is from my side?

Fixed, and make compile test passed locally.

[zhaochenyang20]

Will review it today. Stay tuned!

[zhaochenyang20]

Good improvment. I am wondering do openai still have max tokens for chat API right now? For chat completion api, there should only be max completion tokens. But I don't know what's for chat API.

[zhaochenyang20]

I agree! Nice work.

[zhaochenyang20]

1. Make it a full sentence: Non-chat-completion models only have max tokens.

2. So chat completion models count the max_completion_tokens and chat models (not completion models) count the max_token right?

[mickqian]

1. There's a nuance difference between non-chat-completion models only and Non-chat-completion models only have max tokens, I'm afraid. Changed to Only available for non-chat-completion models, is that ok?
2. yes, to be more specific, in openai's legacy completion api(non-chat completion models only), they only have max_tokens. In their chat-completion api, they have both two params, but:

[mickqian]

Good improvment. I am wondering do openai still have max tokens for chat API right now? For chat completion api, there should only be max completion tokens. But I don't know what's for chat API.

replied in here

[zhaochenyang20]

@mickqian This should be rebased. Thanks 😂

[zhaochenyang20]

@shuaills Could you review this? Thnaks!

[shuaills]

LGTM @zhaochenyang20 Can you help rerun the CI?

[zhaochenyang20]

@mickqian @shuaills keep this CI here. Shuai, plz approve at https://github.com/sgl-project/sglang/pull/3122/files

I will rerun the CI

[mickqian]

@zhaochenyang20

[zhaochenyang20]

@mickqian will help to merge after the CI

[zhaochenyang20]

https://github.com/sgl-project/sglang/actions/runs/13913981893/job/38933560619?pr=3122

@mickqian this should be fixed

[zhaochenyang20]

@mickqian I've rerun the CI. Thanks so much. Do not rebase with main. We can merge this after the CI.

[CatherineSue]

is this PR gonna be merged soon?

[zhyncs]

Can you only update the adapter and not change other parts?

@mickqian Can you do this first? Use other parts for different PRs. Do not mix them together.