Add tools to api_server for InternLM2 model #1763

AllentDan · 2024-06-12T07:02:45Z

No description provided.

Conflicts: lmdeploy/serve/openai/api_server.py

lvhan028 · 2024-06-13T04:26:19Z

Pls fix the UT test error and the PR test error

lvhan028 · 2024-06-24T03:16:17Z

docs/en/serving/api_server.md

+  }
+]
+messages = [{"role": "user", "content": "What's the weather like in Boston today?"}]
+tool_choice={"type": "function", "function": {"name": "get_current_weather"}}


Do we support tool_choice?

Only supports none or specifying a particular tool in json format.

lvhan028 · 2024-06-25T05:19:11Z

lmdeploy/serve/openai/api_server.py

@@ -434,10 +451,21 @@ async def chat_completions_v1(request: ChatCompletionRequest,
        stop_words=request.stop,
        skip_special_tokens=request.skip_special_tokens)

+    tools = None
+    if request.tools and request.tool_choice != 'none':


Shall we make error response to the client when request.tool_choice is not None?

I've made it in the check_request function.

lvhan028 · 2024-06-25T05:46:52Z

Do we support "finish_reason": "tool_calls"?

AllentDan · 2024-06-25T05:53:13Z

Do we support "finish_reason": "tool_calls"?

No, we did not yet.

AllentDan · 2024-06-25T06:15:57Z

@Harold-lkk May help review this PR。

AllentDan · 2024-06-25T09:10:29Z

I got this prompt string if I used the code snippet in our doc.

<|im_start|>system\nYou are an AI assistant whose name is InternLM (书生·浦语).\n- InternLM (书生·浦语) is a conversational language model that is developed by Shanghai AI Laboratory (上海人工智能实验室). It is designed to be helpful, honest, and harmless.\n- InternLM (书生·浦语) can understand and communicate fluently in the language chosen by the user such as English and 中文.\n<|im_end|>\n<|im_start|>system name=<|plugin|>\n[{"description": "Get the current weather in a given location", "name": "get_current_weather", "parameters": {"type": "object", "properties": {"location": {"type": "string", "description": "The city and state, e.g. San Francisco, CA"}, "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}}, "required": ["location"]}}]<|im_end|>\n<|im_start|>user\nWhat\'s the weather like in Boston today?<|im_end|>\n<|im_start|>assistant\n

And only internlm2-chat1_8b return the desired output:

ChatCompletion(id='1', choices=[Choice(finish_reason='tool_calls', index=0, logprobs=None, message=ChatCompletionMessage(content='', role='assistant', function_call=None, tool_calls=[ChatCompletionMessageToolCall(id='1', function=Function(arguments={'location': 'Boston'}, name='get_current_weather'), type='function')]))], created=1719306194, model='/nvme/shared_data/InternLM/internlm2-chat-1_8b', object='chat.completion', system_fingerprint=None, usage=CompletionUsage(completion_tokens=23, prompt_tokens=209, total_tokens=232))

AllentDan · 2024-06-25T09:26:41Z

It turns out that we have to pass skip_special_tokens=False in the client.

lvhan028 · 2024-06-26T07:13:53Z

@zhulinJulia24 we should add test cases for internlm2 function_call

docs/en/serving/api_server_tools.md

lmdeploy/serve/openai/api_server.py

lvhan028 · 2024-07-01T12:40:15Z

lmdeploy/serve/openai/protocol.py

+
+class ToolCall(BaseModel):
+    """Tool call response."""
+    id: str


According to the openai spec, id refers to "The ID of the tool call".
In my test, it increases by 1 automatically.
I view the id as the index of the invoked tool in the tool list. But I am not sure about it.
@Harold-lkk, can you clarify it?

GPT supports multi-function calls at the same time， so the id is used to identify which function call of the response

GPT supports multi-function calls at the same time， so the id is used to identify which function call of the response

Does that mean we have to return 0 for all requests?

I see, we have to find out the index of the returned function in tools list.

What if two functions share the same name? Can it happen or do we have to consider this possibility? @Harold-lkk

lvhan028 · 2024-07-01T12:58:42Z

The test_messages2prompt4internlm2_chat didn't cover the if tools statement. Please add UT for the if tools branch

lvhan028 · 2024-07-01T13:03:19Z

docs/en/serving/api_server_tools.md

+    top_p=0.8,
+    stream=False,
+    tools=tools,
+    extra_body={'skip_special_tokens': False})


Can we hide the extra_body in the implementation instead of the API?
I don't find it in the openai API spec.

extra_body is common when calling openai api. There are documentations in vllm too. https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#extra-parameters

lvhan028 · 2024-07-01T13:11:21Z

lmdeploy/serve/openai/api_server.py

+    if request.tools and request.tool_choice != 'none':
+        if request.stream is True:
+            logger.warning('Set stream to False for tools')
+            request.stream = False


why change it to False?

I suppose it only supports stream=False. In streaming mode, how can client call the function? And it would be hard to extract content inside too.

Can the client call the function when the finish_reason is tool_calls?

lvhan028 · 2024-07-01T13:15:41Z

There is warning for each request

[TM][WARNING] [ProcessInferRequests] [1] total sequence length (209 + 65335) exceeds `session_len` (65544), `request_output_len` is truncated to 65334

I tried the get_current_weather example

lmdeploy/model.py

lmdeploy/serve/openai/api_server.py

RunningLeon · 2024-07-04T07:21:05Z

docs/en/serving/api_server.md

@@ -173,6 +173,10 @@ for message in messages:
        print(item)
 ```

+### Tools


do we need to add the md file to index.rst since pr #1880 is merged. May need to sync with main branch.

Conflicts: examples/vl/qwen_model.py examples/vl/xcomposer_model.py

lmdeploy/serve/openai/protocol.py

lvhan028 · 2024-07-04T15:55:51Z

@zhulinJulia24 may add TC

Conflicts: lmdeploy/model.py

AllentDan added 4 commits June 12, 2024 14:19

Add tools to api_server for InternLM2 model

22f65b3

Update documents

1274224

Merge branch 'main' into tools

4325f5c

Conflicts: lmdeploy/serve/openai/api_server.py

update doc

9548728

fix UT

824b227

lvhan028 added the enhancement New feature or request label Jun 13, 2024

fix

e9ef005

lvhan028 requested review from irexyc and lvhan028 June 20, 2024 20:42

lvhan028 reviewed Jun 24, 2024

View reviewed changes

lvhan028 reviewed Jun 25, 2024

View reviewed changes

lvhan028 requested a review from Harold-lkk June 25, 2024 06:16

AllentDan added 2 commits June 25, 2024 15:01

fix comments

56efee6

remove type in prompt

075e0a2

AllentDan added 2 commits June 25, 2024 18:33

doc

bcca627

update doc

a5503a5

lvhan028 reviewed Jul 1, 2024

View reviewed changes

docs/en/serving/api_server_tools.md Outdated Show resolved Hide resolved

lvhan028 reviewed Jul 1, 2024

View reviewed changes

lmdeploy/serve/openai/api_server.py Outdated Show resolved Hide resolved

lvhan028 requested a review from RunningLeon July 1, 2024 06:32

update doc and logger

b1082e1

lvhan028 reviewed Jul 1, 2024

View reviewed changes

AllentDan added 2 commits July 2, 2024 14:57

update ut

a1818a8

set skip_scpecial_tokens to False manually

8ea46bf

RunningLeon reviewed Jul 3, 2024

View reviewed changes

lmdeploy/model.py Outdated Show resolved Hide resolved

RunningLeon reviewed Jul 3, 2024

View reviewed changes

lmdeploy/serve/openai/api_server.py Outdated Show resolved Hide resolved

comments

e21ecf0

RunningLeon reviewed Jul 4, 2024

View reviewed changes

AllentDan added 2 commits July 4, 2024 16:13

Merge branch 'main' into tools

0fe44bf

Conflicts: examples/vl/qwen_model.py examples/vl/xcomposer_model.py

update index.rst

d379968

lvhan028 reviewed Jul 4, 2024

View reviewed changes

lmdeploy/serve/openai/protocol.py Outdated Show resolved Hide resolved

set arguments str

ae3fd4c

lvhan028 approved these changes Jul 4, 2024

View reviewed changes

id

182c233

Harold-lkk approved these changes Jul 9, 2024

View reviewed changes

Merge branch 'main' into tools

23ada9b

Conflicts: lmdeploy/model.py

lvhan028 merged commit c12786b into InternLM:main Jul 9, 2024
3 of 5 checks passed

lvhan028 mentioned this pull request Jul 10, 2024

[Feature] 为internVL2添加function calling能力（Tools能力） #1987

Closed

zhyncs mentioned this pull request Jul 10, 2024

[Feature] support function calling #1800

Closed

zhulinJulia24 mentioned this pull request Jul 16, 2024

update daily testcase new #2035

Merged

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tools to api_server for InternLM2 model #1763

Add tools to api_server for InternLM2 model #1763

AllentDan commented Jun 12, 2024

lvhan028 commented Jun 13, 2024

lvhan028 Jun 24, 2024

AllentDan Jun 24, 2024

lvhan028 Jun 25, 2024 •

edited

Loading

AllentDan Jun 25, 2024

lvhan028 commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

lvhan028 commented Jun 26, 2024

lvhan028 Jul 1, 2024 •

edited

Loading

Harold-lkk Jul 5, 2024

AllentDan Jul 5, 2024 •

edited

Loading

AllentDan Jul 5, 2024

AllentDan Jul 5, 2024

lvhan028 commented Jul 1, 2024

lvhan028 Jul 1, 2024

AllentDan Jul 2, 2024

lvhan028 Jul 1, 2024

AllentDan Jul 2, 2024

lvhan028 Jul 2, 2024 •

edited

Loading

lvhan028 commented Jul 1, 2024

RunningLeon Jul 4, 2024

lvhan028 commented Jul 4, 2024

Add tools to api_server for InternLM2 model #1763

Add tools to api_server for InternLM2 model #1763

Conversation

AllentDan commented Jun 12, 2024

lvhan028 commented Jun 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvhan028 Jun 25, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvhan028 commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

AllentDan commented Jun 25, 2024

lvhan028 commented Jun 26, 2024

lvhan028 Jul 1, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AllentDan Jul 5, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvhan028 commented Jul 1, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lvhan028 Jul 2, 2024 • edited Loading

Choose a reason for hiding this comment

lvhan028 commented Jul 1, 2024

Choose a reason for hiding this comment

lvhan028 commented Jul 4, 2024

lvhan028 Jun 25, 2024 •

edited

Loading

lvhan028 Jul 1, 2024 •

edited

Loading

AllentDan Jul 5, 2024 •

edited

Loading

lvhan028 Jul 2, 2024 •

edited

Loading