Other Format of Tool Output

[peiwenYe]

Hello, currently, three formats of tool output are supported. Are you considering adding other types, such as audio, code, file, and chart (the difference from image is that it can be edited in real time) , etc.

[colesmcintosh]

Good point about considering additional output formats.

For most output types like code, charts, and files, you could indeed pass them effectively as images or text. An image can capture a code snippet's syntax highlighting, a chart's visual representation, or a file's contents, etc...

Could you share more about the specific types of outputs you're hoping to see supported and how you envision the real time edit feature being used?

[peiwenYe]

Thanks for your reply.

For example, when a voice file is output, it is hoped that there will be a voice tag and it can be played (of course this also depends on the client).

I have tried to use ImageContent to pass audio data, but unfortunately, Claude reports an error saying that audio/x-wav is not a viable image mime_type.

As for chart, which can be edit in real time, it just like

Click on different icons in the upper right corner to display different types of tables, including line charts, bar charts, pie charts, etc. In the bar chart, when I place the mouse on the bar, the detailed information of the bar will be displayed in real time.

Of course, we can pass most of output by TextContent, but if we can add other types, we won’t need too many converts and converts. And now, code, file, url, reference, chart, audio, json are needed.

[dsp-ant]

We will stick to high level primitives such as Text, Images, Audio. We will not support specific semantical types such as charts, code ,etc as it will be impossible to define these. You can use _meta annotations to provide additional informations to client, but most likely, it is best to hint the model by defining your tool output as for example a json that contains a description:

{
  "description": "The following result is a chart",
  "chart": {
  }
}

[peiwenYe]

Got it.
Can you please share the formats you plan to add, such as voice, and an approximate release time? We'd love to contribute code to the repo if possible.