Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .chloggen/schema_of_gen_ai_tool_definitions.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Use this changelog template to create an entry for release notes.
#
# If your change doesn't affect end users you should instead start
# your pull request title with [chore] or use the "Skip Changelog" label.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the area of concern in the attributes-registry, (e.g. http, cloud, db)
component: gen-ai

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Enhance the definition of `gen_ai.tool.definitions` attribute.

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
# The values here must be integers.
issues: [2721, 1835]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: |
The schema of `gen_ai.tool.definitions` attribute is now enhanced to:
- Add JSON schema of `gen_ai.tool.definitions` attribute.
- Document the behavior of capturing tool definitions.
- Capture the tool definitions in a simplified format if the content capturing is disabled.
12 changes: 7 additions & 5 deletions docs/gen-ai/anthropic.md
Original file line number Diff line number Diff line change
Expand Up @@ -189,13 +189,15 @@ section for more details.

**[16] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

---

Expand Down
12 changes: 7 additions & 5 deletions docs/gen-ai/aws-bedrock.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,13 +204,15 @@ section for more details.

**[17] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

The following attributes can be important for making sampling decisions
and SHOULD be provided **at span creation time** (if provided at all):
Expand Down
12 changes: 7 additions & 5 deletions docs/gen-ai/azure-ai-inference.md
Original file line number Diff line number Diff line change
Expand Up @@ -206,13 +206,15 @@ section for more details.

**[17] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

The following attributes can be important for making sampling decisions
and SHOULD be provided **at span creation time** (if provided at all):
Expand Down
12 changes: 7 additions & 5 deletions docs/gen-ai/gen-ai-agent-spans.md
Original file line number Diff line number Diff line change
Expand Up @@ -363,13 +363,15 @@ section for more details.

**[18] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

The following attributes can be important for making sampling decisions
and SHOULD be provided **at span creation time** (if provided at all):
Expand Down
12 changes: 7 additions & 5 deletions docs/gen-ai/gen-ai-events.md
Original file line number Diff line number Diff line change
Expand Up @@ -185,13 +185,15 @@ section for more details.

**[16] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

---

Expand Down
12 changes: 7 additions & 5 deletions docs/gen-ai/gen-ai-spans.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,13 +218,15 @@ section for more details.

**[17] `gen_ai.tool.definitions`:** The value of this attribute matches source system tool definition format.

It's expected to be an array of objects where each object represents a tool definition. In case a serialized string is available
to the instrumentation, the instrumentation SHOULD do the best effort to
deserialize it to an array. When recorded on spans, it MAY be recorded as a JSON string if structured format is not supported and SHOULD be recorded in structured form otherwise.
Instrumentations MUST follow [Tool Definitions JSON Schema](/docs/gen-ai/gen-ai-tool-definitions.json).

When the attribute is recorded on events, it MUST be recorded in structured
form. When recorded on spans, it MAY be recorded as a JSON string if structured
format is not supported and SHOULD be recorded in structured form otherwise.

Since this attribute could be large, it's NOT RECOMMENDED to populate
it by default. Instrumentations MAY provide a way to enable
populating this attribute.
non-required properties by default. Instrumentations MAY provide a way
to enable populating optional properties.

The following attributes can be important for making sampling decisions
and SHOULD be provided **at span creation time** (if provided at all):
Expand Down
86 changes: 86 additions & 0 deletions docs/gen-ai/gen-ai-tool-definitions.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
{
"$defs": {
"FunctionToolDefinition": {
"additionalProperties": true,
"description": "Represents a tool definition in the form of a function.",
"properties": {
"type": {
"const": "function",
"description": "The type of the tool.",
"title": "Type",
"type": "string"
},
"name": {
"description": "The name of the tool.",
"title": "Name",
"type": "string"
},
"description": {
"anyOf": [
{
"type": "string"
},
{
"type": "null"
}
],
"default": null,
"description": "The description of the tool.Since this attribute could be large, it's NOT RECOMMENDED to be populated by default.Instrumentations MAY provide a way to enable populating this property.",
"title": "Description"
},
"parameters": {
"anyOf": [
{},

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason for the parameters field to be anything and not sth more defined like a dict?

If it is Any, how would one parse this field if there is nothing known about the structure of it?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Following on from that

  1. Are there any real world systems that don't use JsonSchema for this? Maybe we can make it a MUST.
  2. If there are, can we have a mutually exclusive variant that is strictly required to be JsonSchema?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And I think there should be a way to make pydantic reference the external schema http://json-schema.org/draft-07/schema# so that the generated schema is fully defined

{
"type": "null"
}
],
"default": null,
"description": "Schema that defines the parameters accepted by the tool. The RECOMMENDED format is JSON Schema.Since this attribute could be large, it's NOT RECOMMENDED to be populated by default.Instrumentations MAY provide a way to enable populating this property.",
"title": "Parameters"
}
},
"required": [
"type",
"name"
],
"title": "FunctionToolDefinition",
"type": "object"
},
"GenericToolDefinition": {
"additionalProperties": true,
"description": "Represents a tool definition in any forms.",
"properties": {
"type": {
"description": "The type of the tool.",
"title": "Type",
"type": "string"
},
"name": {
"description": "The name of the tool.",
"title": "Name",
"type": "string"
}
},
"required": [
"type",
"name"
],
"title": "GenericToolDefinition",
"type": "object"
}
},
"description": "Represents the list of tool definitions available to the GenAI agent or model.",
"items": {
"anyOf": [
{
"$ref": "#/$defs/FunctionToolDefinition"
},
{
"$ref": "#/$defs/GenericToolDefinition"
}
]
},
"title": "ToolDefinitions",
"type": "array"
}
93 changes: 65 additions & 28 deletions docs/gen-ai/non-normative/examples-llm-calls.md
Original file line number Diff line number Diff line change
Expand Up @@ -329,19 +329,31 @@ They are likely to be siblings if there is an encompassing span.

**GenAI client span 1:**

| Property | Value |
| -------------------------------- | ------------------------------------------ |
| Span name | `"chat gpt-4"` |
| `gen_ai.provider.name` | `"openai"` |
| `gen_ai.operation.name` | `"chat"` |
| `gen_ai.request.model` | `"gpt-4"` |
| `gen_ai.request.max_tokens` | `200` |
| `gen_ai.request.top_p` | `1.0` |
| `gen_ai.response.id` | `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` |
| `gen_ai.response.model` | `"gpt-4-0613"` |
| `gen_ai.usage.output_tokens` | `17` |
| `gen_ai.usage.input_tokens` | `47` |
| `gen_ai.response.finish_reasons` | `["tool_calls"]` |
| Property | Value |
| -------------------------------- | ---------------------------------------------------------------------- |
| Span name | `"chat gpt-4"` |
| `gen_ai.provider.name` | `"openai"` |
| `gen_ai.operation.name` | `"chat"` |
| `gen_ai.request.model` | `"gpt-4"` |
| `gen_ai.request.max_tokens` | `200` |
| `gen_ai.request.top_p` | `1.0` |
| `gen_ai.response.id` | `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` |
| `gen_ai.response.model` | `"gpt-4-0613"` |
| `gen_ai.usage.output_tokens` | `17` |
| `gen_ai.usage.input_tokens` | `47` |
| `gen_ai.response.finish_reasons` | `["tool_calls"]` |
| `gen_ai.tool.definitions` | [`gen_ai.tool.definitions`](#gen-ai-tool-definitions-tool-call-span-0) |

<span id="gen-ai-tool-definitions-tool-call-span-0">`gen_ai.tool.definitions` value</span>

```json
[
{
"type": "function",
"name": "get_weather"
}
]
```

**Tool call:**

Expand Down Expand Up @@ -377,21 +389,22 @@ They are likely to be siblings if there is an encompassing span.

**GenAI client span 1:**

| Property | Value |
| -------------------------------- | -------------------------------------------------------------------- |
| Span name | `"chat gpt-4"` |
| `gen_ai.provider.name` | `"openai"` |
| `gen_ai.operation.name` | `"chat"` |
| `gen_ai.request.model` | `"gpt-4"` |
| `gen_ai.request.max_tokens` | `200` |
| `gen_ai.request.top_p` | `1.0` |
| `gen_ai.response.id` | `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` |
| `gen_ai.response.model` | `"gpt-4-0613"` |
| `gen_ai.usage.output_tokens` | `17` |
| `gen_ai.usage.input_tokens` | `47` |
| `gen_ai.response.finish_reasons` | `["tool_calls"]` |
| `gen_ai.input.messages` | [`gen_ai.input.messages`](#gen-ai-input-messages-tool-call-span-1) |
| `gen_ai.output.messages` | [`gen_ai.output.messages`](#gen-ai-output-messages-tool-call-span-1) |
| Property | Value |
| -------------------------------- | ---------------------------------------------------------------------- |
| Span name | `"chat gpt-4"` |
| `gen_ai.provider.name` | `"openai"` |
| `gen_ai.operation.name` | `"chat"` |
| `gen_ai.request.model` | `"gpt-4"` |
| `gen_ai.request.max_tokens` | `200` |
| `gen_ai.request.top_p` | `1.0` |
| `gen_ai.response.id` | `"chatcmpl-9J3uIL87gldCFtiIbyaOvTeYBRA3l"` |
| `gen_ai.response.model` | `"gpt-4-0613"` |
| `gen_ai.usage.output_tokens` | `17` |
| `gen_ai.usage.input_tokens` | `47` |
| `gen_ai.response.finish_reasons` | `["tool_calls"]` |
| `gen_ai.input.messages` | [`gen_ai.input.messages`](#gen-ai-input-messages-tool-call-span-1) |
| `gen_ai.output.messages` | [`gen_ai.output.messages`](#gen-ai-output-messages-tool-call-span-1) |
| `gen_ai.tool.definitions` | [`gen_ai.tool.definitions`](#gen-ai-tool-definitions-tool-call-span-1) |

<span id="gen-ai-input-messages-tool-call-span-1">`gen_ai.input.messages` value</span>

Expand Down Expand Up @@ -430,6 +443,30 @@ They are likely to be siblings if there is an encompassing span.
]
```

<span id="gen-ai-tool-definitions-tool-call-span-1">`gen_ai.tool.definitions` value</span>

```json
[
{
"type": "function",
"name": "get_weather",
"description": "Get the weather in a given location",
"parameters": {
"type": "object",
"properties": {
"location": {
"type": "string",
"description": "The city and state, e.g. San Francisco, CA"
},
"required": [
"location"
]
}
}
}
]
```

**Tool call:**

If tool call is [instrumented according to execute-tool span definition](/docs/gen-ai/gen-ai-spans.md#execute-tool-span), it may look like this:
Expand Down
Loading
Loading