Skip to content

fix: add structured outputs schema logging for Anthropic and Gemini #3454

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nirga merged 11 commits into from
Nov 23, 2025
Merged

fix: add structured outputs schema logging for Anthropic and Gemini #3454

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
1de9ffa
feat: add structured outputs schema logging for Anthropic and Gemini
nirga Nov 20, 2025
d6360b2
fix: remove unused imports in sample apps
nirga Nov 20, 2025
ca5f423
test: add structured outputs tests for Anthropic with VCR
nirga Nov 20, 2025
6f12631
Merge branch 'main' into feat/structured-outputs-logging
nirga Nov 23, 2025
13f1e33
chore: bump anthropic SDK to 0.74.1 for structured outputs support
nirga Nov 23, 2025
0f2d11a
fix: update Event API for OpenTelemetry SDK 1.38.0 compatibility
nirga Nov 23, 2025
f5ca45f
chore: remove skip decorator from structured outputs tests
nirga Nov 23, 2025
c91f7c6
fix: update Anthropic SDK minimum version to 0.74.0 for structured ou…
nirga Nov 23, 2025
8aa83f7
Merge branch 'main' into feat/structured-outputs-logging
nirga Nov 23, 2025
cd7f14d
revert: undo Event API changes to event_emitter.py and test_messages.py
nirga Nov 23, 2025
d8c8b35
fix(anthropic): correct structured outputs API format and update tests
nirga Nov 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions ...telemetry-instrumentation-anthropic/opentelemetry/instrumentation/anthropic/span_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,17 @@ async def aset_input_attributes(span, kwargs):
span, f"{prefix}.input_schema", json.dumps(input_schema)
)

output_format = kwargs.get("output_format")
if output_format and isinstance(output_format, dict):
if output_format.get("type") == "json_schema":
schema = output_format.get("schema")
if schema:
set_span_attribute(
span,
SpanAttributes.LLM_REQUEST_STRUCTURED_OUTPUT_SCHEMA,
json.dumps(schema),
)


async def _aset_span_completions(span, response):
if not should_send_prompts():
Expand Down
1,719 changes: 985 additions & 734 deletions packages/opentelemetry-instrumentation-anthropic/poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion packages/opentelemetry-instrumentation-anthropic/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ pytest = "^8.2.2"
pytest-sugar = "1.0.0"

[tool.poetry.group.test.dependencies]
anthropic = {extras = ["bedrock"], version = ">=0.36.0"}
anthropic = {extras = ["bedrock"], version = ">=0.74.0"}
pytest = "^8.2.2"
pytest-sugar = "1.0.0"
vcrpy = "^6.0.1"
Expand Down
108 changes: 108 additions & 0 deletions ...pic/tests/cassettes/test_structured_outputs/test_anthropic_structured_outputs_legacy.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
interactions:
- request:
body: '{"max_tokens":1024,"messages":[{"role":"user","content":"Tell me a joke
about OpenTelemetry and rate it from 1 to 10"}],"model":"claude-sonnet-4-5-20250929","output_format":{"type":"json_schema","schema":{"type":"object","properties":{"joke":{"type":"string","description":"A
joke about OpenTelemetry"},"rating":{"type":"integer","description":"Rating
of the joke from 1 to 10"}},"required":["joke","rating"],"additionalProperties":false}}}'
headers:
accept:
- application/json
accept-encoding:
- gzip, deflate
anthropic-beta:
- structured-outputs-2025-11-13
anthropic-version:
- '2023-06-01'
connection:
- keep-alive
content-length:
- '440'
content-type:
- application/json
host:
- api.anthropic.com
user-agent:
- Anthropic/Python 0.74.1
x-stainless-arch:
- arm64
x-stainless-async:
- 'false'
x-stainless-lang:
- python
x-stainless-os:
- MacOS
x-stainless-package-version:
- 0.74.1
x-stainless-read-timeout:
- '600'
x-stainless-retry-count:
- '0'
x-stainless-runtime:
- CPython
x-stainless-runtime-version:
- 3.11.7
x-stainless-timeout:
- '600'
method: POST
uri: https://api.anthropic.com/v1/messages?beta=true
response:
body:
string: !!binary |
H4sIAAAAAAAAAwAAAP//dJFPaxsxEMW/ynTOMtjbuK11CaTnkmASeuiWRZFevKq10lYapdma/e5l
TU3/0dPA+703j2FOPCSHwJptMNVhVVKMkNXVartq1s12vWt2rNg71jyUQ7fe7PdV5P3mw3e32T/c
TXdv3g32ZsuKZRqxuFCKOYAV5xQWwZTii5gorNimKIjC+tPp4he8LOQ8NJ9a/pKOaFlTyx/7iZx3
JD3I4Rkhjcj0mGGOVEf65qWn2xHxHgEDJE/XdANrasESmag3jiQlGkycSLKxKJSeFuYz4YV8PK8O
6VBetayo5WzEx8PS/nbm+bPiImnsMkxJkTUjuk5qjvwTFHytiBasYw1BcT1frk/s41ilk3RELKyb
plFsje3R2QwjPsXuT8f6wjOM+x+7ZJcCjD0GZBO67fCv/xfd9H/TWXGq8rv0eqe4ID97i048Mmte
/uVMdjzPPwAAAP//AwD+H/OiIgIAAA==
headers:
CF-RAY:
- 9a30f7a7b8d37da0-TLV
Connection:
- keep-alive
Content-Encoding:
- gzip
Content-Type:
- application/json
Date:
- Sun, 23 Nov 2025 13:21:05 GMT
Server:
- cloudflare
Transfer-Encoding:
- chunked
X-Robots-Tag:
- none
anthropic-organization-id:
- 617d109c-a187-4902-889d-689223d134aa
anthropic-ratelimit-input-tokens-limit:
- '2000000'
anthropic-ratelimit-input-tokens-remaining:
- '2000000'
anthropic-ratelimit-input-tokens-reset:
- '2025-11-23T13:21:04Z'
anthropic-ratelimit-output-tokens-limit:
- '400000'
anthropic-ratelimit-output-tokens-remaining:
- '400000'
anthropic-ratelimit-output-tokens-reset:
- '2025-11-23T13:21:05Z'
anthropic-ratelimit-tokens-limit:
- '2400000'
anthropic-ratelimit-tokens-remaining:
- '2400000'
anthropic-ratelimit-tokens-reset:
- '2025-11-23T13:21:04Z'
cf-cache-status:
- DYNAMIC
request-id:
- req_011CVQtbM68bFJThHxEH4KeC
retry-after:
- '59'
strict-transport-security:
- max-age=31536000; includeSubDomains; preload
x-envoy-upstream-service-time:
- '3058'
status:
code: 200
message: OK
version: 1
108 changes: 108 additions & 0 deletions ...s/test_structured_outputs/test_anthropic_structured_outputs_with_events_with_content.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
interactions:
- request:
body: '{"max_tokens":1024,"messages":[{"role":"user","content":"Tell me a joke
about OpenTelemetry and rate it from 1 to 10"}],"model":"claude-sonnet-4-5-20250929","output_format":{"type":"json_schema","schema":{"type":"object","properties":{"joke":{"type":"string","description":"A
joke about OpenTelemetry"},"rating":{"type":"integer","description":"Rating
of the joke from 1 to 10"}},"required":["joke","rating"],"additionalProperties":false}}}'
headers:
accept:
- application/json
accept-encoding:
- gzip, deflate
anthropic-beta:
- structured-outputs-2025-11-13
anthropic-version:
- '2023-06-01'
connection:
- keep-alive
content-length:
- '440'
content-type:
- application/json
host:
- api.anthropic.com
user-agent:
- Anthropic/Python 0.74.1
x-stainless-arch:
- arm64
x-stainless-async:
- 'false'
x-stainless-lang:
- python
x-stainless-os:
- MacOS
x-stainless-package-version:
- 0.74.1
x-stainless-read-timeout:
- '600'
x-stainless-retry-count:
- '0'
x-stainless-runtime:
- CPython
x-stainless-runtime-version:
- 3.11.7
x-stainless-timeout:
- '600'
method: POST
uri: https://api.anthropic.com/v1/messages?beta=true
response:
body:
string: !!binary |
H4sIAAAAAAAAA3SRX4vbMBDEv8p2nxVI3AvH6aVw5aBQaF8KV2iKUa2prYu90knrNCb4uxe7Df1H
nxbmN7M7SBceokfPlpvejR6bEkWgm5vNflNtq/32rrpjw8Gz5aG09XZ3++bdM874+PD68f70NrTn
+CBPOzasU8LiQimuBRvOsV8EV0oo6kTZcBNFIcr20+XqV5wXsg7LlwM/xSMObOnAj91EPnjSDuRx
Qh8TMn3JcEcaE30L2tH7BPmAHgM0T6/oHo0bC5bIRJ3zpDHS4GSikpwUil/J6VIhRCEnP3Zn9G4R
ShfSGpJIa9OzUsoxuXbFLw5s6MDZaZB2aXg78/zZcNGY6gxXorBliK91zMI/QcHzCGnAVsa+Nzyu
r2MvHCSNWms8QgrbqqoMN67pUDcZ6736T8f2yjOc/x+7ZpcDSB0GZNfX++Ff/y+66/6ms+E46u/S
zUvDBfkUGtQakNny8qfeZc/z/B0AAP//AwA+knxDRgIAAA==
headers:
CF-RAY:
- 9a30f7bc0cccf169-TLV
Connection:
- keep-alive
Content-Encoding:
- gzip
Content-Type:
- application/json
Date:
- Sun, 23 Nov 2025 13:21:09 GMT
Server:
- cloudflare
Transfer-Encoding:
- chunked
X-Robots-Tag:
- none
anthropic-organization-id:
- 617d109c-a187-4902-889d-689223d134aa
anthropic-ratelimit-input-tokens-limit:
- '2000000'
anthropic-ratelimit-input-tokens-remaining:
- '2000000'
anthropic-ratelimit-input-tokens-reset:
- '2025-11-23T13:21:07Z'
anthropic-ratelimit-output-tokens-limit:
- '400000'
anthropic-ratelimit-output-tokens-remaining:
- '400000'
anthropic-ratelimit-output-tokens-reset:
- '2025-11-23T13:21:09Z'
anthropic-ratelimit-tokens-limit:
- '2400000'
anthropic-ratelimit-tokens-remaining:
- '2400000'
anthropic-ratelimit-tokens-reset:
- '2025-11-23T13:21:07Z'
cf-cache-status:
- DYNAMIC
request-id:
- req_011CVQtbb1HQigBLDM6oAQT3
retry-after:
- '53'
strict-transport-security:
- max-age=31536000; includeSubDomains; preload
x-envoy-upstream-service-time:
- '3261'
Comment on lines +58 to +104
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Nov 23, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion | 🟠 Major

Scrub Anthropic identifiers from cassette response headers

The response headers currently include anthropic-organization-id and request-id with what appear to be real identifiers. Per the repo guidelines for cassettes (“never commit secrets or PII; scrub sensitive data”), these should be replaced with stable placeholders (e.g., org_XXXXXXXX / req_XXXXXXXX) before committing.

🤖 Prompt for AI Agents 
In
packages/opentelemetry-instrumentation-anthropic/tests/cassettes/test_structured_outputs/test_anthropic_structured_outputs_with_events_with_content.yaml
around lines 58 to 104, scrub the sensitive Anthropic identifiers in the
response headers by replacing the real anthropic-organization-id and request-id
values with stable placeholders (for example use org_XXXXXXXX and req_XXXXXXXX
respectively), preserving the YAML structure and quoting style so the cassette
remains valid.

status:
code: 200
message: OK
version: 1
108 changes: 108 additions & 0 deletions ...est_structured_outputs/test_anthropic_structured_outputs_with_events_with_no_content.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,108 @@
interactions:
- request:
body: '{"max_tokens":1024,"messages":[{"role":"user","content":"Tell me a joke
about OpenTelemetry and rate it from 1 to 10"}],"model":"claude-sonnet-4-5-20250929","output_format":{"type":"json_schema","schema":{"type":"object","properties":{"joke":{"type":"string","description":"A
joke about OpenTelemetry"},"rating":{"type":"integer","description":"Rating
of the joke from 1 to 10"}},"required":["joke","rating"],"additionalProperties":false}}}'
headers:
accept:
- application/json
accept-encoding:
- gzip, deflate
anthropic-beta:
- structured-outputs-2025-11-13
anthropic-version:
- '2023-06-01'
connection:
- keep-alive
content-length:
- '440'
content-type:
- application/json
host:
- api.anthropic.com
user-agent:
- Anthropic/Python 0.74.1
x-stainless-arch:
- arm64
x-stainless-async:
- 'false'
x-stainless-lang:
- python
x-stainless-os:
- MacOS
x-stainless-package-version:
- 0.74.1
x-stainless-read-timeout:
- '600'
x-stainless-retry-count:
- '0'
x-stainless-runtime:
- CPython
x-stainless-runtime-version:
- 3.11.7
x-stainless-timeout:
- '600'
method: POST
uri: https://api.anthropic.com/v1/messages?beta=true
response:
body:
string: !!binary |
H4sIAAAAAAAAA3SRQY/TMBCF/8owZ7dqoxZYX1ZCggNIcEFCQFDk2o/EW8cO9qRLqPLfUaqtYEF7
Gmm+92aeZs7cJ4fAmm0wo8OqpBghq91qv6o21X5zU92wYu9Yc1/aZrPdurh7J7/efsaX17s2lcP7
u/T8DSuWacCiQimmBSvOKSwNU4ovYqKwYpuiIArrr+erXvBzIZei+VzzXTqiZk01f+omct6RdCCH
E0IakOmQYY40DnTvpaMPA+JHBPSQPN3SK1gzFiyWiSLg4KikHlQGY7Fer+kwymMT3ZtCJmQYN5Fk
Y48+toQT8kTFxzZczJHS92Wqz5QRjPgUS+eHZzUrqjkb8bFdUr+Yef6muEgamgxTUmTNiK6RMUd+
AAU/RkQL1nEMQfF4uZg+s4/DKI2kI2JhXVWVYmtsh8ZmXHY2jxWbK1/SP8Wu3mUBhg49sgnNvv9f
/4duu3/prDiN8ndr91JxQT55i0Y8Mmte/uxMdjzPvwEAAP//AwBUTNo5WgIAAA==
headers:
CF-RAY:
- 9a30f7d1cfdb7d9e-TLV
Connection:
- keep-alive
Content-Encoding:
- gzip
Content-Type:
- application/json
Date:
- Sun, 23 Nov 2025 13:21:12 GMT
Server:
- cloudflare
Transfer-Encoding:
- chunked
X-Robots-Tag:
- none
anthropic-organization-id:
- 617d109c-a187-4902-889d-689223d134aa
anthropic-ratelimit-input-tokens-limit:
- '2000000'
anthropic-ratelimit-input-tokens-remaining:
- '2000000'
anthropic-ratelimit-input-tokens-reset:
- '2025-11-23T13:21:11Z'
anthropic-ratelimit-output-tokens-limit:
- '400000'
anthropic-ratelimit-output-tokens-remaining:
- '400000'
anthropic-ratelimit-output-tokens-reset:
- '2025-11-23T13:21:12Z'
anthropic-ratelimit-tokens-limit:
- '2400000'
anthropic-ratelimit-tokens-remaining:
- '2400000'
anthropic-ratelimit-tokens-reset:
- '2025-11-23T13:21:11Z'
cf-cache-status:
- DYNAMIC
request-id:
- req_011CVQtbqszdJeNPUoVMEFqc
retry-after:
- '49'
strict-transport-security:
- max-age=31536000; includeSubDomains; preload
x-envoy-upstream-service-time:
- '3238'
status:
code: 200
message: OK
version: 1
Loading