feat: add OpenAI Agents SDK support to OpenInference mapper - #366
feat: add OpenAI Agents SDK support to OpenInference mapper#366liramon2 wants to merge 1 commit into
Conversation
|
@strandly-the-agent Review this PR. Consolidate findings into a single comment. |
|
Reviewed at Verdict: changes-requested. The producer works on the two shipped fixtures, but
I patched every finding below locally and re-ran: 385 + 235 still pass, so these are cheap fixes rather than a redesign. 🔴 1 —
|
| # | Manifestation | Who hits it | Result |
|---|---|---|---|
| a | Multi-turn chat — index 1 is the oldest user message | any agent using result.to_input_list() or Session/SQLiteSession (the documented multi-turn pattern) |
user_prompt='What is 15 multiplied by 37?' paired with agent_response='555 divided by 5 is 111.' — a judge scores the answer against the wrong question |
| b | Reasoning models — summary sits at output index 0 | gpt-5/o-series with reasoning.summary="auto" on instrumentor v2.0.0 |
agent_response='The user wants 15*37. Multiply: 15*37=555.' — the scratch-pad is scored as the answer. With encrypted reasoning and no summary, the AGENT span is dropped entirely |
| c | Multimodal / structured content — instrumentor emits contents.0.message_content.text and no scalar .content |
the documented way to send images/files | AgentInvocationSpan count = 0 — span silently dropped. Note _normalize_smolagents_llm_attrs (:176) already has this singular→plural fallback |
| d | Chat-completions path numbers input from 0 | LiteLLM / Azure / any OpenAI-compatible endpoint via the SDK | index 1 is the second message |
| e | Empty-string or tool-call-only final turn — guards are if user_prompt: / if agent_response:, then hunk E requires output.value |
tool_use_behavior="stop_on_first_tool", a max-turns cut-off, a guardrail trip |
AgentInvocationSpan count = 0 — exactly the runs an eval most wants to see |
All five are silent — no warning, and the eval result comes back plausible.
This fix resolves a–d (verified; e needs a policy call from you, not a path fix — see the Questions):
_MSG_IDX_RE = re.compile(r"^llm\.(?:in|out)put_messages\.(\d+)\.")
@classmethod
def _message_indices(cls, attrs: dict, prefix: str) -> list[int]:
"""Descending message indices actually present under prefix (no contiguity assumed)."""
out = set()
for key in attrs:
if key.startswith(prefix) and (m := cls._MSG_IDX_RE.match(key)):
out.add(int(m.group(1)))
return sorted(out, reverse=True)
@classmethod
def _last_user_message(cls, attrs: dict) -> str | None:
"""Text of the highest-indexed user input message, in either content shape."""
for idx in cls._message_indices(attrs, "llm.input_messages."):
base = f"llm.input_messages.{idx}.message"
if attrs.get(f"{base}.role") != "user":
continue
if text := attrs.get(f"{base}.content") or attrs.get(f"{base}.contents.0.message_content.text"):
return text
return None
@classmethod
def _last_assistant_text(cls, attrs: dict) -> str | None:
"""Text of the highest-indexed non-reasoning assistant output message."""
for idx in cls._message_indices(attrs, "llm.output_messages."):
base = f"llm.output_messages.{idx}.message"
if attrs.get(f"{base}.contents.0.message_content.type") == "reasoning":
continue
if text := attrs.get(f"{base}.contents.0.message_content.text") or attrs.get(f"{base}.content"):
return text
return Nonethen user_prompt = self._last_user_message(first_attrs) and agent_response = self._last_assistant_text(last_attrs). Deriving indices from the keys also removes the contiguity assumption in the llm.tools.{idx} copy loop at :313-316.
🔴 2 — multi-agent handoff: the coordinator disappears and its handoff call is misattributed to the agent that received it
Run on your own openai_agents_openinference_adot_spans.json:
raw AGENT spans: 3 -> ['coordinator', 'Agent workflow', 'math_specialist']
surviving AgentInvocationSpans: 1
span_id=2cbdd8e10afa264f
user_prompt="What is 42 * 7, and what's the weather in Seattle?"
available_tools=['add_numbers', 'divide_numbers', 'multiply_numbers']
TOOL 'handoff to math_specialist' parent=None agent_span_id=2cbdd8e10afa264f
TOOL 'multiply_numbers' parent=2cbdd8e10afa264f agent_span_id=2cbdd8e10afa264f
"Agent workflow"is dropped: the walk (:289-299) recurses only intoCHAINchildren and stops at nestedAGENTchildren, so it collects no LLM spans and never getsinput.value.coordinatoris dropped: its LLM span emits onlytool_calls(the handoff decision) and no text, sooutput.valueis never set and hunk E (:452-458) rejects it. Its declaredtransfer_to_math_specialist/transfer_to_research_specialistschemas are lost with it.math_specialistsurvives carrying the original top-level question, so the trace reads as one agent that was asked the whole thing and answered it. The delegation is invisible.- The handoff TOOL span gets
parent=Nonefrombridge_parent_gaps(its real ancestor was dropped), and thenTrace.model_post_init's orphan fallback (types/trace.py:202-206) assignsagent_span_id = math_specialist— attributing the coordinator's delegation to the agent that received it.
Concretely, a ToolSelectionAccuracy judge is asked to justify handoff to math_specialist({}) while the "Available tool-calls" list it was shown is add_numbers/divide_numbers/multiply_numbers — the tool under evaluation isn't in the candidate set it was given.
For contrast, the Claude Agent SDK producer represents delegation legibly, as Agent({'description': …, 'prompt': …}) with the subagent's real answer as tool_result.
The misattribution is wrong regardless of intent, so it needs fixing either way; whether the coordinator should survive is a design call I've put in the Questions. Cheapest correct step: don't let an orphaned tool span inherit agent_span_id from an unrelated agent — leave it None and let the extractors treat it as unowned.
Worth noting test_math_specialist_agent_span currently asserts only "at least one agent span" matching 42 or weather, so it encodes this collapse as expected. There's no agent-count assertion for this fixture (the live one has test_one_agent_span_per_trace).
🟡 3 — the normalizer runs outside every try/except, so one bad span can lose the whole session (including other producers' traces)
map_to_session:137-140 calls the normalizer before any span reaches _build_trace's per-span try/except Exception + logger.warning (:331, :344-345). Reproduced — a healthy LangChain trace batched with one awkward openai_agents trace:
healthy LangChain trace alone -> ['t-langchain']
TypeError ESCAPED map_to_session: '<' not supported between instances of 'int' and 'str'
-> 0 traces returned: the healthy LangChain trace is destroyed too
attributes=None variant: AttributeError ESCAPED map_to_session: 'NoneType' object has no attribute 'get'
Triggers: llm_spans.sort(key=lambda s: s.get("start_time", 0)) (:303) on heterogeneous/None start_time, and attributes: None (:282/:293).
Honest reachability, which is why this is 🟡 and not 🔴: I could not reach it through your own CloudWatchLogsParser — it drops records lacking startTimeUnixNano, so values come out uniform. But SessionMapper.parse_timestamp (session_mapper.py:77) explicitly documents start_time as possibly str/datetime/None, so the class's declared input contract permits the crashing input — and the correct idiom already exists next door in adk_otel_session_mapper.py:194. Every other per-span quirk here degrades to a warning; this one takes unrelated producers down with it.
llm_spans.sort(key=lambda s: self.parse_timestamp(s.get("start_time")))plus wrapping the call:
try:
self._normalize_openai_agents_trace(trace_spans)
except Exception as e:
logger.warning("Failed to normalize OpenAI Agents trace: %s", e)With both applied the batch above returns ['t-langchain'] and logs a warning.
🟡 4 — test gaps: 10 of 22 mutants survive all 385 tests, and hunk D is completely unpinned
Deleting the bridge_parent_gaps call at :349 outright leaves 123 passed in the mapper file and 235 passed in extractors/types/detectors. The correct behaviour it provides (3 spans re-parented onto math_specialist) is real and worth pinning.
Other survivors: input taken from the last LLM descendant instead of the first · the entire llm.tools.* copy made a no-op · the .message.content output fallback deleted · hunk F's _trace_tools_map fallback deleted · the and "name" not in tool_info guard dropped · the or tool_info.get("parameters") fallback dropped · the added ValueError removed · the attrs.get("input.value") skip guard removed · the if user_prompt: guard removed.
Also: hunk F is pinned by exactly one test (test_openai_agent_span_conversion), which hand-builds an AGENT span carrying its own llm.tools.0 — a shape no real producer emits. 5 of the 10 TestOpenAIAgentsScopeSupport tests only assert _is_*_span(...) is True/False with no mapping exercised. And test_handoff_tool_span_present asserts len(tool_spans) >= 1 without filtering by name, so it passes even if the handoff span specifically is dropped.
Two that pass today and close the D / agent-count gaps:
def test_math_specialist_children_reparented(self, openai_agents_adot_session):
all_spans = [s for t in openai_agents_adot_session.traces for s in t.spans]
specialist = next(s for s in all_spans if isinstance(s, AgentInvocationSpan) and "42" in s.user_prompt)
multiply = next(s for s in all_spans if isinstance(s, ToolExecutionSpan) and s.tool_call.name == "multiply_numbers")
inferences = [
s for s in all_spans
if isinstance(s, InferenceSpan) and s.span_info.parent_span_id == specialist.span_info.span_id
]
assert multiply.span_info.parent_span_id == specialist.span_info.span_id
assert len(inferences) == 2
def test_adot_agent_span_count_is_one(self, openai_agents_adot_session):
for trace in openai_agents_adot_session.traces:
assert len([s for s in trace.spans if isinstance(s, AgentInvocationSpan)]) == 1Plus one that fails today and pins the appendix-1 regression: a make_span AGENT with "llm.tools.0.tool.json_schema": "42" should still yield 1 agent span with available_tools == [].
Questions
- Is the coordinator's disappearance the intended representation of a handoff? An argument exists for "the agent that produced the final answer is the trace" — but then
math_specialistinheriting the top-level question and owning the handoff call reads as one agent doing everything, and the coordinator'stransfer_to_*tools vanish. Would you rather a tool-call-only turn counted as a real agent invocation? (blocking — it decides whether 🔴 2 is a bug or a documented limitation, and whether manifestation (e) above needs the same treatment.) - feat(mappers): add OpenAI Agents OTel session mapper #365 vs feat: add OpenAI Agents SDK support to OpenInference mapper #366 — two techniques for one problem. feat(mappers): add OpenAI Agents OTel session mapper #365 attaches tools directly to the converted
AgentInvocationSpan(openai_agents_otel_session_mapper.py:61-68), whereas here hunk C fabricatesllm.tools.*onto the raw AGENT span (:313-316) so hunk F (:701-706) can read it back. Theinput.value/output.valuehalf of C is load-bearing for the:452-458gate and clearly earns its place, but would the tool half be simpler as the feat(mappers): add OpenAI Agents OTel session mapper #365 approach? Both PRs also make the same parent-span-id fix in parallel — is one meant to land first? (non-blocking) - Following the feat: add support for Claude agents to OpenInference mapper #340 → feat: add Claude Agents OpenInference integration tests #353 precedent, is there a tracking issue for
tests_integ/test_openai_agents_*? I couldn't find one. (non-blocking)
Appendix — non-blocking (7)
- ⚪
:889if "function" in tool_info and …assumes a dict. Allm.tools.N.tool.json_schemadecoding to a JSON scalar raisesTypeError, which isn't in theexcept (json.JSONDecodeError, AttributeError, ValueError)at:898; it escapes and_build_trace's broadexcept Exceptiondrops the whole span, where main skipped just that one tool. Strictly contrived — I checked openai_agents v1.6.1/v2.0.0, vendored langchain (which gatesLLM_TOOLSto LLM runs) and claude_agent_sdk (emits nollm.tools.*at all): no real instrumentor can emit it. But it's a regression vs main and the fix is one word:isinstance(tool_info, dict) and …. - ⚪ The descendant walk at
:289-299has novisitedset, unlikebridge_parent_gaps(utils.py:298-301). A duplicatespan_idclosing a cycle hangs the process with unboundedllm_spansgrowth (confirmed, >5s with no termination); self-parented and diamond variants terminate fine. Contrived, one-line fix. - ⚪ Hunk C copies tool schemas from the earliest LLM span only, so a tool list that changes across turns loses entries (crafted case: 1 tool instead of 3). Reachable via
FunctionTool.is_enabledper-turn filtering or MCP withcache_tools_list=False. Uncommon. - ⚪ The normalizer never checks scope —
map_to_session:139gates on "any span in the group is openai_agents" but hands the walk the whole group. Sibling spans are safe (I checked: a smolagents AGENT span sharing atrace_idis not mutated), so this needs a non-openai AGENT span to be an ancestor of openaiCHAIN/LLMspans. Contrived, but a scope check in the loop would make the intent explicit. - ⚪ Hunk F is safe for the 3 existing producers by upstream construction, not by a guard here — worth a comment saying so, since nothing in the suite would catch a future instrumentor that starts attaching
llm.tools.*to AGENT spans. - ⚪ Fixture provenance skew: the live fixture is instrumentor v1.6.1, the ADOT one v2.0.0. Worth a word on whether that's deliberate.
- ⚪ Correcting myself on a couple of things a reviewer might otherwise raise, in case they come up: the in-place mutation of caller span dicts is pre-existing in this class since fix: route smolagents OpenInference spans to OpenInferenceSessionMapper #308 (
main:191,:204,:256,:259) and is observably idempotent, so it isn't this PR's problem; and the_adot_fixture is legitimately ADOT-captured (aws.genai.*,PlatformType: AWS::BedrockAgentCore) — the existingclaude_adotfixture has the same flat shape, so "it doesn't exercise_parse_adot_output" isn't a fault of this PR either.
Three pre-existing items are out of scope here but might deserve their own issues — happy to file them if useful: the orphan-fallback in types/trace.py:202-206 silently attributing any unowned tool span to the sole surviving agent (affects every mapper); the is_langchain dedup exemption at :351-357 having had no test since #340; and no mapper documenting the AgentInvocationSpan-per-Trace expectation that trace_extractor and root_cause_analyzer each interpret differently.
Automated multi-pass review by strandly-the-agent. Every 🔴/🟡 above has a repro I ran at bc6422d, and I applied all the suggested fixes locally to confirm they work and keep 385 + 235 green — but I'm an experimental agent, so please treat this as work for a human to approve rather than a verdict. Happy to dig into any of it further.
|
@strandly-the-agent Here are the instrumentation:
Here are the current findings: For OpenAI agents SDK + traceloop instrumentation:
For OpenAI agents SDK + OpenInference instrumentation:
NOTE: In the current mappers' implementation strategy, I don't think we should use composition, and inheritance. And the mappers are grouped by instrumentation frameworks. I need you to:
|
|
Verified all seven claims against the instrumentation sources (
Design answer (Q2/Q3): agree — no inheritance, and no new classes. Traceloop verification (T1–T4)T1 — GenAI conventions ✅. Fixture chat spans carry T2 —
T3 — ✅ confirmed in both fixtures. Root T4 — ✅ confirmed. In the ADOT fixture,
OpenInference verification (O1–O3)O2 — mechanism ✅, paths and indices need precision. Verified against the v2.0.0 instrumentor source and both #366 fixtures:
O3 — ✅ verified, with one important limit.
Design recommendation (Q2/Q3): no new classes, no inheritance — three named phases inside the two family mappersThe rule that matches the repo you already have: one mapper per emitting-convention family ( Why I'd agree about inheritance specifically: #365's subclass works by overriding The shape I'd standardize (both mappers already approximate it):
Three guardrails, all from bugs already found in these two PRs: each hook is exception-isolated (one producer's quirk must degrade to a warned-and-skipped trace, not kill the session — the #366 normalizer currently runs outside every try/except); each hook checks scope on the spans it touches (the #366 walk doesn't); and each hook gets producer fixtures that pin it (mutation testing showed the Applied to the two PRs:
One thing I'd not do: don't introduce a producer-level abstraction (base class or protocol for "producers") until a third family needs it. Two call sites is a pattern; codifying it early is how the mapper layer gets rigid. Verification by |
Description
Add OpenAI agents support to OpenInference mapper.
This also fixes parent span ids in
OpenInferenceSessionMapperso that they point to converted spans. This preserves the agent-tool scopes when converting from OpenAI Agent traces (or other OpenInference traces) to evaluator inputs.Related Issues
#364
Documentation PR
Type of Change
New feature
Testing
How have you tested the change? Verify that the changes do not break functionality or introduce new warnings.
hatch run prepareChecklist
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.