perf(llm): avoid chat stream metrics JSON roundtrip

[jthomson04]

Summary

• Add a typed internal LLMMetricAnnotation fast path for chat-derived streaming responses.
• Attach metrics directly to NvCreateChatCompletionStreamResponse with #[serde(skip)] so OpenAI Chat, Responses, and Anthropic Messages avoid the per-chunk metrics JSON serialize/parse round trip.
• Keep the existing Annotated.event/comment JSON annotation path as a compatibility fallback for legacy/no-data metric frames.
• Preserve client-visible HTTP/SSE wire shape, request-plane protocol, metric names, and public schemas.

Where should the reviewer start?

• lib/llm/src/preprocessor.rs
• lib/llm/src/http/service/metrics.rs
• lib/llm/src/protocols/common/metrics.rs

Related Issues

• Confirmed no related issue.

Validation

• cargo test -p dynamo-llm http::service::metrics
• cargo test -p dynamo-llm http::service::openai
• cargo test -p dynamo-llm protocols::openai::responses
• cargo test -p dynamo-llm anthropic
• Release Python build via maturin develop --uv --release -m lib/bindings/python/Cargo.toml

Benchmark

Responses API c512 on-CPU profile with tokenizer cache enabled:

• Completed 73,512 requests with 0 errors.
• Throughput: 332.52 req/s.
• Targeted CPU profile reduction:
  • LLMMetricAnnotation / from_annotation: prior 8.6-9.8% of frontend on-CPU samples, now 0.0%.
  • dynamo_llm::http::service::metrics inclusive: prior ~7.1-7.8%, now 2.3%.
  • serde_json::read::StrRead: prior ~1.34%, now 0.003%.

The benchmarked change is scoped to removing the chat-derived streaming metrics JSON round trip. Request-plane msgpack transport work is orthogonal and not included in this PR.

---

Summary by CodeRabbit

• Refactor

  • Refactored LLM metrics collection pipeline for improved consistency in handling token counts and latency metrics across streaming response handlers.

• Tests

  • Updated and added test fixtures to verify metrics collection and SSE event processing.

[coderabbitai]

Walkthrough

This PR introduces typed metrics support for LLM streaming responses. A new shared LLMMetricAnnotation type carries token and latency data, chat responses gain an optional llm_metrics field, and HTTP service observation logic is refactored to prefer typed metrics over legacy annotations while maintaining fallback compatibility.

Changes

Metrics Types and Chat Schema

Layer / File(s)	Summary
Metrics type definition and module structure lib/llm/src/protocols/common/metrics.rs	New module defines LLMMetricAnnotation struct with token counts, worker attribution, and latency fields; constant ANNOTATION_LLM_METRICS; and conversion helpers to_annotation() and from_annotation() for typed serialization.
Chat completions schema and module declaration lib/llm/src/protocols/common.rs, lib/llm/src/protocols/openai/chat_completions.rs	Metrics module exposed via public submodule declaration; NvCreateChatCompletionStreamResponse adds optional #[serde(skip)] llm_metrics field to carry typed metrics internally without client serialization.

Metrics Observation and SSE Conversion Pipeline

Layer / File(s)	Summary
Metrics observation pipeline and helpers lib/llm/src/http/service/metrics.rs	Imports protocol metrics types; refactors process_response_and_observe_metrics to delegate to shared observe_llm_metrics; introduces process_chat_response_and_observe_metrics to prefer typed llm_metrics when present, otherwise falling back to annotation frame path; centralizes SSE event creation and metrics observation into shared helpers observe_llm_metrics, observe_annotation_metrics, annotated_to_sse_event.
Metrics observation tests lib/llm/src/http/service/metrics.rs (tests)	Unit tests validate typed chat metrics observation and HTTP queue guard release on first chunk, confirm typed metrics omitted from SSE JSON, and verify test data includes explicit llm_metrics: None.

Preprocessor Integration and Type Migration

Layer / File(s)	Summary
Preprocessor type migration and helpers lib/llm/src/preprocessor.rs	Removes local metric type definitions, re-exports ANNOTATION_LLM_METRICS and LLMMetricAnnotation from shared metrics module, introduces attach_llm_metrics and attach_metrics_annotation helpers for conditional metric serialization, updates imports to use qualified std::time::Duration.
Preprocessor streaming and usage-chunk metric attachment lib/llm/src/preprocessor.rs	Streaming response postprocessing and final usage-chunk construction replaced inline to_annotation().unwrap_or_else() logic with calls to attach_llm_metrics, which preserves existing event values and only sets annotation fields on successful serialization.

HTTP Service Wiring Update

Layer / File(s)	Summary
Anthropic HTTP service metrics wiring lib/llm/src/http/service/anthropic.rs	Imports and invokes process_chat_response_and_observe_metrics instead of generic process_response_and_observe_metrics in streaming and non-streaming response observation paths.
OpenAI HTTP service metrics wiring lib/llm/src/http/service/openai.rs	Chat-completions and responses endpoints updated to use chat-specific process_chat_response_and_observe_metrics and process_chat_response_using_event_converter_and_observe_metrics functions for both streaming SSE and non-streaming aggregation paths.

Test Fixture Updates

Layer / File(s)	Summary
Protocol module test fixture updates lib/llm/src/protocols/openai/chat_completions/*, lib/llm/src/protocols/openai/responses/*, lib/llm/src/protocols/anthropic/*, lib/llm/src/audit/*	Protocol-level test helpers and struct initializers updated to set llm_metrics: None in response constructors across chat_completions delta/jail/aggregator, responses, and anthropic modules.
Test module and integration fixture updates lib/llm/tests/*, lib/llm/src/agents/*, lib/llm/src/perf/*	Test fixtures across aggregators, logprob analysis, postprocessor parsing, jail, reasoning parser, streaming tool parsers, tool choice, and agent trace integration updated to include llm_metrics: None; Python bindings also updated.

---

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'perf(llm): avoid chat stream metrics JSON roundtrip' directly summarizes the main change: optimizing LLM metrics handling to eliminate JSON serialization/deserialization.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description check	✅ Passed	The pull request description is comprehensive and well-structured, covering all key sections required by the template.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[jthomson04]

/ok to test af8846a