feat(nemo-agents): add support for streaming via Fabric backed agents - #973
Merged
mmogallapalli merged 11 commits intoJul 29, 2026
Conversation
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
mmogallapalli
marked this pull request as ready for review
July 29, 2026 19:50
mmogallapalli
changed the title
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (4)
🚧 Files skipped from review as they are similar to previous changes (4)
📝 WalkthroughWalkthroughAdds Claude harness configuration and upgrades Fabric dependencies. Implements streaming Fabric runtime and session handling, converts Fabric records to OpenAI-compatible SSE, and enables streamed ChangesFabric streaming integration
Sequence Diagram(s)sequenceDiagram
participant Client
participant FabricServer
participant FabricSessionManager
participant FabricRuntimeStream
Client->>FabricServer: POST /v1/chat/completions with stream=true
FabricServer->>FabricSessionManager: Open stream_session
FabricSessionManager->>FabricRuntimeStream: Start streaming invocation
FabricRuntimeStream-->>FabricServer: Fabric records and terminal result
FabricServer-->>Client: OpenAI-compatible SSE deltas and [DONE]
Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
plugins/nemo-agents/examples/nemo-agent-config/README.md (1)
37-44: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a tested Python SDK alternative in a tab set.
This Claude flow only documents CLI usage. Documentation standards require paired Python SDK and CLI examples.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/examples/nemo-agent-config/README.md` around lines 37 - 44, Add a tested Python SDK alternative alongside the existing Claude CLI instructions in the README, using a tab set to present both flows. Include the SDK authentication/setup and equivalent invocation using the existing agent configuration, and ensure the Python example is executable and covered by the relevant documentation tests.Source: Coding guidelines
plugins/nemo-agents/tests/unit/test_fabric_runtime.py (1)
92-92: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse concrete annotations.
Remove the quotes around
_FakeInvokeStream. As per coding guidelines, “prefer concrete type hints over string-based type hints.”- def invoke_stream(self, *, request: Any) -> "_FakeInvokeStream": + def invoke_stream(self, *, request: Any) -> _FakeInvokeStream: ... - def __aiter__(self) -> "_FakeInvokeStream": + def __aiter__(self) -> _FakeInvokeStream:Also applies to: 115-116
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/tests/unit/test_fabric_runtime.py` at line 92, Update the invoke_stream method’s return annotation and the corresponding annotations at the additionally referenced lines to use the concrete _FakeInvokeStream type directly, removing string-based quoting while preserving the existing behavior.Source: Coding guidelines
plugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.py (1)
15-26: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse concrete, normally imported annotation types throughout.
plugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.py#L15-L26: import runtime annotation types normally instead of underTYPE_CHECKING.plugins/nemo-agents/tests/unit/test_fabric_runtime.py#L92-L115: remove quotes from_FakeInvokeStreamannotations.plugins/nemo-agents/tests/unit/test_fabric_session_manager.py#L34-L36: remove quotes from_FakeFabricConfig.As per coding guidelines, “prefer concrete type hints over string-based type hints” and do not import them only under
TYPE_CHECKING.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@plugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.py` around lines 15 - 26, Use concrete runtime annotation types: normally import FabricRuntimeResult and FabricRuntimeStream in streaming.py instead of guarding them with TYPE_CHECKING; remove string quotes from _FakeInvokeStream annotations in plugins/nemo-agents/tests/unit/test_fabric_runtime.py:92-115 and from _FakeFabricConfig in plugins/nemo-agents/tests/unit/test_fabric_session_manager.py:34-36.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@plugins/nemo-agents/pyproject.toml`:
- Line 66: Update the nemo-fabric dependency in the project dependency list to
include the relay extra, preserving the existing version bounds: use
nemo-fabric[relay]>=0.1.0rc4,<0.2.0 so the fabric installation includes
streaming-path Relay dependencies.
In `@plugins/nemo-agents/src/nemo_agents_plugin/fabric/server.py`:
- Around line 244-269: The streaming response path in stream_session must
guarantee cleanup of the entered stream context even if response iteration is
cancelled or never begins. Update the response lifecycle around
_iter_streaming_chat_completion so stream_context.__aexit__ is invoked after
__aenter__ succeeds, and add a regression test at
plugins/nemo-agents/tests/unit/test_fabric_server.py:509-524 covering
cancellation or closure before the first event; no direct change is required
elsewhere.
In `@plugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.py`:
- Around line 156-165: Update the recognized-event branch and OpenAI
choice-delta handling to reject explicit roles not in _ASSISTANT_ROLES before
extracting or emitting text. Preserve current behavior when role is absent or
assistant-compatible, and apply the same guard to choice deltas.
In `@plugins/nemo-agents/tests/unit/test_fabric_server.py`:
- Around line 509-524: Extend
test_streaming_chat_completion_closes_stream_on_generator_close with a separate
case that closes or cancels the events iterator immediately after creation,
before calling __anext__ or receiving any SSE payload. Keep the already-entered
stream_context and assert fabric_stream.aclose_calls and
stream_context.exit_calls are each 1.
---
Nitpick comments:
In `@plugins/nemo-agents/examples/nemo-agent-config/README.md`:
- Around line 37-44: Add a tested Python SDK alternative alongside the existing
Claude CLI instructions in the README, using a tab set to present both flows.
Include the SDK authentication/setup and equivalent invocation using the
existing agent configuration, and ensure the Python example is executable and
covered by the relevant documentation tests.
In `@plugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.py`:
- Around line 15-26: Use concrete runtime annotation types: normally import
FabricRuntimeResult and FabricRuntimeStream in streaming.py instead of guarding
them with TYPE_CHECKING; remove string quotes from _FakeInvokeStream annotations
in plugins/nemo-agents/tests/unit/test_fabric_runtime.py:92-115 and from
_FakeFabricConfig in
plugins/nemo-agents/tests/unit/test_fabric_session_manager.py:34-36.
In `@plugins/nemo-agents/tests/unit/test_fabric_runtime.py`:
- Line 92: Update the invoke_stream method’s return annotation and the
corresponding annotations at the additionally referenced lines to use the
concrete _FakeInvokeStream type directly, removing string-based quoting while
preserving the existing behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: a66bdf29-5b51-43c7-9358-572080599d85
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (14)
plugins/nemo-agents/examples/nemo-agent-config/README.mdplugins/nemo-agents/examples/nemo-agent-config/agent.yamlplugins/nemo-agents/pyproject.tomlplugins/nemo-agents/src/nemo_agents_plugin/fabric/runtime.pyplugins/nemo-agents/src/nemo_agents_plugin/fabric/server.pyplugins/nemo-agents/src/nemo_agents_plugin/fabric/serving_models.pyplugins/nemo-agents/src/nemo_agents_plugin/fabric/session_manager.pyplugins/nemo-agents/src/nemo_agents_plugin/fabric/streaming.pyplugins/nemo-agents/tests/unit/test_fabric_runtime.pyplugins/nemo-agents/tests/unit/test_fabric_server.pyplugins/nemo-agents/tests/unit/test_fabric_serving_models.pyplugins/nemo-agents/tests/unit/test_fabric_session_manager.pyplugins/nemo-agents/tests/unit/test_fabric_streaming.pyplugins/nemo-agents/tests/unit/test_fabric_translator.py
Contributor
|
Signed-off-by: Manjesh Mogallapalli <mmogallapall@nvidia.com>
mmogallapalli
force-pushed
the
mmogallapall/aircore-952-support-streaming-chat-completions-for-fabric-backed-agent
branch
from
July 29, 2026 20:53
6638717 to
03e6082
Compare
mikeknep
approved these changes
Jul 29, 2026
mmogallapalli
deleted the
mmogallapall/aircore-952-support-streaming-chat-completions-for-fabric-backed-agent
branch
July 29, 2026 22:20
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds streaming chat-completion support for Fabric-backed NeMo Agents as
part of AIRCORE-952.
The main flow added here is:
This keeps Fabric/Relay as the source of streaming telemetry while Platform owns
the OpenAI-compatible serving shape expected by existing clients.
Changes
runtime.invoke_stream(...).stream=truehandling into the Fabric/v1/chat/completionsserving path.aclose().agent.yaml.Adapter Validation
Validated Studio/deployed streaming for:
Design Notes
Validation
Passed:
Result:
Also passed:
Working in Studio:
Summary by CodeRabbit
[DONE], and structured mid-stream error events.