Forward-merge release/0.1 into main - #125
Merged
Merged
Conversation
#### Overview Implements streaming backed by NVIDIA NeMo Relay for NVIDIA NeMo Fabric runtimes, promoting the streaming proof of concept from #104 into the supported Python SDK surface. `Runtime.invoke_stream()` exposes raw ATOF records while an invocation is running and keeps the normalized terminal `RunResult` out of band. Streaming is **opt-in**: callers pass `streaming=True` to `start_runtime(...)` (with NeMo Relay enabled) to provision it, so NeMo Relay-enabled runtimes that do not stream pay no additional cost and keep their existing telemetry shape. The streaming API change is additive — a new optional `streaming` parameter — and introduces no new dependencies. #### Details - Add an SDK-owned loopback HTTP listener for NeMo Relay NDJSON/ATOF delivery with a bounded, byte-budgeted record queue (16 MiB total / 1 MiB per record), TCP backpressure, and a bounded post-turn drain window. - Gate provisioning behind `start_runtime(..., streaming=True)`: a NeMo Relay-enabled runtime binds the loopback listener and injects the `nemo-fabric-stream` ATOF sink only when streaming is requested. Requesting `streaming=True` without NeMo Relay enabled raises `FabricConfigError`. - Add `Runtime.invoke_stream()`, `Runtime.supports_streaming`, `InvokeStream.result()`, and explicit early-exit cleanup via `InvokeStream.aclose()`. - Configure the listener host through an environment variable so deployments can override the loopback default when needed. - Correlate records to a single invocation by ATOF scope tree: Deep Agents and Hermes stamp a request ID on the turn's root scope; gateway harnesses match on NeMo Relay's turn role and index. The Hermes correlation scope is added only for streaming runs, so non-streaming NeMo Relay runs keep their prior trajectory shape. - Surface otherwise-silent failures as `RuntimeWarning`s: no NeMo Relay connection reached the listener, data arrived but no record matched the active turn, or a chunked upload was interrupted mid-turn. A clean stream completion does not warn. - Validate streaming requests before returning an `InvokeStream`, make stream iteration cancellation-safe, and allow a manual `Runtime.stop()` to finalize a completed stream after `result()`. - Document the public streaming contract, terminal failure semantics, adapter behavior, runnable examples, and the generated Python API reference. #### Validation - Focused streaming, adapter, generated-reference, and Fern-sync tests: 135 passed. - `uv run --no-sync pre-commit run --all-files --show-diff-on-failure` — all hooks passed, including Cargo/uv lockfile freshness, attribution generation, and the dependency license diff. - `just docs` — passed; Fern reported only the expected unauthenticated redirect check warning. - `just test-python` — 455 passed and 44 skipped locally. Two failures and two setup errors remain in `test_installed_adapter_discovery.py` because uv's local macOS Python 3.13 virtual environments cannot load `libpython3.13.dylib`; the streaming-focused tests pass independently. - End-to-end streaming turns run under `warnings.simplefilter("error", RuntimeWarning)` and assert non-empty records: - Deep Agents — `tests/e2e/test_deepagents.py` (persistent host, mock model). - Claude and Codex gateways — live tests in `tests/e2e/test_claude.py` and `tests/e2e/test_codex.py`, validating gateway turn correlation. #### Where should the reviewer start? Start with `python/src/nemo_fabric/streaming.py` for the listener, turn correlation, and stream lifecycle; then `python/src/nemo_fabric/client.py` for the `streaming=True` opt-in gate and `python/src/nemo_fabric/runtime.py` for eager validation and manual shutdown. The focused behavioral coverage is in `tests/python/test_streaming.py`, with end-to-end coverage under `tests/e2e/`. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Supersedes the implementation POC in #104 - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. ## Summary by CodeRabbit * **New Features** * Added NeMo Relay-backed streaming to the Python SDK via `Runtime.invoke_stream()` with an `InvokeStream` async iterator of raw ATOF records and a terminal `await stream.result()`. * Added `Runtime.supports_streaming` and streaming-aware runtime startup/stop lifecycle handling. * Introduced typed v2 NeMo Relay ATOF sink configuration via file/stream sink models. * **Documentation** * Added the Streaming API reference and updated Python and integration guides with end-to-end streaming workflows. * Updated NeMo Relay ATOF examples to use the typed v2 `sinks` model. * **Tests** * Expanded/updated streaming, documentation contract, and adapter configuration tests to match the new streaming and typed sink behavior. Authors: - Yuchen Zhang (https://github.com/yczhang-nv) Approvers: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) - Ajay Thorve (https://github.com/AjayThorve) URL: #109
Author
|
SUCCESS - forward-merge complete. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Forward-merge triggered by push to release/0.1 that creates a PR to keep main up-to-date. If this PR is unable to be immediately merged due to conflicts, it will remain open for the team to manually merge. See forward-merger docs for more info.