Repository navigation
docs: document LLM request and tool execution outcomes #341
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
rapids-bot
merged 4 commits into
NVIDIA:main
from
bbednarski9:docs/llm-intercept-pending-marks
Jul 2, 2026
Merged
Changes from 1 commit
Commits
Show all changes
4 commits
Select commit
Hold shift + click to select a range
279e53e
docs: document LLM request intercept outcomes
bbednarski9 93f37b1
docs: document tool execution intercept outcomes
bbednarski9 24ef58f
docs: clarify tool outcome reference
bbednarski9 eafb1c6
Merge branch 'main' into docs/llm-intercept-pending-marks
bbednarski9 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| --- | ||
| title: "LLM Request Intercept Outcomes" | ||
| description: "Canonical request-intercept result and managed lifecycle behavior." | ||
| --- | ||
| {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| SPDX-License-Identifier: Apache-2.0 */} | ||
|
|
||
| Every LLM request intercept returns one canonical outcome: | ||
|
|
||
| ```json | ||
| { | ||
| "request": {"headers": {}, "content": {}}, | ||
| "annotated_request": null, | ||
| "pending_marks": [] | ||
| } | ||
| ``` | ||
|
|
||
| `request` is required. `annotated_request` defaults to `null` when omitted on | ||
| input, and `pending_marks` defaults to an empty list. Canonical serialization | ||
| includes all three fields. A pending mark contains only `name`, optional | ||
|
bbednarski9 marked this conversation as resolved.
Resolved
|
||
| `category` and `category_profile`, and optional `data` and `metadata`. Relay | ||
| owns event UUIDs, parent UUIDs, and timestamps. | ||
|
|
||
| ## Request Authority | ||
|
|
||
| The provider-body source of truth depends only on whether a request codec is | ||
|
bbednarski9 marked this conversation as resolved.
Resolved
|
||
| active: | ||
|
|
||
| | Request codec | Provider body source | Header source | | ||
| | --- | --- | --- | | ||
| | No codec | `outcome.request.content` | `outcome.request.headers` | | ||
| | Active codec | `outcome.annotated_request` | `outcome.request.headers` | | ||
|
|
||
| With an active codec, `request.content` is read-only context. Every intercept | ||
| must return an annotation and make provider-body changes through that | ||
| annotation, including its flattened `extra` fields for provider-specific data. | ||
| Relay rejects a changed raw body or missing annotation at the offending | ||
| intercept before invoking later middleware or creating an LLM lifecycle. | ||
|
|
||
|
lvojtku marked this conversation as resolved.
Resolved
|
||
| ```mermaid | ||
| flowchart TD | ||
| INPUT["Original LlmRequest"] --> CODEC{"Request codec active?"} | ||
|
|
||
| CODEC -->|No| RAWCHAIN["Run intercept chain"] | ||
| RAWCHAIN --> RAWPROVIDER["Provider receives outcome.request"] | ||
|
|
||
| CODEC -->|Yes| DECODE["Decode content into annotated_request"] | ||
| DECODE --> INTERCEPT["Invoke next intercept"] | ||
| INTERCEPT --> CHECKANN{"Annotation returned?"} | ||
| CHECKANN -->|No| FAIL["Fail before lifecycle"] | ||
| CHECKANN -->|Yes| CHECKRAW{"request.content unchanged?"} | ||
| CHECKRAW -->|No| FAIL | ||
| CHECKRAW -->|Yes| MORE{"More intercepts?"} | ||
| MORE -->|Yes| INTERCEPT | ||
| MORE -->|No| ENCODE["Encode final annotated_request"] | ||
| ENCODE --> HEADERS["Apply final request.headers"] | ||
| HEADERS --> PROVIDER["Provider receives one resolved LlmRequest"] | ||
| ``` | ||
|
|
||
| Python callbacks return `LLMRequestInterceptOutcome`; Rust callbacks return | ||
| `LlmRequestInterceptOutcome`; Go callbacks return | ||
| `LLMRequestInterceptOutcome`; and Node.js and WebAssembly callbacks return | ||
| `{ request, annotated?, pendingMarks? }`, with `categoryProfile` on each | ||
| JavaScript pending-mark DTO. The canonical JSON forms retain `pending_marks` | ||
| and `category_profile`. Public C callbacks write one owned canonical outcome | ||
| JSON string. Native ABI v1 uses one host-owned outcome JSON string. Rust and | ||
| Python `grpc-v1` worker SDKs return their canonical outcome type in a | ||
| `JsonEnvelope` whose schema is | ||
| `nemo.relay.LlmRequestInterceptOutcome@1`. | ||
|
bbednarski9 marked this conversation as resolved.
Resolved
|
||
|
|
||
| The standalone request-intercept helper returns the complete outcome but does | ||
| not emit its pending marks because it does not own an LLM lifecycle. | ||
|
|
||
| ## Managed Lifecycle | ||
|
|
||
| Managed execution runs all effective global and scope-local intercepts before | ||
| creating the LLM handle. Each accepted request/annotation pair feeds the next | ||
|
bbednarski9 marked this conversation as resolved.
Resolved
|
||
| intercept under the authority rules above, while pending marks append in | ||
| middleware order. A breaking | ||
| intercept's marks are retained. If any intercept fails or its boundary result | ||
| is malformed, Relay discards all accumulated marks and creates no LLM | ||
| lifecycle. | ||
|
|
||
| After successful interception, Relay creates the handle and captures one | ||
| subscriber snapshot. It emits the LLM start at `T`, every pending mark at | ||
| `T + 1µs` in returned order with the LLM UUID as parent, and the LLM end no | ||
| earlier than `T + 1µs`. Streaming and non-streaming calls use the same rules. | ||
| Pending marks are never added to the provider request, annotated request, | ||
| codec input, sanitizer input, or start payload. | ||
|
|
||
| ## Migration | ||
|
|
||
| This finalizes unpublished native ABI v1 and `grpc-v1` contracts. Rebuild all | ||
| development native plugins and workers. Replace tuple results, split C/Go | ||
| outputs, metadata envelopes, and parallel mark-aware registrations with the | ||
| canonical outcome and the existing `register_llm_request_intercept` | ||
| registration name. | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.