Add OCI Generative AI provider codecs and LangChain integration guide - #549
Add OCI Generative AI provider codecs and LangChain integration guide#549fede-kamel wants to merge 2 commits into
Conversation
Documents NeMo Relay observability for LangChain agents backed by OCI Generative AI via langchain-oci, covering both on-demand models and dedicated AI cluster endpoints (including imported NVIDIA Nemotron models). Adds a langchain-oci optional-dependency extra and callback tests for OCI payload scope naming, metadata pass-through, and lifecycle handling. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
Implements the LlmCodec and LlmResponseCodec protocols for the OCI Generative AI chat API in a new nemo_relay.providers package. The request codec normalizes both GENERIC (Meta, Google, xAI, OpenAI, and imported open-weights models such as NVIDIA Nemotron on dedicated AI clusters) and COHERE chat formats into AnnotatedLLMRequest, mapping parameters onto GenerationParams fields and flat OCI tool calls onto the normalized nested shape, and merges intercept edits back into the original payload without dropping provider-specific or envelope fields. The response codec normalizes ChatResult payloads, including usage counters, for LLMEnd event annotation, tolerating camelCase, snake_case, and kebab-case key conventions. Unit tests cover round-trips, redaction-style edits, tool calls, and response decoding for both API formats. An opt-in live integration test (NEMO_RELAY_OCI_LIVE=1) executes a real chat through nemo_relay.llm.execute with both codecs, posting the codec-encoded payload verbatim to the signed OCI REST endpoint; validated against a dedicated AI cluster endpoint hosting an imported NVIDIA Nemotron 3 model. Signed-off-by: Federico Kamelhar <federico.kamelhar@oracle.com>
github-actions
Bot
added
size:XL
WalkthroughAdds OCI Generative AI provider codecs for GENERIC and COHERE payloads, LangChain OCI integration coverage, an optional dependency group, documentation, unit tests, and opt-in live endpoint validation. ChangesOCI Generative AI support
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant LangChain
participant NemoRelay
participant OCIGenAIChatCodec
participant OCI_Generative_AI
LangChain->>NemoRelay: execute OCI chat request
NemoRelay->>OCIGenAIChatCodec: normalize request
OCIGenAIChatCodec->>NemoRelay: encode OCI request
NemoRelay->>OCI_Generative_AI: send chat request
OCI_Generative_AI->>NemoRelay: return chat result
NemoRelay->>OCIGenAIChatCodec: decode response
OCIGenAIChatCodec->>NemoRelay: return normalized response
Important Pre-merge checks failedPlease resolve all errors before merging. Addressing warnings is optional. ❌ Failed checks (1 error, 1 warning)
✅ Passed checks (3 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/supported-integrations/oci-generative-ai.mdx`:
- Around line 99-111: Update the ChatOCIGenAI example to use provider="generic"
for imported open-weights models such as NVIDIA Nemotron, or explicitly instruct
users to select the provider matching their imported model family; do not leave
provider="meta", which is specific to Meta Llama.
In `@python/nemo_relay/providers/oci_genai.py`:
- Around line 378-391: Normalize OCI tool calls before assigning them to the
response by mapping each entry through _oci_tool_call_to_normalized in both the
COHERE branch and the GENERIC choices branch of the response codec. Preserve
empty or missing tool-call behavior while ensuring
AnnotatedLLMResponse.tool_calls contains the normalized function payload shape.
- Around line 223-226: Update encode() to reuse the envelope key detected by
decode() instead of always writing the camelCase chatRequest key, preserving
chat-request and chat_request round-trips without leaving the original key
behind. Update _decode_generic() and _decode_cohere() to retrieve request
parameters through the same variant-aware lookup used by _get_first, so
camelCase, kebab-case, and snake_case parameter names are all retained during
decode/encode.
In `@python/tests/integrations/langchain_tests/test_oci_genai.py`:
- Around line 128-133: Update test_end_pops_scope to inspect the metadata passed
during handler.on_chain_end and assert that otel.status_code is "OK", while
retaining the existing mock_nemo_relay.scope.pop assertion.
In `@python/tests/providers/test_oci_genai_live.py`:
- Around line 40-79: Convert the test-setup helpers _signer_and_endpoint and
_serving_mode into pytest fixtures, using explicit fixture names such as
signer_and_endpoint and serving_mode with appropriately named fixture functions.
Update the affected test to receive these values through fixture injection,
while preserving the existing credential loading, endpoint construction,
environment handling, and skip behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: e09454eb-ab64-4f0d-836a-a6b11261f100
⛔ Files ignored due to path filters (1)
uv.lockis excluded by!**/*.lock
📒 Files selected for processing (7)
docs/supported-integrations/oci-generative-ai.mdxpyproject.tomlpython/nemo_relay/providers/__init__.pypython/nemo_relay/providers/oci_genai.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/tests/providers/test_oci_genai_live.py
📜 Review details
🧰 Additional context used
📓 Path-based instructions (23)
**/*.{rs,py}
📄 CodeRabbit inference engine (AGENTS.md)
Follow binding naming conventions in Rust and Python: use
snake_case.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*.{rs,py,js,mjs,cjs,ts,tsx}
📄 CodeRabbit inference engine (AGENTS.md)
**/*.{rs,py,js,mjs,cjs,ts,tsx}: UseJson = serde_json::Valuein Rust-facing runtime APIs where the existing code expects JSON payloads.
UseResult<T>withFlowErrorin core runtime paths, and keep errors explicit and binding-appropriate at the wrapper layer.
Keep async behavior on the existing tokio-based model; bindings should preserve callback and future lifetimes rather than blocking or hiding async work unexpectedly.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
python/nemo_relay/**/*.py
📄 CodeRabbit inference engine (AGENTS.md)
Python wrapper modules live under
python/nemo_relay/, and the native extension is built fromcrates/pythonwithmaturin.
Files:
python/nemo_relay/providers/__init__.pypython/nemo_relay/providers/oci_genai.py
**/*.py
📄 CodeRabbit inference engine (CONTRIBUTING.md)
**/*.py: When changing the Python wrapper package, tests, or docs tooling, lint with Ruff (E,F,W,I), format with Ruff formatter (120-character lines, double quotes), and passtytype checking.
Add the SPDX license header to all Python source files using the#comment form.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*.{rs,py,go,js,ts,c,h}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Use language-appropriate naming conventions: Rust
snake_case, C FFI exports prefixednemo_relay_, GoPascalCase, Node.jscamelCase, and Pythonsnake_case.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
{crates/python/src/py_api/mod.rs,python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go,crates/node/src/api/**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update the language-native bindings for every exposed surface in Python, Go, and Node.js.
Files:
python/nemo_relay/providers/__init__.pypython/nemo_relay/providers/oci_genai.py
{python/nemo_relay/**/*.py,python/nemo_relay/**/*.pyi,go/nemo_relay/**/*.go}
📄 CodeRabbit inference engine (.agents/skills/add-binding-feature/SKILL.md)
Update language wrapper helpers such as Python wrapper modules, Python type stubs, and Go shorthand packages when the new behavior belongs in those helper layers.
Files:
python/nemo_relay/providers/__init__.pypython/nemo_relay/providers/oci_genai.py
{crates/**/src/**/*.rs,python/**/*.py}
📄 CodeRabbit inference engine (.agents/skills/maintain-dynamic-plugins/SKILL.md)
Do not add tests under
src; Rust tests belong in cratetests/trees, and Python SDK tests belong underpython/tests.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*.{py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/maintain-observability/SKILL.md)
Keep Python, Go, and Node.js config objects and subscriber/exporter methods aligned so all bindings expose the same logical knobs and semantics.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
**/*: Format changed files with the language-native formatter before the final lint/test pass.
If dynamic plugin behavior changed, usemaintain-dynamic-pluginsand include the native SDK, worker protocol, Python SDK, docs, packaging, and Codecov surfaces in the validation plan.
If code changes alter APIs, bindings, commands, paths, packaging behavior, observability/adaptive semantics, or documented best practices, update any dependent maintainer or consumer skills in the same branch.
During iteration, preferuv run pre-commit run --files <changed files...>.
Before review or handoff, runuv run pre-commit run --all-files.
Files:
python/nemo_relay/providers/__init__.pypyproject.tomldocs/supported-integrations/oci-generative-ai.mdxpython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*.{rs,py,go,js,ts}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If a language surface changed, always run that language's test target even when Rust core did not change.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
**/*.{rs,py,js,ts,tsx,go,java,kt,swift}
📄 CodeRabbit inference engine (.agents/skills/add-middleware/SKILL.md)
Add tests covering registration and duplicate names, deregistration and missing names, priority ordering, callback failure policy, scope-local inheritance and cleanup, event payload semantics, immutable mark and scope fields, and parity across affected bindings.
Files:
python/nemo_relay/providers/__init__.pypython/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.pypython/nemo_relay/providers/oci_genai.py
python/nemo_relay/**/*
⚙️ CodeRabbit configuration file
python/nemo_relay/**/*: Review Python wrapper changes for typed API consistency, contextvars-based scope isolation, async behavior, and parity with the native extension.
Stubs and runtime implementations should stay aligned.
Files:
python/nemo_relay/providers/__init__.pypython/nemo_relay/providers/oci_genai.py
**/*.toml
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all TOML files using the
#comment form.
Files:
pyproject.toml
pyproject.toml
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Keep Python packaging metadata in the root
pyproject.tomlconsistent with the package’s published name, imports, and build behavior.
Files:
pyproject.toml
**/*.mdx
📄 CodeRabbit inference engine (.agents/skills/review-doc-style/SKILL.md)
MDX top-of-file SPDX comments must use {/* ... */} delimiters instead of HTML comment delimiters (Must-Fix)
In MDX files, top-of-file comments must use JSX comment delimiters (
{/*to open and*/}to close); do not use HTML comments for MDX SPDX headers
Files:
docs/supported-integrations/oci-generative-ai.mdx
**/*.{md,mdx}
📄 CodeRabbit inference engine (AGENTS.md)
Update
README.md,fern/, package READMEs, and binding-support notes when public behavior, package names, examples, or supported bindings change.
**/*.{md,mdx}: Prefer the documented public API, not internal shortcuts
Keep package names, repo references, and build commands current
Keep release-process and release-notes guidance in repo-maintainer docs such asRELEASING.md, not as user-facing docs pages orCHANGELOG.md
Keep stable user-facing wrappers atscripts/root in docs and examples; only point at namespaced helper paths when documenting internal maintenance work
When detailed dynamic plugin guides exist, keep Rust native plugin examples, Python worker plugin examples, andgrpc-v1protocol details on separate pagesIf links in documentation change, run
just docs-linkcheck.
Files:
docs/supported-integrations/oci-generative-ai.mdx
**/*.{md,markdown,mdx}
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Add the SPDX license header to all Markdown/MDX documentation files using the HTML comment block form.
Files:
docs/supported-integrations/oci-generative-ai.mdx
{docs,examples}/**/*
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update docs and examples.
Files:
docs/supported-integrations/oci-generative-ai.mdx
docs/**/*
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If documentation examples or commands under
docs/change, run the targeted docs checks appropriate to the change.
Files:
docs/supported-integrations/oci-generative-ai.mdx
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}
⚙️ CodeRabbit configuration file
{docs/**,README.md,CONTRIBUTING.md,RELEASING.md,SECURITY.md}: Review documentation for technical accuracy against the current API, command correctness, and consistency across language bindings.
Flag stale examples, missing SPDX headers where required, and instructions that no longer match CI or pre-commit behavior.
Files:
docs/supported-integrations/oci-generative-ai.mdx
python/tests/**/*.py
📄 CodeRabbit inference engine (.agents/skills/test-python-binding/SKILL.md)
python/tests/**/*.py: Pytest is used to run tests.
Do not add@pytest.mark.asyncioto any test; async tests are automatically detected and run by the async runner.
Do not add a-> Nonereturn type annotation to test functions.
When mocking a class, do not define a new class; useunittest.mock.MagicMockorunittest.mock.AsyncMock, with thespecconstructor argument when necessary.
Name mocked classes with themockprefix, notfake.
Prefer pytest fixtures over helper methods.
Do not repeat fixtures; if a fixture is needed in multiple test files, place it in aconftest.pyfile.
When creating a fixture, use@pytest.fixture(name="<fixture_name>"[, scope="<scope>"])and define the fixture function asdef <fixture_name>_fixture() -> <return_type>:; only specifyscopewhen it is notfunction.
Preferpytest.mark.parametrizeover creating individual tests for different input types.
Files:
python/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.py
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}
⚙️ CodeRabbit configuration file
{crates/**/tests/**,python/tests/**,go/nemo_relay/**/*_test.go}: Tests should cover the behavior promised by the changed API surface, including error paths and cross-request isolation where relevant.
Prefer assertions on lifecycle events, scope stacks, middleware ordering, and binding parity over shallow smoke tests.
Files:
python/tests/providers/test_oci_genai_live.pypython/tests/integrations/langchain_tests/test_oci_genai.pypython/tests/providers/test_oci_genai_codec.py
🪛 ast-grep (0.44.1)
python/tests/providers/test_oci_genai_live.py
[warning] 106-106: Request-controlled URL passed to requests; validate against an allowlist to prevent SSRF.
Context: requests.post(endpoint, json=request.content, auth=signer, timeout=120)
Note: [CWE-918] Server-Side Request Forgery (SSRF).
(ssrf-requests)
[warning] 54-54: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(os.path.expanduser(token_file), encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🪛 Ruff (0.15.21)
python/tests/providers/test_oci_genai_live.py
[warning] 40-40: Missing return type annotation for private function _signer_and_endpoint
(ANN202)
[warning] 106-106: Missing return type annotation for private function call_oci
(ANN202)
python/tests/integrations/langchain_tests/test_oci_genai.py
[warning] 49-49: Use @pytest.fixture over @pytest.fixture()
Remove parentheses
(PT001)
[warning] 56-56: Use @pytest.fixture over @pytest.fixture()
Remove parentheses
(PT001)
[warning] 57-57: Unused function argument: mock_nemo_relay
(ARG001)
python/tests/providers/test_oci_genai_codec.py
[warning] 16-16: Dynamically typed expressions (typing.Any) are disallowed in _j
(ANN401)
python/nemo_relay/providers/oci_genai.py
[warning] 59-59: Dynamically typed expressions (typing.Any) are disallowed in _get_first
(ANN401)
[warning] 81-81: Dynamically typed expressions (typing.Any) are disallowed in content
(ANN401)
[warning] 81-81: Dynamically typed expressions (typing.Any) are disallowed in _generic_content_to_text
(ANN401)
[warning] 97-97: Dynamically typed expressions (typing.Any) are disallowed in content
(ANN401)
[warning] 97-97: Dynamically typed expressions (typing.Any) are disallowed in _text_to_generic_content
(ANN401)
[warning] 104-104: Dynamically typed expressions (typing.Any) are disallowed in usage
(ANN401)
[warning] 121-121: Dynamically typed expressions (typing.Any) are disallowed in tool_call
(ANN401)
[warning] 121-121: Dynamically typed expressions (typing.Any) are disallowed in _oci_tool_call_to_normalized
(ANN401)
[warning] 135-135: Dynamically typed expressions (typing.Any) are disallowed in tool_call
(ANN401)
[warning] 135-135: Dynamically typed expressions (typing.Any) are disallowed in _normalized_tool_call_to_oci
(ANN401)
🔇 Additional comments (12)
python/tests/integrations/langchain_tests/test_oci_genai.py (2)
1-124: LGTM!
135-145: LGTM!pyproject.toml (1)
94-97: LGTM!docs/supported-integrations/oci-generative-ai.mdx (2)
1-98: LGTM!
114-124: LGTM!python/nemo_relay/providers/oci_genai.py (4)
59-145: LGTM!
157-183: LGTM!
228-346: LGTM!
404-407: LGTM!python/nemo_relay/providers/__init__.py (1)
1-16: LGTM!python/tests/providers/test_oci_genai_codec.py (1)
82-318: LGTM!python/tests/providers/test_oci_genai_live.py (1)
106-109: LGTM!
| To call a model hosted on an OCI dedicated AI cluster — including imported | ||
| open-weights models such as NVIDIA Nemotron — pass the endpoint OCID as the | ||
| model ID. Everything else, including the NeMo Relay integration, stays the | ||
| same: | ||
|
|
||
| ```python | ||
| model = ChatOCIGenAI( | ||
| model_id="ocid1.generativeaiendpoint.oc1.<region>.<endpoint-id>", | ||
| service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com", | ||
| compartment_id="<compartment-ocid>", | ||
| auth_type="API_KEY", | ||
| provider="meta", | ||
| ) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the relevant docs and implementation.
git ls-files 'docs/supported-integrations/oci-generative-ai.mdx' 'README.md' 'docs/**' 'src/**' 'libs/**' 'packages/**' 2>/dev/null | sed -n '1,200p'
printf '\n--- provider references ---\n'
rg -n 'provider\s*=' docs/supported-integrations/oci-generative-ai.mdx README.md docs src libs packages -g '!**/dist/**' -g '!**/build/**' || true
printf '\n--- ChatOCIGenAI references ---\n'
rg -n 'ChatOCIGenAI|langchain-oci|generic|meta|Nemotron|OCI dedicated AI cluster|generativeaiendpoint' docs src libs packages -g '!**/dist/**' -g '!**/build/**' || trueRepository: NVIDIA/NeMo-Relay
Length of output: 26238
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the most relevant file sections with line numbers.
wc -l docs/supported-integrations/oci-generative-ai.mdx
sed -n '80,130p' docs/supported-integrations/oci-generative-ai.mdxRepository: NVIDIA/NeMo-Relay
Length of output: 1637
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Read the implementation around the provider argument.
git ls-files | rg 'oci|generativeai|ChatOCIGenAI|langchain-oci' || trueRepository: NVIDIA/NeMo-Relay
Length of output: 399
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '90,120p' docs/supported-integrations/oci-generative-ai.mdxRepository: NVIDIA/NeMo-Relay
Length of output: 1177
🌐 Web query:
langchain-oci provider generic meta imported model Nemotron ChatOCIGenAI documentation
💡 Result:
To use an imported NVIDIA Nemotron model on Oracle Cloud Infrastructure (OCI) with LangChain, you should utilize the ChatOCIGenAI class provided by the langchain-oci package [1][2]. When using imported models (such as Nemotron 3 Super) hosted on a Dedicated AI Cluster (DAC) within the OCI Generative AI service, you configure the ChatOCIGenAI client by specifying the model's endpoint OCID and setting the provider argument to "generic" [2][3]. Implementation example: from langchain_oci import ChatOCIGenAI # Using an imported model on a Dedicated AI Cluster llm = ChatOCIGenAI( model_id="ocid1.generativeaiendpoint.oc1.us-chicago-1.example...", # Your endpoint OCID provider="generic", # Required for imported models service_endpoint="https://inference.generativeai.us-chicago-1.oci.oraclecloud.com", compartment_id="ocid1.compartment.oc1..example...", model_kwargs={"temperature": 0.7, "max_tokens": 500},) Key details for this setup: - Package: You must use the langchain-oci package [4]. The older OCI integrations in langchain-community are deprecated [4]. - Provider: The provider parameter must be set to "generic" for imported models to properly interface with the custom endpoints created via OCI's Model Import capability [2][3]. - Model ID: For imported models, the model_id parameter requires the full OCID of the Generative AI endpoint (beginning with ocid1.generativeaiendpoint), not just the model name [2][3]. - Infrastructure: OCI supports importing open-weights models like NVIDIA Nemotron through the OCI Generative AI Model Import feature, which allows these models to be deployed on Dedicated AI Clusters and accessed using standard OCI Generative AI APIs and LangChain wrappers [5][6].
Citations:
- 1: https://docs.langchain.com/oss/python/integrations/chat/oci_generative_ai
- 2: https://pypi.org/project/langchain-oci/
- 3: https://github.com/oracle/langchain-oracle/tree/main/libs/oci
- 4: https://docs.oracle.com/en-us/iaas/Content/generative-ai/setup-oci-langchain.htm
- 5: https://blogs.oracle.com/cloud-infrastructure/nvidia-nemotron-3-super-on-oci
- 6: https://oracle.hydrogen.sagittarius.connect.product.adaptavist.com/iaas/releasenotes/generative-ai/nvidia-nemotron-3-super.htm
Use generic for imported OCI models like Nemotron. provider="meta" is for Meta Llama; this example should use generic or explicitly tell users to choose the provider that matches the imported model family.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@docs/supported-integrations/oci-generative-ai.mdx` around lines 99 - 111,
Update the ChatOCIGenAI example to use provider="generic" for imported
open-weights models such as NVIDIA Nemotron, or explicitly instruct users to
select the provider matching their imported model family; do not leave
provider="meta", which is specific to Meta Llama.
Source: Path instructions
| if has_envelope: | ||
| content["chatRequest"] = chat_request | ||
| return LLMRequest(original.headers, content) | ||
| return LLMRequest(original.headers, chat_request) |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
p = Path('python/nemo_relay/providers/oci_genai.py')
print(p, p.exists())
text = p.read_text()
for needle in ['def _get_first', 'chatRequest', '_decode_generic', '_decode_cohere', 'def encode', 'def decode']:
idx = text.find(needle)
print('\n###', needle, 'idx', idx)
if idx != -1:
start = max(0, text.rfind('\n', 0, idx-200))
end = min(len(text), idx+900)
snippet = text[start:end]
print(snippet)
PYRepository: NVIDIA/NeMo-Relay
Length of output: 7052
🏁 Script executed:
#!/bin/bash
set -euo pipefail
python3 - <<'PY'
from pathlib import Path
text = Path('python/nemo_relay/providers/oci_genai.py').read_text().splitlines()
def show(start, end):
for i in range(start-1, end):
if 0 <= i < len(text):
print(f"{i+1:4d}: {text[i]}")
for label, needle in [('_get_first', 'def _get_first'),
('decode', 'def decode'),
('encode', 'def encode'),
('decode_generic', 'def _decode_generic'),
('decode_cohere', 'def _decode_cohere')]:
for idx, line in enumerate(text, 1):
if needle in line:
print(f"\n## {label} at line {idx}")
show(max(1, idx-20), idx+80)
break
PYRepository: NVIDIA/NeMo-Relay
Length of output: 24615
Preserve the original envelope key on encode
decode()accepts OCI CLI kebab/snake envelopes via_get_first, butencode()always writes backcontent["chatRequest"]. Forchat-request/chat_requestinputs, that leaves the original key in place and breaksencode(decode(x), x)round-trips._decode_generic()and_decode_cohere()still read params with exact camelCase keys, so kebab/snake request params are dropped as well.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/nemo_relay/providers/oci_genai.py` around lines 223 - 226, Update
encode() to reuse the envelope key detected by decode() instead of always
writing the camelCase chatRequest key, preserving chat-request and chat_request
round-trips without leaving the original key behind. Update _decode_generic()
and _decode_cohere() to retrieve request parameters through the same
variant-aware lookup used by _get_first, so camelCase, kebab-case, and
snake_case parameter names are all retained during decode/encode.
| if api_format == "COHERE": | ||
| message = _get_first(chat_response, "text") | ||
| tool_calls = _get_first(chat_response, "toolCalls") | ||
| finish_reason = _get_first(chat_response, "finishReason") | ||
| else: | ||
| choices = _get_first(chat_response, "choices") or [] | ||
| if choices: | ||
| first_choice = choices[0] | ||
| finish_reason = _get_first(first_choice, "finishReason") | ||
| raw_message = _get_first(first_choice, "message") | ||
| if isinstance(raw_message, dict): | ||
| message = _generic_content_to_text(_get_first(raw_message, "content")) | ||
| tool_calls = _get_first(raw_message, "toolCalls") | ||
|
|
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
printf '\n== File outline ==\n'
ast-grep outline python/nemo_relay/providers/oci_genai.py --view expanded || true
printf '\n== Relevant lines in oci_genai.py ==\n'
sed -n '240,430p' python/nemo_relay/providers/oci_genai.py | cat -n
printf '\n== Search for normalization helper and response model ==\n'
rg -n "_oci_tool_call_to_normalized|AnnotatedLLMResponse|tool_calls" python/nemo_relay -S
printf '\n== Read model definitions if found ==\n'
files=$(rg -l "class AnnotatedLLMResponse|def _oci_tool_call_to_normalized" python/nemo_relay -S || true)
for f in $files; do
echo "--- $f ---"
sed -n '1,260p' "$f" | cat -n
doneRepository: NVIDIA/NeMo-Relay
Length of output: 41470
🏁 Script executed:
#!/bin/bash
set -euo pipefail
pwd
lsRepository: NVIDIA/NeMo-Relay
Length of output: 577
🏁 Script executed:
#!/bin/bash
set -euo pipefail
rg -n "class AnnotatedLLMResponse|AnnotatedLLMResponse\\(|tool_calls|_oci_tool_call_to_normalized" \
python/nemo_relay -S
printf '\n== oci_genai.py around helper ==\n'
sed -n '1,340p' python/nemo_relay/providers/oci_genai.py | cat -n
printf '\n== oci_genai.py around response decode ==\n'
sed -n '340,430p' python/nemo_relay/providers/oci_genai.py | cat -nRepository: NVIDIA/NeMo-Relay
Length of output: 24442
Normalize OCI response tool calls before populating AnnotatedLLMResponse.tool_calls
AnnotatedLLMResponse.tool_calls is documented as normalized, but this codec forwards OCI toolCalls raw in both COHERE and GENERIC paths. Map each entry through _oci_tool_call_to_normalized so response annotations match the request-side shape and downstream consumers see a consistent function payload.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/nemo_relay/providers/oci_genai.py` around lines 378 - 391, Normalize
OCI tool calls before assigning them to the response by mapping each entry
through _oci_tool_call_to_normalized in both the COHERE branch and the GENERIC
choices branch of the response codec. Preserve empty or missing tool-call
behavior while ensuring AnnotatedLLMResponse.tool_calls contains the normalized
function payload shape.
| def test_end_pops_scope(self, handler: NemoRelayCallbackHandler, mock_nemo_relay: MagicMock): | ||
| run_id = uuid4() | ||
| handler.on_chain_start({"id": OCI_CHAT_MODEL_ID}, {"input": "test"}, run_id=run_id) | ||
| handler.on_chain_end({"output": "72 degrees"}, run_id=run_id) | ||
|
|
||
| mock_nemo_relay.scope.pop.assert_called_once() |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Assert the successful lifecycle status.
on_chain_end supplies otel.status_code="OK", but this test only checks that pop occurred. Assert the metadata value so a regression in successful completion status is detected. As per path instructions, tests should cover promised API behavior, including lifecycle events.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/tests/integrations/langchain_tests/test_oci_genai.py` around lines 128
- 133, Update test_end_pops_scope to inspect the metadata passed during
handler.on_chain_end and assert that otel.status_code is "OK", while retaining
the existing mock_nemo_relay.scope.pop assertion.
Source: Path instructions
| def _signer_and_endpoint(): | ||
| """Build a request signer and the regional chat endpoint URL. | ||
|
|
||
| Signing raw HTTP (instead of using SDK model classes) sends the codec's | ||
| encoded payload to the service verbatim, so the test also validates that | ||
| ``OCIGenAIChatCodec.encode()`` produces the exact OCI wire format. | ||
| """ | ||
| oci = pytest.importorskip("oci") | ||
|
|
||
| profile = os.environ.get("OCI_CLI_PROFILE", "DEFAULT") | ||
| region = os.environ.get("OCI_GENAI_REGION", "us-chicago-1") | ||
| config = oci.config.from_file(profile_name=profile) | ||
|
|
||
| token_file = config.get("security_token_file") | ||
| if token_file: | ||
| with open(os.path.expanduser(token_file), encoding="utf-8") as handle: | ||
| token = handle.read().strip() | ||
| private_key = oci.signer.load_private_key_from_file(config["key_file"]) | ||
| signer = oci.auth.signers.SecurityTokenSigner(token, private_key) | ||
| else: | ||
| signer = oci.signer.Signer( | ||
| tenancy=config["tenancy"], | ||
| user=config["user"], | ||
| fingerprint=config["fingerprint"], | ||
| private_key_file_location=config["key_file"], | ||
| pass_phrase=config.get("pass_phrase"), | ||
| ) | ||
|
|
||
| endpoint = f"https://inference.generativeai.{region}.oci.oraclecloud.com/20231130/actions/chat" | ||
| return signer, endpoint | ||
|
|
||
|
|
||
| def _serving_mode() -> dict[str, str]: | ||
| endpoint_id = os.environ.get("OCI_GENAI_ENDPOINT_ID") | ||
| if endpoint_id: | ||
| return {"servingType": "DEDICATED", "endpointId": endpoint_id} | ||
| model_id = os.environ.get("OCI_GENAI_MODEL_ID") | ||
| if not model_id: | ||
| pytest.skip("set OCI_GENAI_ENDPOINT_ID or OCI_GENAI_MODEL_ID") | ||
| return {"servingType": "ON_DEMAND", "modelId": model_id} |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
Prefer pytest fixtures over setup helpers.
_signer_and_endpoint and _serving_mode are test-setup helpers; per repo test conventions these should be pytest fixtures (e.g. @pytest.fixture(name="signer_and_endpoint") / def signer_and_endpoint_fixture()), injected into the test.
As per coding guidelines: "Prefer pytest fixtures over helper methods."
🧰 Tools
🪛 ast-grep (0.44.1)
[warning] 54-54: File path is request-/variable-derived; validate and normalize to prevent path traversal.
Context: open(os.path.expanduser(token_file), encoding="utf-8")
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(open-filename-from-request)
🪛 Ruff (0.15.21)
[warning] 40-40: Missing return type annotation for private function _signer_and_endpoint
(ANN202)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@python/tests/providers/test_oci_genai_live.py` around lines 40 - 79, Convert
the test-setup helpers _signer_and_endpoint and _serving_mode into pytest
fixtures, using explicit fixture names such as signer_and_endpoint and
serving_mode with appropriately named fixture functions. Update the affected
test to receive these values through fixture injection, while preserving the
existing credential loading, endpoint construction, environment handling, and
skip behavior.
Source: Coding guidelines
willkill07
left a comment
willkill07
left a comment
There was a problem hiding this comment.
If the goal of the PR is to have the OCI Codec integration, then we need to do it in Rust with appropriate matching logic, exhaustive testing, and exposure to language bindings.
If the goal of the PR is to have OCI example, then there needs to be an actual value-add -- as written, there is nothing strictly requiring langchain-oci in NeMo Relay. (See the minimal surface area of langchain-nvidia-ai-endpoints as an example).
There was a problem hiding this comment.
These types of codecs should probably live in the Rust core so any language can leverage it natively. As written, this would only be compatible in Python (which is uneven support and harder to justify).
There was a problem hiding this comment.
This file makes no reference to the created codecs. I claim that in order for any middleware to function, it would require the Relay library itself to be aware of the created Codecs.
I believe this supports my other comment.
| "aiohttp>=3.14.1", | ||
| ] | ||
|
|
||
| langchain-oci = [ |
There was a problem hiding this comment.
I don't see the need to have this extra. There is no code that depends on the langchain-oci package
|
Thanks for the fast review, @willkill07 — agreed on all three points, and they resolve each other neatly:
I'll close the loop on #548 with the split plan. If you have a preference on where the Rust codec should live relative to the existing three (same module vs. a providers submodule), happy to follow it. |
|
Closing per the review: the codec belongs in Rust core with matching logic, exhaustive tests, and exposure through the language bindings — reworking it that way, using this PR's Python implementation and live-validated wire fixtures as the spec. The docs guide will return separately only if it carries real value-add beyond example code, per the langchain-nvidia-ai-endpoints bar. Tracking in #548. |
#### Overview Adds Oracle Cloud Infrastructure (OCI) Generative AI as a built-in provider across the Rust core and the Python/Node bindings. Originally opened as layer 1 of a four-part stacked series; per reviewer request ([comment](#554 (comment))), the remaining three layers are now folded into this single PR, so it carries the complete feature. The four capability layers, now all in this PR (one commit each, reviewable in sequence): 1. **Typed variants + response codec** — `ApiSpecific` OCI variants and `LlmResponseCodec` for `GENERIC`, `COHERE`, and `COHEREV2` `ChatResult` payloads 2. **Request codec** — `LlmCodec` decode plus merge-not-replace encode with `encode(decode(x), x) == x` identity and unmodeled-field preservation 3. **Provider surface + streaming** — `ProviderSurface::OCIGenAI` registration and detection, `OCIGenAIStreamingCodec` (SSE for both formats), and OCI awareness in guardrails, the PII-redaction overlay, and adaptive request surfaces 4. **Bindings** — `OCIGenAIChatCodec` exposed to Python (pyo3, `.pyi`, `codecs.py`) and Node (napi, `.d.ts`), with binding tests - [x] I confirm this contribution is my own work, or I have the right to submit it under this project's license. - [x] I searched existing issues and open pull requests, and this does not duplicate existing work. #### Details **What**: `ApiSpecificRequest::OCIGenAI { compartment_id, serving_mode, api_format }` and `ApiSpecificResponse::OCIGenAI { api_format, model_version }` (serde tag `"oci_genai"`) in `crates/types`, plus `crates/core/src/codec/oci_genai.rs` introducing `OCIGenAIChatCodec`: - **Response decode** covers all three documented `apiFormat`s — `GENERIC` (`choices`-based; Meta Llama, Google, xAI, OpenAI, and imported open-weights models such as NVIDIA Nemotron on dedicated AI clusters), `COHERE` (`text`-based), and `COHEREV2` (single assistant `message` with typed content parts and nested-function tool calls) — normalizing model, response id, message content, tool calls, finish reasons, and usage counters (including `promptTokensDetails.cachedTokens`). Envelope and chat-response fields outside the normalized shape (`timeCreated`, `serviceTier`, `chatHistory`, grounding metadata) are preserved in `extra`. - **Request decode/encode** normalizes OCI `ChatDetails` for both formats and re-encodes by merging into the original payload rather than replacing it, so unmodeled fields survive intercept round-trips; the encode/decode identity is asserted in tests. - **Provider surface**: `ProviderSurface::OCIGenAI` with detection keyed on the strongest-signal envelope fields (placed first in the resolver), so OCI payloads resolve without an explicit codec. Streaming is handled by `OCIGenAIStreamingCodec` (SSE lifecycle for GENERIC and COHERE). Guardrails, the PII-redaction overlay, and adaptive request surfaces gain OCI-format awareness with parity cases mirroring the existing providers. - **Bindings**: `OCIGenAIChatCodec` classes in Python and Node with the same construction and resolver-integration semantics as the other built-in codecs. - **Codec identity**: `BuiltinLlmCodec::OCIGenAI` surfaced end to end — `codec_identity()` on both codec trait impls, worker SDK proto decoding (`crates/worker`), native plugin SDK enum and async invocation context (`crates/plugin`), the Node `LlmCodecIdentity` typing, and PII-redaction surface routing — so sanitize callbacks in every SDK receive `{ kind: "builtin", id: "oci_genai" }` instead of `opaque`. The codec accepts the REST wire format only (camelCase, as documented); converting alternate renderings produced by Oracle tooling is the caller's responsibility, per review discussion. **Why**: OCI GenAI calls are currently opaque to Relay. This PR provides normalized `LLMEnd` annotations (model, finish reason, token usage), safe request editing for intercepts (redaction, policy), automatic provider detection including streaming, and access from every primary binding. **How**: Follows the built-in provider pattern (`anthropic.rs` as template): unit-struct codec, `FinishReason` mapping for the three formats' vocabularies (`stop`/`length`/`max_tokens`/`tool_calls`; `COMPLETE`/`MAX_TOKENS`; `TOOL_CALL`/`STOP_SEQUENCE`), positional `call_{index}` fallback ids for COHERE tool calls (no `id` on the wire). Binding classes mirror `AnthropicMessagesCodec`/`OpenAIChatCodec` exposure. **Testing**: Full workspace suite green, `cargo clippy --workspace --all-targets` zero warnings, `cargo fmt --check` clean, `missing_docs` satisfied; `just test-python` (567 passed) and `just test-node` (294 passed) green on the built bindings. OCI-specific coverage includes: GENERIC/COHERE fixtures from live OCI wire captures (simple, tool-call, mixed-content, tool-call-only, model-family sweep across Meta, OpenAI, Google, xAI, Cohere), COHEREV2 fixtures per the published `CohereChatResponseV2` schema with the wire shape confirmed against the live service, request round-trip identity and unmodeled-field preservation, wire-format-only contract tests, streaming lifecycle tests for both formats, resolver detection and parity cases, PII-redaction overlay coverage, invalid-content error paths, codec-identity tests across core, worker, plugin, and pii-redaction, and binding tests in both languages. The combined content was also validated end-to-end: the codec-encoded request was posted verbatim to the signed OCI REST chat endpoint (dedicated AI cluster serving an imported NVIDIA Nemotron 3 model) and the live reply decoded with correct finish reason and token usage. Known unrelated flake: `install_registrations_covers_registry_error_edges` (dynamic worker plugin) is parallelism-sensitive on `main`; the worker code here is byte-identical to `main` and the test passes serially and in isolation. **Breaking changes**: None — additive enum variants, a new codec module, a new provider surface, and new binding classes. #### Where should the reviewer start? `crates/core/src/codec/oci_genai.rs` top-to-bottom (module docs explain the three formats and the wire-only contract), then the fixtures and round-trip tests in `crates/core/tests/unit/codec/oci_genai_tests.rs`, then the resolver registration in `crates/core/src/codec/resolve.rs`. The four commits after the `main` merge apply the layers in order if commit-by-commit review is preferred. #### Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to) - Closes #548 - Relates to #549, #552 (superseded approaches; this PR now carries the complete consolidated content) ## Summary by CodeRabbit * **New Features** * Added OCI Generative AI support for GENERIC, COHERE, and COHEREV2 requests, responses, and streaming. * Added Python and Node.js APIs for encoding and decoding OCI payloads. * Added support for OCI metadata, tool calls, usage, finish reasons, and model details. * Added OCI compatibility to PII redaction, NeMo Guardrails, caching, replay, and configuration schemas. * **Bug Fixes** * Improved OCI payload detection, normalization, response processing, tool-call handling, and streaming extraction. Authors: - Fede Kamelhar (https://github.com/fede-kamel) - Will Killian (https://github.com/willkill07) Approvers: - Will Killian (https://github.com/willkill07) URL: #554
2 participants
Overview
Adds Oracle Cloud Infrastructure (OCI) Generative AI support to NeMo Relay: request/response provider codecs implementing the
LlmCodec/LlmResponseCodecprotocols, a supported-integrations guide for LangChain agents usinglangchain-oci, alangchain-ocioptional-dependency extra, and unit plus opt-in live integration tests.Details
What:
nemo_relay.providers.oci_genaiwithOCIGenAIChatCodecandOCIGenAIResponseCodec, covering both OCI chat wire formats —GENERIC(Meta Llama, Google Gemini, xAI Grok, OpenAI, and imported open-weights models such as NVIDIA Nemotron on dedicated AI clusters) andCOHERE(Cohere Command) — plusdocs/supported-integrations/oci-generative-ai.mdxand alangchain-ociextra.Why: OCI Generative AI traffic is currently opaque to Relay middleware and observability. With these codecs, request intercepts (PII redaction, guardrails, policy) and
LLMEndannotations work on OCI payloads exactly as they do for the built-in OpenAI and Anthropic codecs. Closes the provider gap for agents running against OCI-hosted models, including imported NVIDIA Nemotron models.How: Follows the Provider Codecs guide contract:
encode()uses baseline-compare-and-patch: it decodes the original request as a baseline and rewrites only messages/params that intercepts actually changed, soencode(decode(original), original) == originalat the JSON-value level and unmodeled provider fields (envelope fields,topK,seed, per-message extras, unknown future fields) survive edits. This identity guarantee is fully met forGENERIC; forCOHERE, identity holds for unedited requests, while message edits rebuild the modeledmessage/chatHistory/preambleOverridefields.customapi_specificvariant ({"api": "custom", "api_name": "oci_genai", ...}).GenerationParamsfields; flat OCItoolCallsmap to the normalized nested function shape and back; usage counters map onto normalizedUsagefields. The response codec tolerates camelCase (SDK/REST), snake_case, and kebab-case (CLI) key conventions.Testing:
python/tests/providers/test_oci_genai_codec.py): decode/encode round-trips for both formats, identity-invariant tests with unmodeled fields at envelope/request/message level, redaction-style edits proving untouched messages pass through verbatim, tool-call conversion, params patching, response decoding, key-convention tolerance.python/tests/providers/test_oci_genai_live.py, gated onNEMO_RELAY_OCI_LIVE=1): executes a real chat throughnemo_relay.llm.executewith both codecs, posting the codec-encoded payload verbatim to the signed OCI REST endpoint. Validated against a dedicated AI cluster endpoint serving an imported NVIDIA Nemotron 3 model (and the docs example against the on-demand catalog).test_dynamic_plugin_host.pyerrors reproduce identically without this change).ruff check,ruff format, pre-commit type check (ty), andfern checkare clean.Breaking changes: None. New package, new docs page, new optional extra; no existing modules changed except adding the
langchain-ociextra topyproject.toml.Where should the reviewer start?
python/nemo_relay/providers/oci_genai.py— theencode()baseline-compare-and-patch logic is the key design decision (mirrors the built-in codecs' documented identity guarantee). ThenTestIdentityInvariantinpython/tests/providers/test_oci_genai_codec.pyfor the proof, and the live test for the wire-format validation.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit