Repository navigation
docs: add OpenCode MLflow tracing on RHOAI - #209
Conversation
📝 WalkthroughWalkthroughAdds MLflow tracing support for OpenCode with a custom image build, a tracing overlay, and setup and validation documentation. ChangesOpenCode: MLflow tracing integration
Sequence DiagramsequenceDiagram
participant Operator
participant Kustomize
participant EntryPoint as entrypoint.sh
participant SAToken as k8s service account token
participant MLflowSDK as Python mlflow SDK
participant MLflowServer as MLflow tracking server
participant OpenCode as opencode
Operator->>Kustomize: apply mlflow-tracing overlay
Kustomize->>EntryPoint: inject env vars and entrypoint ConfigMap
EntryPoint->>EntryPoint: build OPENCODE_CONFIG_CONTENT
EntryPoint->>SAToken: read token for MLFLOW_TRACKING_TOKEN
EntryPoint->>MLflowSDK: resolve or create experiment
MLflowSDK->>MLflowServer: get_experiment_by_name / create_experiment
MLflowServer-->>MLflowSDK: experiment_id
MLflowSDK-->>EntryPoint: MLFLOW_EXPERIMENT_ID
EntryPoint->>OpenCode: exec opencode web or cli
OpenCode->>MLflowServer: export traces on session.idle
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
459bac3 to
bec0a7c
Compare
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agents/claude-code/deployment/Containerfile`:
- Around line 78-82: The git clone command pulling MLflow from upstream master
branch (lines 78-82) does not pin to a specific revision, which creates security
and reproducibility issues. Replace the generic clone operation with a pinned
commit hash or tag reference (using the --branch flag with a specific tag or
adding a commit hash), and add a comment documenting the exact pinned revision
being used. This ensures each build pulls the same immutable code and prevents
unreviewed upstream changes from being incorporated.
In `@agents/claude-code/deployment/deployment-ogx-vllm.yaml`:
- Around line 205-210: The current documentation for the MLflow npm plugin setup
recommends disabling TLS certificate verification by setting
NODE_TLS_REJECT_UNAUTHORIZED to "0", which is a security vulnerability. Replace
this recommendation in all four deployment files (deployment-ogx-vllm.yaml,
deployment.yaml, deployment-vllm.yaml, and deployment-vertex.yaml) with guidance
to mount the cluster or enterprise CA certificate and use the
NODE_EXTRA_CA_CERTS environment variable instead. Keep
NODE_TLS_REJECT_UNAUTHORIZED=0 only as a commented option for temporary
troubleshooting, making it clear this should not be used in production. Update
the comment blocks around the MLflow Tracing npm plugin sections to reflect this
secure-by-default approach.
In `@agents/claude-code/deployment/entrypoint.sh`:
- Around line 293-301: The code silently ignores a missing service account token
file without checking if MLFLOW_TRACKING_TOKEN is already set elsewhere, which
can cause silent authentication failures. After the FileNotFoundError exception
handler where sa_token_path is read, add a check to verify that
MLFLOW_TRACKING_TOKEN is present in the env dictionary. If the token was not
found in the file and is not already set in the environment, emit a warning
message before the json.dump call. Only proceed with the success log message at
the end if MLFLOW_TRACKING_TOKEN is actually defined in env, ensuring users are
aware when authentication credentials are missing rather than silently failing.
In `@agents/opencode/deployment/Containerfile.mlflow`:
- Around line 7-18: The Containerfile uses mutable references that prevent
reproducible builds. Replace the `:latest` tag in the base image reference
(quay.io/opendatahub/odh-opencode-rhel9:latest) with a specific version tag or
digest, change the open-ended Python dependency constraint from `>=3.14` to a
specific version like `==3.14.x`, and add a specific git ref to the git clone
command (such as `--branch` or `--tag`) instead of using the default branch.
This will ensure the image builds consistently and predictably across different
builds and environments.
In `@agents/opencode/deployment/entrypoint.sh`:
- Around line 32-84: The MLflow plugin is installed and its environment
variables are configured, but the plugin configuration is never added to
OPENCODE_CONFIG_CONTENT, so web mode cannot load the plugin. After the MLflow
environment variables are persisted to the mlflow-env file within the
MLFLOW_TRACKING_URI conditional block, inject the plugin configuration into
OPENCODE_CONFIG_CONTENT by appending or constructing a plugin entry that
references the `@mlflow/opencode` plugin with its required configuration (such as
tracking URI and experiment ID). This must be done before the final export
statement that sets OPENCODE_CONFIG_CONTENT, or as part of constructing that
variable's value, to ensure the plugin is actually loaded when the application
runs in web mode.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: fb591ed0-8a9d-494c-9d5c-ab4c42795e53
⛔ Files ignored due to path filters (2)
agents/opencode/deployment/docs/screenshots/ogx-tetris-trace.pngis excluded by!**/*.pngagents/opencode/deployment/docs/screenshots/vllm-tetris-trace.pngis excluded by!**/*.png
📒 Files selected for processing (12)
agents/claude-code/README.mdagents/claude-code/deployment/Containerfileagents/claude-code/deployment/deployment-ogx-vllm.yamlagents/claude-code/deployment/deployment-vertex.yamlagents/claude-code/deployment/deployment-vllm.yamlagents/claude-code/deployment/deployment.yamlagents/claude-code/deployment/entrypoint.shagents/claude-code/mlflow-tracing.mdagents/opencode/deployment/Containerfile.mlflowagents/opencode/deployment/docs/mlflow-tracing-setup.mdagents/opencode/deployment/docs/mlflow-tracing.mdagents/opencode/deployment/entrypoint.sh
0f945d5 to
02e3a7e
Compare
Add MLflow tracing docs and kustomize overlay for OpenCode: - docs/mlflow-tracing.md: investigation and validation results - docs/mlflow-tracing-setup.md: setup guide - overlays/mlflow-tracing/: kustomize overlay - Containerfile.mlflow: extended image with mlflow + plugin Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
02e3a7e to
f9a0122
Compare
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml (1)
11-21: 🔒 Security & Privacy | 🟠 MajorTLS certificate verification is disabled, creating MITM vulnerability.
Lines 17–20 disable MLflow tracking certificate verification despite using HTTPS (line 12). Setting
MLFLOW_TRACKING_INSECURE_TLS=trueandNODE_TLS_REJECT_UNAUTHORIZED=0bypasses TLS validation, allowing man-in-the-middle attacks on credential and trace data.While this may be necessary for RHOAI dev/test environments with self-signed certificates, it is not acceptable for production. Recommend:
- Document this as a development-only setting with security warnings.
- For production: use proper CA certificates or configure MLflow with a trusted certificate authority.
- Consider a conditional approach (dev vs. prod overlays) to avoid accidental production deployment with these insecure settings.
[major]
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml` around lines 11 - 21, The deployment patch has insecure TLS settings (MLFLOW_TRACKING_INSECURE_TLS and NODE_TLS_REJECT_UNAUTHORIZED) that disable certificate verification. Add a prominent comment in the YAML file documenting these as development-only settings with security warnings. Additionally, create separate overlay directories or conditional configurations for development and production environments, ensuring the production overlay removes or properly configures these TLS settings with actual CA certificates instead of disabling verification. This prevents accidental deployment of insecure settings to production.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml`:
- Line 14: The deployment-patch.yaml file contains a hardcoded placeholder
string "YOUR-NAMESPACE" on the value field that must be replaced before
deployment to properly configure MLflow workspace scoping. Either replace the
placeholder "YOUR-NAMESPACE" directly with the actual RHOAI namespace value, or
refactor it to use Kustomize variable substitution syntax like $(NAMESPACE) to
make it dynamic and reusable across different overlays. Ensure documentation or
deployment scripts explicitly require this substitution or automate it as part
of the deployment process.
In `@agents/opencode/deployment/overlays/mlflow-tracing/entrypoint-patch.yaml`:
- Around line 48-68: The service account token is being read once at entrypoint
time (line 49 reading from SA_TOKEN_PATH) and cached into the
MLFLOW_TRACKING_TOKEN variable which is then persisted to ~/.mlflow-env (line
64), but Kubernetes rotates these projected tokens on disk so the cached value
becomes stale. Instead of exporting the cached MLFLOW_TRACKING_TOKEN value to
~/.mlflow-env, modify the file generation section (lines 64-68) to have
MLFLOW_TRACKING_TOKEN dynamically read from the SA_TOKEN_PATH file at use time
(e.g., by using a command substitution like cat "$SA_TOKEN_PATH") rather than
using the pre-cached variable value, so the token stays current across CLI
sessions.
- Around line 51-62: The MLFLOW_EXPERIMENT_NAME variable is being directly
interpolated into the Python string literal using shell syntax, which creates a
security vulnerability where names containing single quotes or backslashes could
inject arbitrary Python code. Instead of interpolating the variable directly
into the quoted Python string, ensure MLFLOW_EXPERIMENT_NAME is available as an
environment variable and read it from os.environ inside the Python code using
os.environ.get() to safely access the variable value.
---
Outside diff comments:
In `@agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml`:
- Around line 11-21: The deployment patch has insecure TLS settings
(MLFLOW_TRACKING_INSECURE_TLS and NODE_TLS_REJECT_UNAUTHORIZED) that disable
certificate verification. Add a prominent comment in the YAML file documenting
these as development-only settings with security warnings. Additionally, create
separate overlay directories or conditional configurations for development and
production environments, ensuring the production overlay removes or properly
configures these TLS settings with actual CA certificates instead of disabling
verification. This prevents accidental deployment of insecure settings to
production.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: 8b4df0e7-6711-48de-aa18-83cc5cacde2c
⛔ Files ignored due to path filters (2)
agents/opencode/deployment/docs/screenshots/ogx-tetris-trace.pngis excluded by!**/*.pngagents/opencode/deployment/docs/screenshots/vllm-tetris-trace.pngis excluded by!**/*.png
📒 Files selected for processing (6)
agents/opencode/deployment/Containerfile.mlflowagents/opencode/deployment/docs/mlflow-tracing-setup.mdagents/opencode/deployment/docs/mlflow-tracing.mdagents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yamlagents/opencode/deployment/overlays/mlflow-tracing/entrypoint-patch.yamlagents/opencode/deployment/overlays/mlflow-tracing/kustomization.yaml
✅ Files skipped from review due to trivial changes (2)
- agents/opencode/deployment/docs/mlflow-tracing-setup.md
- agents/opencode/deployment/docs/mlflow-tracing.md
🚧 Files skipped from review as they are similar to previous changes (1)
- agents/opencode/deployment/Containerfile.mlflow
- Warn that NODE_TLS_REJECT_UNAUTHORIZED=0 is process-wide - Recommend NODE_EXTRA_CA_CERTS for production - Link to upstream issues mlflow#24140 (scoped TLS) and mlflow#24141 (K8s auth) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2146ec0 to
6af8e0e
Compare
| # - mlflow[kubernetes] for experiment creation and SA token auth | ||
| # - Pre-built @mlflow/core from upstream master (workspace header fix) | ||
|
|
||
| FROM quay.io/opendatahub/odh-opencode-rhel9:latest |
There was a problem hiding this comment.
Nit: the base image uses :latest — worth pinning to :1.4.4 (or a digest) for reproducible builds. We just hit the same thing in PR #219 and pinned it there.
Reviewed by Claude Code with @aakankshaduggal's supervision
|
|
||
| USER 0 | ||
|
|
||
| RUN pip3 install --no-cache-dir 'mlflow[kubernetes]>=3.14' |
There was a problem hiding this comment.
Minor: >=3.14 means builds could pull a future major version with breaking changes. Pinning to something like ==3.14.* would keep builds predictable.
Reviewed by Claude Code with @aakankshaduggal's supervision
| # with this fix yet. Building from master includes it. | ||
| # The base image already has nodejs 16 — just need npm. | ||
| RUN microdnf install -y --nodocs npm \ | ||
| && git clone --depth 1 https://github.com/mlflow/mlflow.git /tmp/mlflow \ |
There was a problem hiding this comment.
The git clone --depth 1 pulls whatever's on main at build time. Could you pin to a specific commit hash or tag? That way builds are reproducible and you're not pulling unreviewed upstream changes. Something like git clone --depth 1 --branch <tag-or-sha>.
Reviewed by Claude Code with @aakankshaduggal's supervision
| EXPERIMENT_ID=$(python3 -c " | ||
| import warnings; warnings.filterwarnings('ignore') | ||
| import os; import mlflow | ||
| os.environ.setdefault('MLFLOW_TRACKING_INSECURE_TLS', 'false') |
There was a problem hiding this comment.
Small safety thing: ${MLFLOW_EXPERIMENT_NAME} gets shell-expanded directly into the Python string, so an experiment name with quotes or backslashes could break (or worse). Reading it via os.environ['MLFLOW_EXPERIMENT_NAME'] inside the Python code would be safer.
Reviewed by Claude Code with @aakankshaduggal's supervision
|
|
||
| ## Prerequisites | ||
|
|
||
| - OpenCode deployment manifests from [aicatalyst-team/opencode-openshift](https://github.com/aicatalyst-team/opencode-openshift/pull/1) |
There was a problem hiding this comment.
Tiny thing: the link points to opencode-openshift/pull/1 which is already merged. Linking to the repo root (https://github.com/aicatalyst-team/opencode-openshift) would be more durable.
Reviewed by Claude Code with @aakankshaduggal's supervision
| # Requires: https://github.com/aicatalyst-team/opencode-openshift | ||
| resources: | ||
| - ../../manifests | ||
|
|
There was a problem hiding this comment.
The resources: [../../manifests] reference points to the opencode-openshift manifests, but this overlay lives in agentic-starter-kits. Might be worth adding a note that users need to clone the opencode-openshift repo and adjust the path, or use a remote resource URL.
Reviewed by Claude Code with @aakankshaduggal's supervision
- Pin base image to :1.4.4 - Pin mlflow to ==3.14.* - Pin git clone to commit 4953f67 (workspace header fix) - Fix shell injection: use os.environ instead of shell interpolation - Fix token rotation: .mlflow-env cats the SA token file at use time - Link to repo root instead of PR #1 - Add path adjustment note in kustomization.yaml Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Thanks for the review @aakankshaduggal! All comments addressed in the latest commit:
|
There was a problem hiding this comment.
Review: OpenCode MLflow Tracing on RHOAI
Overall the PR is well-structured and production-aware — good kustomize overlay pattern, proper USER directives, smart plugin override approach, and thorough TLS documentation. However, there are several issues that should be addressed before merge.
Important (6)
1. Full mlflow repo clone + short commit SHA — fragile and slow build
Containerfile.mlflow:19-20
&& git clone https://github.com/mlflow/mlflow.git /tmp/mlflow \
&& cd /tmp/mlflow && git checkout 4953f67599d1 && cd .. \
The mlflow repo is 700MB+. No --depth 1. The 12-char short SHA is fragile — could collide or be GC'd. Fix: git clone --depth 1 --branch v3.14.0 ... or at minimum add --depth 1 and use the full 40-char SHA.
2. NODE_TLS_REJECT_UNAUTHORIZED=0 with no production alternative overlay
deployment-patch.yaml:24-25
- name: NODE_TLS_REJECT_UNAUTHORIZED
value: "0"
The warning comment is good, but anyone applying this overlay gets TLS verification disabled for ALL Node.js connections. The NODE_EXTRA_CA_CERTS fix is only in a comment. Fix: Provide a second overlay (e.g. mlflow-tracing-prod/) using NODE_EXTRA_CA_CERTS, or remove this env var from the patch and document it as opt-in.
3. Plugin install stderr swallowed — makes debugging impossible
entrypoint-patch.yaml:39-41
opencode plugin @mlflow/opencode 2>/dev/null \
&& echo "[entrypoint] MLflow plugin installed" \
|| echo "[entrypoint] WARN: MLflow plugin install failed"
stderr goes to /dev/null. If plugin install fails, the user sees "WARN" with no reason. Fix: Change 2>/dev/null to 2>&1 so failure details appear in pod logs.
4. Quoted heredoc prevents experiment ID persistence in new shells
entrypoint-patch.yaml:66-69
cat > "$HOME/.mlflow-env" <<'MLENV'
export MLFLOW_TRACKING_TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token 2>/dev/null)
export MLFLOW_EXPERIMENT_ID="${MLFLOW_EXPERIMENT_ID:-}"
MLENV
The <<'MLENV' (quoted) prevents variable expansion at write time. The token line works fine (runs cat at source time). But ${MLFLOW_EXPERIMENT_ID:-} is written literally — in a new shell where the env var isn't inherited, it expands to empty. Fix: Either use unquoted <<MLENV so the experiment ID is baked in at write time, or write it separately: echo "export MLFLOW_EXPERIMENT_ID=\"${MLFLOW_EXPERIMENT_ID}\"" >> "$HOME/.mlflow-env".
5. Doc says "upstream master" but Containerfile pins to a specific commit
mlflow-tracing-setup.md:91 vs Containerfile.mlflow:13-14
The setup doc Key notes say "the entrypoint replaces the cached plugin with a build from upstream master" but the Containerfile builds from a pinned commit (4953f67) described as v3.14.0. "Upstream master" implies a moving target. Fix: Change "upstream master" to "the v3.14.0 release tag (commit 4953f67)".
6. Deploy command will fail without explaining external manifests dependency
mlflow-tracing-setup.md:55-58 + kustomization.yaml:8
The doc says oc apply -k overlays/mlflow-tracing/ but kustomization.yaml references ../../manifests which doesn't exist in the agentic-starter-kits repo. A user following the setup doc hits a kustomize error immediately. The existing DEPLOYMENT.md walks through cloning the external opencode-openshift repo first — this doc skips that step. Fix: Add a step before oc apply explaining users need to clone opencode-openshift and adjust the resource path.
1. Use --depth 1 --branch v3.14.0 for git clone (fast, reproducible) 2. Make NODE_TLS_REJECT_UNAUTHORIZED opt-in (commented out), add NODE_EXTRA_CA_CERTS as production option 3. Show plugin install stderr (2>&1 instead of 2>/dev/null) 4. Use unquoted heredoc so MLFLOW_EXPERIMENT_ID is baked in at write time 5. Change "upstream master" to "v3.14.0 release tag" in docs 6. Add git clone step and image requirement note before oc apply -k Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Thanks @andrewdonheiser! All 6 items addressed:
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml`:
- Around line 19-25: The production TLS option still points NODE_EXTRA_CA_CERTS
at an outdated service-account CA path, so update the deployment patch to use a
CA bundle that is actually mounted into the pod instead of the deprecated
service-ca.crt location. Adjust the commented guidance in deployment-patch.yaml
so the Option A entry references the real mounted CA file used by the container,
keeping the NODE_TLS_REJECT_UNAUTHORIZED fallback only for dev/test.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Enterprise
Run ID: 937f48f9-5de4-4b61-a1de-4546ac1a5569
📒 Files selected for processing (4)
agents/opencode/deployment/Containerfile.mlflowagents/opencode/deployment/docs/mlflow-tracing-setup.mdagents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yamlagents/opencode/deployment/overlays/mlflow-tracing/entrypoint-patch.yaml
✅ Files skipped from review due to trivial changes (1)
- agents/opencode/deployment/docs/mlflow-tracing-setup.md
🚧 Files skipped from review as they are similar to previous changes (2)
- agents/opencode/deployment/Containerfile.mlflow
- agents/opencode/deployment/overlays/mlflow-tracing/entrypoint-patch.yaml
| # TLS: uncomment ONE of the following for clusters with non-public certs. | ||
| # Option A (production): mount the cluster CA bundle | ||
| # - name: NODE_EXTRA_CA_CERTS | ||
| # value: "/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt" | ||
| # Option B (dev/test only): disable TLS for ALL Node.js connections | ||
| # - name: NODE_TLS_REJECT_UNAUTHORIZED | ||
| # value: "0" |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== Reviewed overlay patch =="
sed -n '1,120p' agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml
echo
echo "== Search overlay + base manifests for CA bundle mounts / cert paths =="
MANIFEST_DIR="$(fd -t d '^manifests$' agents/opencode/deployment | head -n1 || true)"
if [ -n "${MANIFEST_DIR}" ]; then
rg -n -C3 'NODE_EXTRA_CA_CERTS|service-ca\.crt|ca\.crt|volumeMounts:|volumes:' \
agents/opencode/deployment/overlays/mlflow-tracing "${MANIFEST_DIR}"
else
echo "No manifests directory found under agents/opencode/deployment"
fiRepository: red-hat-data-services/agentic-starter-kits
Length of output: 1302
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== File tree around opencode deployment =="
find agents/opencode/deployment -maxdepth 4 -type f | sort
echo
echo "== Search for service-ca.crt / NODE_EXTRA_CA_CERTS / TLS env settings =="
rg -n -C 3 'service-ca\.crt|NODE_EXTRA_CA_CERTS|NODE_TLS_REJECT_UNAUTHORIZED|volumeMounts:|volumes:' agents/opencode/deployment || true
echo
echo "== Inspect likely deployment sources =="
for f in \
agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml \
agents/opencode/deployment/overlays/mlflow-tracing/kustomization.yaml \
agents/opencode/deployment/overlays/mlflow-tracing/entrypoint-patch.yaml
do
if [ -f "$f" ]; then
echo "--- $f ---"
sed -n '1,220p' "$f"
echo
fi
doneRepository: red-hat-data-services/agentic-starter-kits
Length of output: 9226
🌐 Web query:
"OpenShift /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt NODE_EXTRA_CA_CERTS standard mount"
💡 Result:
The file path /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt is deprecated and no longer available in OpenShift pods [1][2]. This path was previously used for the service-serving CA bundle but has been removed in modern OpenShift versions [1][2]. To access the service CA or a custom trust bundle, you must use the following standard OpenShift mechanisms: 1. Service CA: For services using service-serving certificates, annotate a ConfigMap with service.beta.openshift.io/inject-cabundle=true [3][4][5]. The cluster will automatically inject the service CA certificate into the service-ca.crt key of that ConfigMap [3][4]. You should then mount this ConfigMap into your pod as a volume [3][4]. 2. Custom CA Trust: To provide custom CA certificates that your applications should trust, follow these steps [6][7]: - Create a ConfigMap in the openshift-config namespace containing your custom CA certificates [6][8]. - If you require the cluster to use these for internal processes (e.g., pulling images), reference this ConfigMap in the cluster proxy's trustedCA field or the image configuration [6][8][5]. - For individual pods, mount the custom CA ConfigMap as a volume [9][7]. 3. Application Runtime Configuration: To ensure your applications (such as those using Node.js, Go, or Python) recognize these certificates, you must map the mounted certificate file to the appropriate environment variable for your runtime [10][9]. For Node.js, this is often NODE_EXTRA_CA_CERTS [10]. Example: - Mount your CA ConfigMap to a path like /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem [9][7]. - Set the environment variable: - NODE_EXTRA_CA_CERTS=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem [10] - SSL_CERT_FILE=/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem (for many other runtimes) [10][9] Do not attempt to rely on the deprecated /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt path, as it will not be present [1][2].
Citations:
- 1: https://bugzilla.redhat.com/show_bug.cgi?id=1843949
- 2: https://bugzilla.redhat.com/show_bug.cgi?id=1813894
- 3: https://docs.okd.io/latest/security/certificates/service-serving-certificate.html
- 4: https://docs.okd.io/4.22/security/certificates/service-serving-certificate.html
- 5: https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/security_and_compliance/configuring-certificates
- 6: https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/configuring_network_settings/configuring-a-custom-pki
- 7: https://docs.redhat.com/en/documentation/openshift_container_platform/4.17/html/configuring_network_settings/configuring-a-custom-pki
- 8: https://docs.redhat.com/en/documentation/openshift_container_platform/4.19/html/builds_using_buildconfig/setting-up-trusted-ca
- 9: https://docs.redhat.com/en/documentation/red_hat_hardened_images/1-latest/html/build_and_deploy_secure_minimal_containers_with_red_hat_hardened_images/use-custom-ca-certificates
- 10: https://kubernetes.recipes/recipes/security/custom-ca-openshift-kubernetes/
Replace the deprecated CA path. NODE_EXTRA_CA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt won’t work in modern OpenShift pods, so the “production” option is still broken and users may fall back to disabling TLS. Point NODE_EXTRA_CA_CERTS at a CA bundle that’s actually mounted in the pod instead.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@agents/opencode/deployment/overlays/mlflow-tracing/deployment-patch.yaml`
around lines 19 - 25, The production TLS option still points NODE_EXTRA_CA_CERTS
at an outdated service-account CA path, so update the deployment patch to use a
CA bundle that is actually mounted into the pod instead of the deprecated
service-ca.crt location. Adjust the commented guidance in deployment-patch.yaml
so the Option A entry references the real mounted CA file used by the container,
keeping the NODE_TLS_REJECT_UNAUTHORIZED fallback only for dev/test.
Source: Path instructions
Summary
Add MLflow tracing support for OpenCode on RHOAI using the
@mlflow/opencodeplugin. Tested end-to-end in both web and CLI modes with vLLM direct and OGX backends.Addresses RHAIENG-5536, RHAIENG-5537, RHAIENG-5538
Files
agents/opencode/deployment/docs/mlflow-tracing.md— Trace schema, Tetris game comparison across vLLM and OGX, latency benchmarksagents/opencode/deployment/docs/mlflow-tracing-setup.md— Step-by-step setup guideagents/opencode/deployment/Containerfile.mlflow— Extends base OpenCode image with mlflow + plugin build from upstream masteragents/opencode/deployment/overlays/mlflow-tracing/— Kustomize overlay (deployment patch, entrypoint patch)agents/opencode/deployment/docs/screenshots/— Trace screenshots from MLflow UIKey findings
@mlflow/opencodeplugin is the recommended tracing path"plugin"toOPENCODE_CONFIG_CONTENT)Test plan
opencode run "What is 2+2?"→ trace exported to MLflow