From c851380acbaa26ce69ff67bfb4f621fd0a237e3d Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 10 Jul 2026 12:28:00 -0400 Subject: [PATCH 01/25] RHAIENG-5920: add Codex CLI container image for RHOAI Build a validated container image for OpenAI's Codex CLI (Apache 2.0) on Red Hat OpenShift AI, following the base-image-plus-flavor pattern established by OpenCode. Base image (Containerfile.base): - Compiles Codex v0.144.0 from source (Rust) via cross-compilation - UBI 9 minimal runtime with git, jq, tar, nodejs, npm - Includes bwrap (bubblewrap) sandbox binary - Non-root user (UID 1001, GID 0) for restricted-v2 SCC - Published to quay.io/hmoghani/codex-base:0.144.0 RHOAI flavor (Containerfile): - Extends base with entrypoint.sh for runtime configuration - Inference endpoint configurable via OPENAI_BASE_URL env var - MCP server injection via TOML file, mounted ConfigMap, or JSON env var - Git credential injection via GITHUB_PAT - Session persistence via PVC-backed CODEX_HOME - codex-run wrapper script for oc exec convenience Deployment (deployment.yaml): - ImageStream, BuildConfig, ConfigMaps, PVC, Deployment - Recreate strategy for RWO PVC compatibility - Full restricted-v2 SCC security context Known issue: Codex CLI uses the OpenAI Responses API exclusively with namespace tool types. vLLM 0.19.1 (current RHOAI EA build) does not support these tool types, returning "tool type namespace not supported". Direct /v1/responses calls without tools work correctly. Full Codex compatibility requires a newer vLLM version with namespace tool type support. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 664 ++++++++++++++++++ agents/codex/deployment/Containerfile | 45 ++ agents/codex/deployment/Containerfile.base | 146 ++++ .../codex/deployment/Containerfile.openshell | 2 +- agents/codex/deployment/README.md | 58 +- agents/codex/deployment/deployment.yaml | 222 ++++++ agents/codex/deployment/entrypoint.sh | 343 +++++++++ 7 files changed, 1460 insertions(+), 20 deletions(-) create mode 100644 agents/codex/README.md create mode 100644 agents/codex/deployment/Containerfile create mode 100644 agents/codex/deployment/Containerfile.base create mode 100644 agents/codex/deployment/deployment.yaml create mode 100644 agents/codex/deployment/entrypoint.sh diff --git a/agents/codex/README.md b/agents/codex/README.md new file mode 100644 index 00000000..401596e8 --- /dev/null +++ b/agents/codex/README.md @@ -0,0 +1,664 @@ +# Codex on OpenShift + +A guide to deploying [OpenAI Codex CLI](https://github.com/openai/codex) as a containerized coding agent on Red Hat OpenShift. Connects to a vLLM inference endpoint via the OpenAI Responses API (`/v1/responses`). + +## Table of Contents + +- [Licensing](#licensing) +- [Prerequisites](#prerequisites) +- [Step 1: Build the Container Image](#step-1-build-the-container-image) +- [Step 2: Create an OpenShift Project](#step-2-create-an-openshift-project) +- [Step 3: Deploy](#step-3-deploy) +- [Local Test (Optional)](#local-test-optional) +- [Using Codex](#using-codex) + - [Interactive Mode](#interactive-mode) + - [Headless Execution](#headless-execution) + - [Retrieving Files from the Container](#retrieving-files-from-the-container) +- [Configuration](#configuration) + - [Session Persistence](#session-persistence) + - [config.toml Overrides](#configtoml-overrides) + - [MCP Servers](#mcp-servers) + - [Extending the Container Image](#extending-the-container-image) +- [Security Considerations](#security-considerations) +- [Troubleshooting](#troubleshooting) +- [Cleanup](#cleanup) +- [Known Limitations](#known-limitations) + +--- + +## Licensing + +Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/codex/blob/main/LICENSE). The Containerfiles in this directory and the resulting images are freely redistributable. No proprietary components are included. + +--- + +## Prerequisites + +- `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) +- `oc` CLI installed and logged into your OpenShift cluster +- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` (vLLM v0.10.0+) + +Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses`. See [Responses API Compatibility](#responses-api-compatibility) in Troubleshooting if you encounter errors. + +All shell commands in this guide assume you are in the `deployment/` subdirectory, where the Containerfile and deployment manifests live: + +```bash +cd deployment/ +``` + +--- + +## Step 1: Build the Container Image + +Codex uses a two-stage image build: a **base image** containing the Codex Rust binary on UBI 9 minimal, and a **flavor image** that extends it with the RHOAI entrypoint, MCP injection, and session persistence. + +### Build the base image + +The base image uses a multi-stage Dockerfile. The builder stage compiles Codex from the upstream Apache 2.0 source using the official Rust toolchain, producing an auditable binary with no opaque pre-built artifacts. Only the compiled binary is copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js 16 and npm from UBI repos for npx-based MCP servers. + +```bash +podman build --platform linux/amd64 \ + -t codex-base:latest \ + -f Containerfile.base . + +# Build with a specific version (use rust-v* git tags) +podman build --platform linux/amd64 \ + --build-arg CODEX_VERSION=rust-v0.144.0 \ + -t codex-base:0.144.0 \ + -f Containerfile.base . +``` + +### Build the RHOAI flavor image + +```bash +podman build --platform linux/amd64 \ + -t codex:latest \ + -f Containerfile \ + --build-arg BASE_IMAGE=codex-base:latest . +``` + +### Version pinning + +The `CODEX_VERSION` build arg controls which git tag to build from. It defaults to `rust-v0.144.0`. Pin it explicitly for reproducible builds: + +```bash +podman build --platform linux/amd64 \ + --build-arg CODEX_VERSION=rust-v0.144.0 \ + -t codex-base:0.144.0 \ + -f Containerfile.base . +``` + +Available tags are listed at [github.com/openai/codex/releases](https://github.com/openai/codex/releases). Use the `rust-v*` tags. + +### Image sizes + +| Image | Approximate Size | Contents | +|-------|-----------------|----------| +| `codex-base` | ~579 MB | UBI 9 minimal + Codex Rust binary (compiled from source) + git, jq, tar, nodejs, npm | +| `codex` (flavor) | ~579 MB | Base + entrypoint.sh, codex-run wrapper | + +The flavor image adds only shell scripts, so its size is effectively the same as the base. The base image build takes ~12 minutes because it cross-compiles Codex from Rust source. Cross-compiling on Apple Silicon requires 32GB+ RAM allocated to the podman VM (`podman machine set --memory 32768`). Native x86_64 builds (e.g., on the cluster via `oc start-build`) do not have this constraint. + +### Build with `--platform linux/amd64` + +Always pass `--platform linux/amd64` when building on Apple Silicon. The builder stage cross-compiles the Codex Rust binary for `x86_64-unknown-linux-gnu`, which is the architecture used by OpenShift nodes. + +--- + +## Step 2: Create an OpenShift Project + +```bash +oc new-project my-codex-project +``` + +--- + +## Step 3: Deploy + +### Configure the vLLM endpoint + +Edit `deployment.yaml` and update the placeholder values in the Deployment env section: + +| Variable | Purpose | Example | +|----------|---------|---------| +| `OPENAI_BASE_URL` | vLLM server base URL (must serve `/v1/responses`) | `http://vllm-svc.namespace.svc.cluster.local/v1` | +| `OPENAI_API_KEY` | API key for the vLLM endpoint (use `"dummy"` if none required) | `dummy` | +| `OPENAI_MODEL` | Model ID as reported by `GET /v1/models` | `your-model-id` | + +The entrypoint automatically generates a `[model_providers.vllm]` block in `config.toml` from these environment variables. You do not need to write TOML configuration manually for the basic case. + +### Apply manifests and build + +The deployment manifest contains all required resources: ImageStream, BuildConfig, ConfigMaps (config.toml and MCP), a 1Gi PVC, and the Deployment. + +```bash +# Apply all resources (ImageStream, BuildConfig, ConfigMaps, PVC, Deployment) +oc apply -f deployment.yaml + +# Option A: Build on the cluster (requires internet for docker.io/rust:1-bookworm) +# +# Create a BuildConfig for the base image +oc new-build --strategy=docker --binary --name=codex-base +oc start-build codex-base --from-dir=. --follow +# +# Then build the RHOAI flavor +oc start-build codex --from-dir=. --follow + +# Option B: Build locally and push (recommended — avoids Docker Hub rate limits) +# +# Build locally (see Step 1), then push to the internal registry: +REGISTRY=$(oc get route -n openshift-image-registry image-registry -o jsonpath='{.spec.host}') +podman login "${REGISTRY}" -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false +podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" +podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false +# +# Import into the ImageStream +oc tag --source=docker \ + image-registry.openshift-image-registry.svc:5000/$(oc project -q)/codex:latest \ + codex:latest + +# Wait for rollout +oc rollout status deployment/codex +``` + +### Verify the deployment + +```bash +# Check the pod is running and the entrypoint configured correctly +oc exec deployment/codex -- bash -c 'codex --version && cat ~/.codex/config.toml' + +# Test connectivity to the vLLM endpoint +oc exec deployment/codex -- bash -c \ + 'curl -s http://vllm-svc.your-namespace.svc.cluster.local/v1/models | jq .' +``` + +--- + +## Local Test (Optional) + +Test the image locally with podman before deploying to OpenShift: + +```bash +# Headless execution +podman run --rm \ + -e OPENAI_BASE_URL="http://YOUR_VLLM_ENDPOINT/v1" \ + -e OPENAI_API_KEY="dummy" \ + -e OPENAI_MODEL="YOUR_MODEL_ID" \ + codex:latest \ + codex exec "What is 2+2?" + +# Interactive mode +podman run -it --rm \ + -e OPENAI_BASE_URL="http://YOUR_VLLM_ENDPOINT/v1" \ + -e OPENAI_API_KEY="dummy" \ + -e OPENAI_MODEL="YOUR_MODEL_ID" \ + -v $(pwd):/workspace/projects:z \ + codex:latest \ + codex +``` + +For interactive mode, both `-i` (stdin) and `-t` (tty) flags are required. + +--- + +## Using Codex + +The deployment runs `sleep infinity` so the pod stays alive for `oc exec` sessions. All examples below use `codex` as the deployment name. + +### The codex-run Wrapper + +The entrypoint generates a `codex-run` wrapper script at `~/.codex/codex-run` that includes all container-configured arguments (model, sandbox mode, provider config). Use it for all `oc exec` sessions: + +```bash +oc exec deployment/codex -- bash -c ' + codex-run exec "Your prompt here" +' +``` + +You can also source the environment directly: + +```bash +oc exec deployment/codex -- bash -c ' + source ~/.codex/env.sh + codex ${CODEX_EXTRA_ARGS} exec "Your prompt here" +' +``` + +### Interactive Mode + +For multi-turn conversations with a TTY: + +```bash +oc exec -it deployment/codex -- bash -c ' + codex-run +' +``` + +Interactive mode requires the `-it` flags on `oc exec`. Without them, the TUI will not render and Codex will exit immediately. + +### Headless Execution + +The `codex exec` subcommand runs a prompt non-interactively and exits when complete. This is the recommended mode for scripted or automated usage: + +```bash +# Single prompt +oc exec deployment/codex -- bash -c ' + codex-run exec "Explain the structure of this project" +' + +# With a specific working directory +oc exec deployment/codex -- bash -c ' + cd /workspace/projects/my-repo && codex-run exec "Run the tests and fix any failures" +' +``` + +`codex exec` requires `CODEX_API_KEY` to be set. The entrypoint automatically propagates `OPENAI_API_KEY` to `CODEX_API_KEY` if only the former is set. + +### Retrieving Files from the Container + +When Codex generates files (reports, patches, code) that you want locally without pushing to Git, use `oc cp`: + +```bash +# Copy a single file +oc cp :/workspace/projects/report.md ./report.md + +# Copy a directory +oc cp :/workspace/projects/output/ ./output/ + +# Find the pod name +oc get pods -l app=codex -o name + +# List files in the container +oc exec deployment/codex -- ls -la /workspace/projects/ +``` + +### Rebuilding the Image + +After modifying the Containerfile or entrypoint, rebuild locally and push: + +```bash +podman build --platform linux/amd64 -t codex:latest -f Containerfile \ + --build-arg BASE_IMAGE=codex-base:latest . + +REGISTRY=$(oc get route -n openshift-image-registry image-registry -o jsonpath='{.spec.host}') +podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" +podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false + +oc import-image codex:latest --confirm +oc rollout restart deployment/codex +oc rollout status deployment/codex +``` + +--- + +## Configuration + +### Session Persistence + +Session history and configuration persist across pod restarts via the workspace PVC. The `CODEX_HOME` environment variable defaults to `/workspace/.codex`. + +**Directory structure:** + +```text +/workspace/ <- PVC mount (persistent) +|-- .codex/ <- Codex config (CODEX_HOME) +| |-- config.toml <- Model provider, MCP, preferences +| |-- env.sh <- Generated env for oc exec sessions +| |-- codex-run <- Wrapper script +| +-- instructions.md <- Persisted project instructions ++-- projects/ <- WORKDIR (where users run Codex) +``` + +The entrypoint creates a symlink `~/.codex` -> `/workspace/.codex` so that standard `~/.codex/` paths work as expected. + +**What persists:** + +| Data | Location | Persisted? | +|------|----------|------------| +| Configuration | `/workspace/.codex/config.toml` | Yes | +| Project instructions | `/workspace/.codex/instructions.md` | Yes | +| Working files | `/workspace/projects/` | Yes | +| Codex CLI args | `/workspace/.codex/env.sh` | Regenerated on each restart | +| Wrapper script | `/workspace/.codex/codex-run` | Regenerated on each restart | + +**Disabling persistence:** Set `CODEX_HOME` to a non-PVC path: + +```yaml +env: + - name: CODEX_HOME + value: "/tmp/.codex" +``` + +### config.toml Overrides + +Codex CLI uses TOML configuration at `$CODEX_HOME/config.toml`. The deployment manifest includes a ConfigMap (`codex-config`) that is staged at `/etc/codex-config/config.toml`. On first start, the entrypoint copies it to `${CODEX_HOME}/config.toml` on the writable PVC. On subsequent restarts, the existing file is preserved so that runtime changes survive. + +The entrypoint auto-generates model provider configuration from the `OPENAI_BASE_URL` and `OPENAI_MODEL` environment variables. Use the ConfigMap only for settings beyond what the entrypoint generates. + +**Generated config.toml example** (produced by entrypoint.sh when `OPENAI_BASE_URL` and `OPENAI_MODEL` are set): + +```toml +# Auto-configured by entrypoint.sh +model = "your-model-id" +model_provider = "vllm" + +[model_providers.vllm] +name = "vLLM" +base_url = "http://vllm-svc.namespace.svc.cluster.local/v1" +env_key = "OPENAI_API_KEY" +wire_api = "responses" +``` + +**Manual config.toml example** (for advanced setups, place in the ConfigMap): + +```toml +model = "granite-3.3-8b-instruct" +model_provider = "my-provider" + +[model_providers.my-provider] +name = "Custom Provider" +base_url = "https://inference.example.com/v1" +env_key = "OPENAI_API_KEY" +wire_api = "responses" +``` + +To update the ConfigMap after deployment: + +```bash +oc patch configmap codex-config -p '{ + "data": { + "config.toml": "model = \"my-model\"\nmodel_provider = \"vllm\"\n\n[model_providers.vllm]\nname = \"vLLM\"\nbase_url = \"http://my-vllm-endpoint/v1\"\nenv_key = \"OPENAI_API_KEY\"\nwire_api = \"responses\"\n" + } +}' +``` + +To force a config reset (pick up ConfigMap changes), delete the file from the PVC before restarting: + +```bash +oc exec deployment/codex -- rm /workspace/.codex/config.toml +oc rollout restart deployment/codex +``` + +### MCP Servers + +MCP (Model Context Protocol) servers extend Codex with additional tools. Codex uses TOML-native MCP configuration with `[mcp_servers.*]` blocks in `config.toml`. The entrypoint supports three injection methods: + +#### Method 1: Mounted ConfigMap (recommended) + +The deployment manifest mounts a ConfigMap at `/etc/codex-mcp/mcp-servers.toml`. The entrypoint appends its contents to `config.toml` at startup if the file contains `[mcp_servers.*]` blocks. + +**TOML format:** + +```toml +[mcp_servers.github] +url = "https://api.githubcopilot.com/mcp/" + +[mcp_servers.local-tool] +command = "npx" +args = ["-y", "@some/mcp-server"] +``` + +Update the ConfigMap: + +```bash +oc patch configmap codex-mcp-config -p '{ + "data": { + "mcp-servers.toml": "[mcp_servers.my-api]\nurl = \"https://mcp.example.com/v1\"\n" + } +}' +oc rollout restart deployment/codex +``` + +#### Method 2: TOML file path (MCP_CONFIG_TOML) + +Point to a TOML file containing `[mcp_servers.*]` blocks. The entrypoint appends its contents to `config.toml`. + +```yaml +env: + - name: MCP_CONFIG_TOML + value: "/etc/codex-mcp/mcp-servers.toml" +``` + +This is useful when you have a TOML file mounted from a Secret or other volume. + +#### Method 3: Inline JSON (MCP_CONFIG_JSON) + +For compatibility with the Claude Code `MCP_CONFIG_JSON` pattern, the entrypoint accepts inline JSON and converts it to TOML automatically using `jq`: + +```yaml +env: + - name: MCP_CONFIG_JSON + value: '{"mcpServers":{"my-api":{"url":"https://mcp.example.com/v1"},"local-tool":{"command":"npx","args":["-y","@some/mcp-server"]}}}' +``` + +The JSON-to-TOML conversion supports `url`, `command`, and `args` fields. For MCP servers requiring environment variable references (e.g., `bearer_token_env_var`), use Method 1 or 2 with native TOML instead. + +#### MCP server types + +| Type | TOML field | Use case | Requirements | +|------|-----------|----------|--------------| +| Remote HTTP | `url` | Remote MCP servers | Network access to endpoint | +| Local process | `command` + `args` | Process-based MCP servers | Executable must exist in container | + +The base image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, and `npm`. Command-based MCP servers using `npx` work out of the box. For servers requiring other runtimes, see [Extending the Container Image](#extending-the-container-image). + +### Extending the Container Image + +The base image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, `npm`, and `bash`. For real coding workflows, add language runtimes so the agent can run tests, lint, and build. Including the runtimes your project uses significantly improves agent output quality since the agent can run tests and catch its own mistakes. + +```dockerfile +# In the "System dependencies" section of Containerfile.base +RUN microdnf install -y --nodocs \ + git \ + jq \ + tar \ + nodejs \ + npm \ + # Build tools + make \ + gcc \ + diffutils \ + which \ + # Python + python3.12 \ + python3.12-pip \ + # Go + golang \ + # Java + java-21-openjdk-devel \ + maven \ + && microdnf clean all \ + && rm -rf /var/cache/yum +``` + +The GitHub CLI (`gh`) is not available in UBI repos and must be installed separately: + +```dockerfile +# Install GitHub CLI +ARG GH_VERSION=2.74.1 +RUN curl -fsSL "https://github.com/cli/cli/releases/download/v${GH_VERSION}/gh_${GH_VERSION}_linux_amd64.tar.gz" \ + | tar -xz -C /usr/local/bin --strip-components=2 "gh_${GH_VERSION}_linux_amd64/bin/gh" +``` + +--- + +## Security Considerations + +### Sandbox Mode + +The deployment manifest sets `CODEX_SANDBOX=danger-full-access` by default, passing `--sandbox danger-full-access` to the Codex CLI. This disables Codex's built-in sandboxing (which uses bubblewrap on Linux). + +**Why it is set this way:** + +- The container itself is the isolation boundary, running as non-root (UID 1001) under OpenShift's restricted-v2 SCC +- The pod security context drops all capabilities and enforces `allowPrivilegeEscalation: false` with a `RuntimeDefault` seccomp profile +- Codex only has access to the isolated `/workspace` PVC, not host filesystems +- The bubblewrap sandbox inside a container-in-a-container creates unnecessary complexity and can conflict with OpenShift security policies + +**Tradeoffs:** Running in an isolated container removes much of the reason you would normally keep Codex's built-in sandbox enabled. However, `danger-full-access` also means the agent has unrestricted access to external services it can reach via the network (e.g., Git remotes, MCP endpoints). Rely on repository safeguards (branch protection, required reviews, limited PAT scopes) to mitigate risks from automated operations. + +To use a different sandbox mode: + +```yaml +env: + - name: CODEX_SANDBOX + value: "read-only-fs" +``` + +### Credential Isolation + +When credentials such as GitHub PATs or API keys are passed as environment variables, the agent can read them directly. This means the agent could inadvertently expose credentials in conversation output, commit messages, or through MCP server calls. + +For many use cases, this risk is acceptable when combined with standard safeguards: limiting PAT scope to the minimum required permissions, protecting branches, and requiring PR reviews. For environments with strict security requirements, consider evaluating container isolation solutions that separate the agent from the credentials it depends on. + +### Repository Safeguards + +When an AI agent has push access to a Git repository, ensure the repository has protections against mistakes: + +- **Branch protection rules**: Protect your main/default branch so direct pushes are blocked +- **Required pull request reviews**: Require at least one approval before merging +- **Limit PAT scope**: Grant only the minimum permissions needed +- **Status checks**: Require CI checks to pass before a PR can be merged + +--- + +## Troubleshooting + +### Responses API Compatibility + +Codex CLI uses the OpenAI Responses API (`/v1/responses`), not the Chat Completions API (`/v1/chat/completions`). If you see errors like "404 Not Found" or "endpoint not found", verify your vLLM server supports the Responses API: + +```bash +# Check if /v1/responses endpoint exists +curl -s -o /dev/null -w "%{http_code}" \ + -X POST "http://YOUR_VLLM_ENDPOINT/v1/responses" \ + -H "Content-Type: application/json" \ + -d '{ + "model": "YOUR_MODEL_ID", + "input": "What is 2+2?" + }' +``` + +The Responses API requires vLLM v0.10.0 or later. Full tool calling support (including namespace tool types used by Codex) may require newer versions — check [vLLM releases](https://github.com/vllm-project/vllm/releases) for the latest compatibility. + +### Network Connectivity + +If the vLLM server is outside the cluster (e.g., on EC2 or another cloud), verify the pod can reach it: + +```bash +# Find the cluster's egress IP +oc exec deployment/codex -- curl -s ifconfig.me + +# Test connectivity to the vLLM server +oc exec deployment/codex -- curl -s -o /dev/null -w "%{http_code}" \ + http://YOUR_HOST:8000/v1/models +``` + +Common causes of connection failure include security group rules, network policies, and firewalls. If the server uses IP-based access rules, add the cluster egress IP to the allow list. + +### Testing Endpoints Directly + +Verify the vLLM server is reachable and serving the expected model: + +```bash +# Health check +curl -s "http://YOUR_VLLM_ENDPOINT/health" + +# Model listing +curl -s "http://YOUR_VLLM_ENDPOINT/v1/models" + +# Non-streaming Responses API request +curl -s -X POST "http://YOUR_VLLM_ENDPOINT/v1/responses" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer dummy" \ + -d '{ + "model": "YOUR_MODEL_ID", + "input": "What is 2+2? Reply with just the number." + }' + +# Streaming Responses API request +curl -s -X POST "http://YOUR_VLLM_ENDPOINT/v1/responses" \ + -H "Content-Type: application/json" \ + -H "Authorization: Bearer dummy" \ + -d '{ + "model": "YOUR_MODEL_ID", + "input": "Say hello", + "stream": true + }' +``` + +### Config Debugging + +To inspect the generated `config.toml` and verify model provider and MCP configuration: + +```bash +# View the generated config +oc exec deployment/codex -- cat /workspace/.codex/config.toml + +# View entrypoint logs +oc logs deployment/codex | grep "\[entrypoint\]" + +# Check env.sh (generated CLI args and environment) +oc exec deployment/codex -- cat ~/.codex/env.sh + +# Verify Codex version +oc exec deployment/codex -- codex --version +``` + +### CODEX_API_KEY Missing + +`codex exec` (headless mode) requires `CODEX_API_KEY` to be set. The entrypoint automatically propagates `OPENAI_API_KEY` to `CODEX_API_KEY` if only the former is set. If you see authentication errors in headless mode, verify both are set: + +```bash +oc exec deployment/codex -- bash -c 'echo "OPENAI_API_KEY=${OPENAI_API_KEY:+set}" && echo "CODEX_API_KEY=${CODEX_API_KEY:+set}"' +``` + +### Pod Restart Loop + +If the pod enters a CrashLoopBackOff, check the entrypoint logs: + +```bash +oc logs deployment/codex --previous +``` + +Common causes: +- `CODEX_HOME` directory not writable (PVC not mounted or permissions issue) +- Invalid TOML in ConfigMap (syntax error in config.toml or mcp-servers.toml) +- Missing base image (flavor Containerfile references a base image that has not been built) + +--- + +## Cleanup + +```bash +oc delete deployment codex +oc delete buildconfig codex codex-base 2>/dev/null +oc delete imagestream codex codex-base 2>/dev/null +oc delete configmap codex-config codex-mcp-config +oc delete pvc codex-workspace +oc delete project my-codex-project +``` + +--- + +## Known Limitations + +### Responses API Only + +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Your vLLM server must be v0.10.0+ with Responses API support enabled. Full tool calling support may require newer versions. + +### Open Source Model Quality + +Codex CLI is designed for OpenAI's models. Open source models may produce lower quality results, particularly for complex multi-step tasks and tool use chains. Including language runtimes in the container helps because the agent can run tests and iterate on its own output. + +### No Built-in Permission System + +Unlike Claude Code, Codex CLI does not have a granular permission system for file operations or shell commands. The `--sandbox` flag controls the sandbox mode, but `danger-full-access` grants unrestricted access within the container. Rely on OpenShift's container isolation and repository safeguards for security. + +### TOML Configuration Only + +Codex CLI uses TOML for all configuration (`config.toml`). There is no JSON settings file equivalent. The entrypoint's `MCP_CONFIG_JSON` support converts JSON to TOML at startup for compatibility, but only a subset of fields (`url`, `command`, `args`) are converted. For full MCP server configuration (e.g., `bearer_token_env_var`, `env` blocks), use native TOML. + +### OpenShell Sandbox Variant + +The `Containerfile.openshell` builds a separate image for the OpenShell sandbox environment. This variant uses a different base image (`openshell-base`), installs Codex via npm (not the extracted Rust binary), and does not include the RHOAI entrypoint or session persistence. It is intended for ephemeral sandbox sessions, not persistent OpenShift deployments. diff --git a/agents/codex/deployment/Containerfile b/agents/codex/deployment/Containerfile new file mode 100644 index 00000000..6504e1a3 --- /dev/null +++ b/agents/codex/deployment/Containerfile @@ -0,0 +1,45 @@ +# Codex CLI — RHOAI Flavor +# +# Extends the Codex base image with RHOAI-specific configuration: +# - Entrypoint with env-var-driven setup (auth, model provider, MCP, git) +# - codex-run wrapper for oc exec convenience +# - Session persistence via PVC-backed CODEX_HOME +# +# Build the base image first, then this flavor: +# podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . +# podman build --platform linux/amd64 -t codex:latest -f Containerfile \ +# --build-arg BASE_IMAGE=codex-base:latest . +# +# Features: +# - Configurable inference endpoint (vLLM, OpenAI, OpenRouter) +# - MCP server configuration injectable at runtime via environment variables: +# MCP_CONFIG_TOML - path to a TOML file with [mcp_servers.*] tables +# MCP_CONFIG_JSON - inline JSON string with MCP server definitions +# - Git credential injection via GITHUB_PAT +# - Non-root user compatible with restricted-v2 SCC + +ARG BASE_IMAGE=quay.io/hmoghani/codex-base:latest +FROM ${BASE_IMAGE} + +LABEL name="codex" \ + vendor="Red Hat" \ + summary="Codex CLI agent for OpenShift AI" \ + description="RHOAI-flavored Codex CLI with entrypoint, MCP injection, and session persistence" \ + license="Apache-2.0" + +USER 0 + +# Remove ~/.codex created during base build — entrypoint will create a symlink +# to /workspace/.codex for PVC-backed persistence +RUN rm -rf /home/codex-agent/.codex + +COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint.sh + +USER 1001 +WORKDIR /workspace/projects + +ENTRYPOINT ["entrypoint.sh"] + +# Default: non-interactive exec mode +# Override at runtime for interactive use or sleep infinity for oc exec +CMD ["codex", "exec", "echo hello"] diff --git a/agents/codex/deployment/Containerfile.base b/agents/codex/deployment/Containerfile.base new file mode 100644 index 00000000..6c2131da --- /dev/null +++ b/agents/codex/deployment/Containerfile.base @@ -0,0 +1,146 @@ +# Codex CLI Base Image +# +# Multi-stage build that compiles Codex CLI from source (Apache 2.0) +# and produces a clean UBI 9 minimal runtime image. Extended by flavor +# images (Containerfile, Containerfile.mlflow, etc.). +# +# The builder uses cross-compilation so it runs natively on the host +# (e.g., ARM64 on Apple Silicon) while producing an x86_64 binary. +# This avoids QEMU emulation which crashes rustc with SIGSEGV. +# Cross-compiling requires 32GB+ RAM in the podman VM. +# +# Build: +# podman build --platform linux/amd64 \ +# -t codex-base:latest \ +# -f Containerfile.base . +# +# Build with specific version (git tag): +# podman build --platform linux/amd64 \ +# --build-arg CODEX_VERSION=rust-v0.144.0 \ +# -t codex-base:0.144.0 \ +# -f Containerfile.base . +# +# License: Apache-2.0 (Codex CLI), Apache-2.0 (this file) + +# ============================================================================= +# Stage 1: Builder — cross-compile Codex from source +# ============================================================================= +# $BUILDPLATFORM ensures this stage runs natively (no QEMU). +# We cross-compile to the $TARGETPLATFORM using Rust's --target flag. + +FROM --platform=$BUILDPLATFORM docker.io/library/rust:1-bookworm AS builder + +ARG CODEX_VERSION=rust-v0.144.0 +ARG TARGETARCH + +# Cross-compilation environment for C/C++ dependencies (openssl, v8, lzma, etc.) +ENV CC_x86_64_unknown_linux_gnu=x86_64-linux-gnu-gcc \ + CXX_x86_64_unknown_linux_gnu=x86_64-linux-gnu-g++ \ + CARGO_TARGET_X86_64_UNKNOWN_LINUX_GNU_LINKER=x86_64-linux-gnu-gcc \ + PKG_CONFIG_ALLOW_CROSS=1 \ + PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig \ + OPENSSL_DIR=/usr \ + OPENSSL_INCLUDE_DIR=/usr/include/x86_64-linux-gnu \ + OPENSSL_LIB_DIR=/usr/lib/x86_64-linux-gnu + +# Install system build deps + cross-compilation toolchain +# Enable amd64 as a foreign architecture for cross-compilation headers +RUN dpkg --add-architecture amd64 \ + && apt-get update && apt-get install -y --no-install-recommends \ + git pkg-config make cmake \ + libssl-dev \ + # x86_64 cross-compilation toolchain + gcc-x86-64-linux-gnu g++-x86-64-linux-gnu \ + libc6-dev-amd64-cross \ + # x86_64 headers for cross-compiling native dependencies + libssl-dev:amd64 \ + liblzma-dev:amd64 \ + libcap-dev:amd64 \ + && rm -rf /var/lib/apt/lists/* + +# Clone source first so we can read its rust-toolchain.toml +RUN git clone --depth 1 --branch "${CODEX_VERSION}" \ + https://github.com/openai/codex.git /src + +# The repo pins a specific Rust version via rust-toolchain.toml (e.g., 1.95.0). +# cargo auto-installs this toolchain but only for the native arch. +# We must add the x86_64 target to the PROJECT's pinned toolchain, not the +# image default, otherwise cargo can't find the x86_64 standard library. +RUN cd /src/codex-rs \ + && PINNED=$(sed -n 's/^channel *= *"\(.*\)"/\1/p' rust-toolchain.toml) \ + && echo "Project pins Rust ${PINNED}" \ + && rustup toolchain install "${PINNED}" \ + && rustup target add x86_64-unknown-linux-gnu --toolchain "${PINNED}" \ + && mkdir -p /root/.cargo \ + && printf '[target.x86_64-unknown-linux-gnu]\nlinker = "x86_64-linux-gnu-gcc"\n' \ + >> /root/.cargo/config.toml + +WORKDIR /src/codex-rs + +# Build +# The upstream release profile uses thin LTO which requires 20GB+ RAM +# during cross-compilation linking. Disable it for cross builds — +# the binary is functionally identical, just slightly larger. +RUN CARGO_PROFILE_RELEASE_LTO=false \ + cargo build --release --target x86_64-unknown-linux-gnu --bin codex --bin bwrap \ + && cp target/x86_64-unknown-linux-gnu/release/codex /opt/codex \ + && cp target/x86_64-unknown-linux-gnu/release/bwrap /opt/bwrap \ + && chmod +x /opt/codex /opt/bwrap \ + && rm -rf target /src /usr/local/cargo/registry + +# ============================================================================= +# Stage 2: Runtime — clean UBI 9 minimal +# ============================================================================= + +FROM registry.access.redhat.com/ubi9/ubi-minimal@sha256:8201445bebcb5bd4fe23fcc2a76cd5fec029ab401d270926a1563c03b36f0137 + +ARG CODEX_VERSION=rust-v0.144.0 +ARG USER_NAME=codex-agent +ARG USER_UID=1001 +ARG USER_GID=0 + +LABEL name="codex-base" \ + version="${CODEX_VERSION}" \ + vendor="Red Hat" \ + summary="Codex CLI base image for OpenShift AI" \ + description="OpenAI Codex CLI (Apache 2.0) compiled from source on UBI 9 minimal" \ + license="Apache-2.0" \ + io.openshift.tags="codex,agent,cli" \ + io.k8s.description="Codex CLI coding agent base image" + +# System dependencies +# - git: Version control (agent clones repos, makes commits) +# - jq: JSON processor (config manipulation) +# - tar: Required by oc cp for file retrieval +# - nodejs + npm: Runtime for npx-based MCP servers +# Note: curl-minimal is pre-installed in ubi-minimal; full curl conflicts +RUN microdnf install -y --nodocs \ + git \ + jq \ + tar \ + nodejs \ + npm \ + && microdnf clean all \ + && rm -rf /var/cache/yum + +# Copy the compiled binaries from the builder +COPY --from=builder /opt/codex /usr/local/bin/codex +COPY --from=builder /opt/bwrap /usr/local/bin/bwrap + +# Create non-root user (OpenShift restricted-v2 SCC compatible) +RUN useradd -u ${USER_UID} -g ${USER_GID} -d /home/${USER_NAME} -m -s /bin/bash ${USER_NAME} + +# Workspace and config directories +RUN mkdir -p /workspace/projects /home/${USER_NAME}/.codex \ + && chown -R ${USER_UID}:${USER_GID} /workspace /home/${USER_NAME} \ + && chmod -R 775 /workspace /home/${USER_NAME} + +ENV HOME="/home/${USER_NAME}" \ + PATH="/home/${USER_NAME}/.local/bin:${PATH}" + +# Verify +RUN codex --version + +USER ${USER_UID} +WORKDIR /workspace/projects +ENTRYPOINT ["codex"] diff --git a/agents/codex/deployment/Containerfile.openshell b/agents/codex/deployment/Containerfile.openshell index ac15ce0f..a315ba10 100644 --- a/agents/codex/deployment/Containerfile.openshell +++ b/agents/codex/deployment/Containerfile.openshell @@ -16,7 +16,7 @@ RUN microdnf install -y --nodocs nodejs npm && \ rm -rf /var/cache/yum # Install Codex CLI (Apache 2.0) -ARG CODEX_VERSION=0.139.0 +ARG CODEX_VERSION=0.144.0 RUN npm install -g @openai/codex@${CODEX_VERSION} # Verify diff --git a/agents/codex/deployment/README.md b/agents/codex/deployment/README.md index a5153796..51a7bdae 100644 --- a/agents/codex/deployment/README.md +++ b/agents/codex/deployment/README.md @@ -1,29 +1,49 @@ -# Codex — OpenShell Sandbox Deployment +# Codex Deployment Files -This directory contains an OpenShell-compatible Containerfile for running [OpenAI Codex CLI](https://github.com/openai/codex) inside an OpenShell sandbox. +This directory contains Containerfiles, entrypoint, and deployment manifests for running [OpenAI Codex CLI](https://github.com/openai/codex) on OpenShift. -## Prerequisites +For the full deployment guide, see the [Codex on OpenShift README](../README.md). -- [Podman](https://podman.io/) or Docker installed -- [OpenShell CLI](https://github.com/NVIDIA/OpenShell-Community) installed -- An OpenShell gateway running +## Files -## Build and Run +| File | Description | +|------|-------------| +| `Containerfile.base` | Base image build (multi-stage: compiles Codex from source, UBI 9 minimal runtime) | +| `Containerfile` | RHOAI flavor (extends base with entrypoint, MCP injection, session persistence) | +| `Containerfile.openshell` | OpenShell sandbox variant (extends openshell-base) | +| `entrypoint.sh` | Container entrypoint (auth, model provider, MCP, git credential setup) | +| `deployment.yaml` | OpenShift manifests (ImageStream, BuildConfig, ConfigMaps, PVC, Deployment) | -```bash -podman build --platform linux/amd64 -t codex-sandbox:latest -f Containerfile.openshell . -openshell sandbox create --from codex-sandbox:latest -e OPENAI_API_KEY=sk-... -``` +## Quick Reference -## What `Containerfile.openshell` does +| Setting | Environment Variable | Default | +|---------|---------------------|---------| +| API key | `OPENAI_API_KEY` | (required) | +| Inference endpoint | `OPENAI_BASE_URL` | (required for vLLM) | +| Model | `OPENAI_MODEL` | (provider default) | +| Config directory | `CODEX_HOME` | `/workspace/.codex` | +| Sandbox mode | `CODEX_SANDBOX` | `danger-full-access` | +| MCP config (TOML file) | `MCP_CONFIG_TOML` | (none) | +| MCP config (inline JSON) | `MCP_CONFIG_JSON` | (none) | +| GitHub PAT | `GITHUB_PAT` | (none) | -Builds on the shared base image (`quay.io/hmoghani/openshell-base`) which provides the `sandbox` user, system packages, and OpenShell entrypoint. This flavor adds: +## Build Order -- Node.js and npm (from UBI repos) -- Codex CLI via npm (version pinned, Apache 2.0) +```bash +# 1. Build base image +podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . + +# 2. Build RHOAI flavor +podman build --platform linux/amd64 -t codex:latest -f Containerfile \ + --build-arg BASE_IMAGE=codex-base:latest . +``` -## Notes +## Comparison: RHOAI vs OpenShell -- OpenShell's supervisor takes over as PID 1 and does not automatically run the Codex CLI. Start it manually inside the sandbox. -- Build with `--platform linux/amd64` when targeting x86_64 clusters from Apple Silicon machines. -- Tested on OpenShell v0.0.58, OpenShift 4.21 (June 2026). Codex CLI version 0.139.0. +| | RHOAI (Containerfile) | OpenShell (Containerfile.openshell) | +|--|----------------------|-------------------------------------| +| **Base** | UBI 9 minimal (multi-stage) | openshell-base | +| **Codex install** | Compiled from source (Rust) | npm install in image | +| **Runtime** | Standalone pod, `oc exec` | OpenShell gateway sandbox | +| **Auth** | Env vars, entrypoint setup | Manual inside sandbox | +| **Use case** | Production RHOAI deployment | Sandboxed experimentation | diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml new file mode 100644 index 00000000..798111a9 --- /dev/null +++ b/agents/codex/deployment/deployment.yaml @@ -0,0 +1,222 @@ +# Codex CLI — OpenShift Deployment (vLLM) +# +# Deploys Codex CLI as a persistent pod on OpenShift, connected to a +# vLLM inference endpoint via the OpenAI Responses API. +# +# Prerequisites: +# - oc CLI logged into an OpenShift cluster +# - A vLLM endpoint serving /v1/responses (vLLM v0.10.0+) +# +# Usage: +# oc new-project +# oc apply -f deployment.yaml +# oc start-build codex --from-dir=. --follow +# oc rollout status deployment/codex +# +# Then connect: +# oc exec -it deployment/codex -- bash +# codex-run exec "What files are in the current directory?" +# +# Update the OPENAI_BASE_URL, OPENAI_MODEL, and OPENAI_API_KEY below +# to match your vLLM endpoint. + +--- +# ============================================================================= +# ImageStream — allows the Deployment to reference the image by name +# ============================================================================= +apiVersion: image.openshift.io/v1 +kind: ImageStream +metadata: + name: codex + labels: + app: codex +spec: + lookupPolicy: + local: true + +--- +# ============================================================================= +# BuildConfig — binary source build using the Containerfile +# ============================================================================= +# Recommended: build locally and push to the internal registry. +# See the README for full instructions. +# +# podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . +# podman build --platform linux/amd64 -t codex:latest -f Containerfile \ +# --build-arg BASE_IMAGE=codex-base:latest . +# podman push //codex:latest +apiVersion: build.openshift.io/v1 +kind: BuildConfig +metadata: + name: codex + labels: + app: codex +spec: + source: + type: Binary + strategy: + type: Docker + dockerStrategy: + dockerfilePath: Containerfile + output: + to: + kind: ImageStreamTag + name: codex:latest + triggers: [] + +--- +# ============================================================================= +# ConfigMap — Codex config.toml (model provider, preferences) +# ============================================================================= +# The entrypoint auto-configures the model provider from OPENAI_BASE_URL +# and OPENAI_MODEL env vars. Use this ConfigMap only if you need additional +# config.toml settings beyond what the entrypoint generates. +apiVersion: v1 +kind: ConfigMap +metadata: + name: codex-config + labels: + app: codex +data: + config.toml: | + # Codex CLI configuration + # Model and provider are auto-configured by entrypoint.sh + # when OPENAI_BASE_URL and OPENAI_MODEL are set. + # + # Add additional config here if needed, for example: + # notify = "off" + +--- +# ============================================================================= +# ConfigMap — MCP server definitions (TOML format) +# ============================================================================= +apiVersion: v1 +kind: ConfigMap +metadata: + name: codex-mcp-config + labels: + app: codex +data: + mcp-servers.toml: | + # MCP server definitions in Codex-native TOML format. + # Add servers here: + # + # [mcp_servers.my-api] + # url = "https://mcp.example.com/v1" + # bearer_token_env_var = "API_TOKEN" + # + # [mcp_servers.local-tool] + # command = "npx" + # args = ["-y", "@some/mcp-server"] + +--- +# ============================================================================= +# PersistentVolumeClaim — workspace persistence across restarts +# ============================================================================= +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: codex-workspace + labels: + app: codex +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 1Gi + +--- +# ============================================================================= +# Deployment +# ============================================================================= +apiVersion: apps/v1 +kind: Deployment +metadata: + name: codex + labels: + app: codex +spec: + replicas: 1 + # Recreate strategy required because the PVC is ReadWriteOnce — + # the old pod must release the volume before the new pod can mount it. + strategy: + type: Recreate + selector: + matchLabels: + app: codex + template: + metadata: + labels: + app: codex + spec: + securityContext: + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + containers: + - name: codex + image: codex:latest + imagePullPolicy: Always + command: ["entrypoint.sh"] + args: ["sleep", "infinity"] + env: + - name: HOME + value: "/home/codex-agent" + - name: CODEX_HOME + value: "/workspace/.codex" + + # ── vLLM connection ────────────────────────────────────── + # UPDATE THESE to match your vLLM endpoint. + - name: OPENAI_BASE_URL + value: "http://vllm-svc.your-namespace.svc.cluster.local/v1" + - name: OPENAI_API_KEY + value: "dummy" + - name: OPENAI_MODEL + value: "your-model-id" + + # ── Sandbox ────────────────────────────────────────────── + # Container itself is the isolation boundary + - name: CODEX_SANDBOX + value: "danger-full-access" + + volumeMounts: + - name: workspace + mountPath: /workspace + - name: codex-config + mountPath: /etc/codex-config/config.toml + subPath: config.toml + - name: mcp-config + mountPath: /etc/codex-mcp/mcp-servers.toml + subPath: mcp-servers.toml + readOnly: true + + resources: + requests: + memory: "256Mi" + cpu: "100m" + limits: + memory: "1Gi" + cpu: "1" + + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + runAsNonRoot: true + seccompProfile: + type: RuntimeDefault + + volumes: + - name: workspace + persistentVolumeClaim: + claimName: codex-workspace + - name: codex-config + configMap: + name: codex-config + optional: true + - name: mcp-config + configMap: + name: codex-mcp-config + optional: true diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh new file mode 100644 index 00000000..e99b635e --- /dev/null +++ b/agents/codex/deployment/entrypoint.sh @@ -0,0 +1,343 @@ +#!/usr/bin/env bash +# ============================================================================= +# Codex CLI Container Entrypoint +# ============================================================================= +# +# Sets up the runtime environment for Codex CLI inside a container. +# Handles authentication, model provider configuration, MCP injection, +# and workspace setup before executing the provided command. +# +# Supported environment variables: +# +# Authentication: +# OPENAI_API_KEY API key for OpenAI or compatible endpoint +# CODEX_API_KEY API key for codex exec (overrides OPENAI_API_KEY) +# +# Model / Provider: +# OPENAI_BASE_URL Custom API endpoint (vLLM, OpenRouter, etc.) +# OPENAI_MODEL Model name (e.g., granite-3.3-8b-instruct) +# +# MCP Configuration: +# MCP_CONFIG_TOML Path to a TOML file with [mcp_servers.*] tables +# MCP_CONFIG_JSON Inline JSON with MCP server definitions +# (converted to TOML automatically) +# +# Git Credentials: +# GITHUB_PAT GitHub Personal Access Token +# GIT_USER_NAME Git commit author (default: codex-agent) +# GIT_USER_EMAIL Git commit email (default: codex-agent@noreply.github.com) +# +# Session Persistence: +# CODEX_HOME Codex config directory (default: /workspace/.codex) +# +# Sandbox: +# CODEX_SANDBOX Sandbox mode (default: danger-full-access) +# +# ============================================================================= + +set -euo pipefail + +# Temp files to clean up on exit +TEMP_FILES=() + +cleanup_temp_files() { + for f in "${TEMP_FILES[@]}"; do + rm -f "$f" 2>/dev/null || true + done +} + +trap cleanup_temp_files EXIT + +# ============================================================================= +# Helper Functions +# ============================================================================= + +log_info() { echo "[entrypoint] $*" >&2; } +log_warn() { echo "[entrypoint] WARNING: $*" >&2; } +log_error() { echo "[entrypoint] ERROR: $*" >&2; } + +# ============================================================================= +# Environment Validation +# ============================================================================= + +validate_environment() { + local has_auth=false + + if [[ -n "${OPENAI_API_KEY:-}" ]]; then + log_info "OPENAI_API_KEY is set" + has_auth=true + fi + + if [[ -n "${CODEX_API_KEY:-}" ]]; then + log_info "CODEX_API_KEY is set" + has_auth=true + fi + + if [[ "${has_auth}" != "true" ]]; then + log_warn "No API key found. Set OPENAI_API_KEY or CODEX_API_KEY." + log_warn "Codex may not be able to connect to an inference endpoint." + fi + + if [[ -n "${OPENAI_BASE_URL:-}" ]]; then + log_info "Custom endpoint: ${OPENAI_BASE_URL}" + fi + + if [[ -n "${OPENAI_MODEL:-}" ]]; then + log_info "Model: ${OPENAI_MODEL}" + fi +} + +# ============================================================================= +# Git Credentials +# ============================================================================= + +setup_git_credentials() { + if [[ -z "${GITHUB_PAT:-}" ]]; then + return + fi + + log_info "Configuring git credentials" + git config --global credential.helper store + echo "https://x-access-token:${GITHUB_PAT}@github.com" > "${HOME}/.git-credentials" + chmod 600 "${HOME}/.git-credentials" + + git config --global user.name "${GIT_USER_NAME:-codex-agent}" + git config --global user.email "${GIT_USER_EMAIL:-codex-agent@noreply.github.com}" +} + +# ============================================================================= +# Codex Home / Config Directory +# ============================================================================= + +setup_codex_home() { + CODEX_HOME="${CODEX_HOME:-/workspace/.codex}" + export CODEX_HOME + + mkdir -p "${CODEX_HOME}" 2>/dev/null || true + chmod g+w "${CODEX_HOME}" 2>/dev/null || true + + mkdir -p /workspace/projects 2>/dev/null || true + + # Symlink ~/.codex -> CODEX_HOME for PVC-backed persistence + local target="${HOME}/.codex" + if [[ -L "${target}" ]]; then + local current + current=$(readlink -f "${target}") + if [[ "${current}" != "$(readlink -f "${CODEX_HOME}")" ]]; then + rm -f "${target}" + ln -s "${CODEX_HOME}" "${target}" + log_info "Updated symlink: ~/.codex -> ${CODEX_HOME}" + fi + elif [[ -d "${target}" ]]; then + log_warn "~/.codex is a directory, replacing with symlink" + rm -rf "${target}" + ln -s "${CODEX_HOME}" "${target}" + else + ln -s "${CODEX_HOME}" "${target}" + log_info "Created symlink: ~/.codex -> ${CODEX_HOME}" + fi + + # Copy staged config from read-only ConfigMap mount (if it exists + # and destination is empty or missing) + local staged="/etc/codex-config/config.toml" + local config_file="${CODEX_HOME}/config.toml" + if [[ -f "${staged}" && ! -s "${config_file}" ]]; then + cp "${staged}" "${config_file}" + chmod 660 "${config_file}" + log_info "Copied config from ${staged}" + fi + + # Ensure config.toml exists (even if empty) + touch "${config_file}" +} + +# ============================================================================= +# Model Provider Configuration +# ============================================================================= + +setup_model_provider() { + local config_file="${CODEX_HOME}/config.toml" + + # Skip if no custom endpoint is specified + if [[ -z "${OPENAI_BASE_URL:-}" ]]; then + # Just set the model if specified + if [[ -n "${OPENAI_MODEL:-}" ]]; then + if ! grep -q '^model\s*=' "${config_file}" 2>/dev/null; then + echo "model = \"${OPENAI_MODEL}\"" >> "${config_file}" + fi + fi + return + fi + + # Skip if provider config already exists (e.g., from staged ConfigMap) + if grep -q 'model_provider' "${config_file}" 2>/dev/null; then + log_info "Provider config already exists in config.toml, skipping" + return + fi + + local model="${OPENAI_MODEL:-default}" + + cat >> "${config_file}" <> "${config_file}" + cat "${MCP_CONFIG_TOML}" >> "${config_file}" + else + log_error "MCP_CONFIG_TOML file not found: ${MCP_CONFIG_TOML}" + fi + fi + + # Method 2: Inline JSON (converted to TOML for compatibility with + # the Claude Code MCP_CONFIG_JSON pattern) + if [[ -n "${MCP_CONFIG_JSON:-}" ]]; then + log_info "Converting MCP JSON config to TOML" + local toml_block + toml_block=$(echo "${MCP_CONFIG_JSON}" | jq -r ' + .mcpServers // {} | to_entries[] | + "\n[mcp_servers.\(.key)]", + (if .value.url then "url = \"\(.value.url)\"" else empty end), + (if .value.command then "command = \"\(.value.command)\"" else empty end), + (if .value.args then "args = [\(.value.args | map("\"" + . + "\"") | join(", "))]" else empty end), + "" + ' 2>/dev/null) || { + log_warn "Failed to convert MCP JSON to TOML. Check MCP_CONFIG_JSON format." + return + } + + if [[ -n "${toml_block}" ]]; then + echo "${toml_block}" >> "${config_file}" + log_info "Appended MCP config from MCP_CONFIG_JSON" + fi + fi + + # Method 3: Mounted ConfigMap (TOML file at well-known path) + local mounted_mcp="/etc/codex-mcp/mcp-servers.toml" + if [[ -f "${mounted_mcp}" ]]; then + # Only append if the file has actual content (not just comments) + if grep -qE '^\[mcp_servers\.' "${mounted_mcp}" 2>/dev/null; then + log_info "Loading MCP config from: ${mounted_mcp}" + echo "" >> "${config_file}" + cat "${mounted_mcp}" >> "${config_file}" + fi + fi +} + +# ============================================================================= +# Build CLI Arguments +# ============================================================================= + +build_codex_args() { + local args=() + + # Model selection + if [[ -n "${OPENAI_MODEL:-}" ]]; then + args+=("--model" "${OPENAI_MODEL}") + fi + + # Sandbox mode — container itself is the isolation boundary + local sandbox="${CODEX_SANDBOX:-danger-full-access}" + args+=("--sandbox" "${sandbox}") + + # Skip git repo check — workspace may not be a git repo initially + args+=("--skip-git-repo-check") + + export CODEX_EXTRA_ARGS="${args[*]:-}" + + # Persist args for oc exec sessions + local codex_dir="${HOME}/.codex" + mkdir -p "${codex_dir}" 2>/dev/null || true + + cat > "${codex_dir}/env.sh" < "${codex_dir}/codex-run" <<'WRAPPER' +#!/bin/bash +# Wrapper for running Codex with pre-configured args. +# Usage: codex-run [codex arguments...] +# codex-run exec "fix the bug in main.py" +# codex-run (interactive mode) +source "${HOME}/.codex/env.sh" 2>/dev/null || true +exec codex ${CODEX_EXTRA_ARGS} "$@" +WRAPPER + chmod +x "${codex_dir}/codex-run" + + # Add to PATH via .bashrc for oc exec sessions + if ! grep -q 'codex/env.sh' "${HOME}/.bashrc" 2>/dev/null; then + cat >> "${HOME}/.bashrc" <<'BASHRC' +# Codex agent configuration +source "${HOME}/.codex/env.sh" 2>/dev/null || true +export PATH="${HOME}/.codex:${PATH}" +BASHRC + fi + + log_info "CLI args: ${args[*]:-}" +} + +# ============================================================================= +# Main +# ============================================================================= + +main() { + log_info "Starting Codex container" + log_info "Codex version: $(codex --version 2>/dev/null || echo 'unknown')" + + validate_environment + setup_git_credentials + setup_codex_home + setup_model_provider + setup_mcp + build_codex_args + + # Propagate OPENAI_API_KEY to CODEX_API_KEY for codex exec + if [[ -n "${OPENAI_API_KEY:-}" && -z "${CODEX_API_KEY:-}" ]]; then + export CODEX_API_KEY="${OPENAI_API_KEY}" + fi + + if [[ $# -eq 0 ]]; then + log_info "No command provided. Running: codex --help" + exec codex --help + fi + + if [[ "$1" == "codex" ]]; then + shift + log_info "Running: codex ${CODEX_EXTRA_ARGS} $*" + exec codex ${CODEX_EXTRA_ARGS} "$@" + fi + + log_info "Running: $*" + exec "$@" +} + +main "$@" From 1c19f6aa668761e91ac8c7fa4e9b326c3e6a5de2 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 10 Jul 2026 12:34:31 -0400 Subject: [PATCH 02/25] =?UTF-8?q?fix:=20address=20markdown=20lint=20?= =?UTF-8?q?=E2=80=94=20blank=20line=20before=20list?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/agents/codex/README.md b/agents/codex/README.md index 401596e8..61bee7cd 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -622,6 +622,7 @@ oc logs deployment/codex --previous ``` Common causes: + - `CODEX_HOME` directory not writable (PVC not mounted or permissions issue) - Invalid TOML in ConfigMap (syntax error in config.toml or mcp-servers.toml) - Missing base image (flavor Containerfile references a base image that has not been built) From 1ca1727b4405b356ee0b0efbe3b3c767e28b1e8e Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 10 Jul 2026 12:39:36 -0400 Subject: [PATCH 03/25] docs(codex): add base image build requirements (RAM, disk, time) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 39 ++++++++++++++++++++++++++++++++------- 1 file changed, 32 insertions(+), 7 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 61bee7cd..ddb6679f 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -54,7 +54,34 @@ Codex uses a two-stage image build: a **base image** containing the Codex Rust b ### Build the base image -The base image uses a multi-stage Dockerfile. The builder stage compiles Codex from the upstream Apache 2.0 source using the official Rust toolchain, producing an auditable binary with no opaque pre-built artifacts. Only the compiled binary is copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js 16 and npm from UBI repos for npx-based MCP servers. +The base image uses a multi-stage Dockerfile. The builder stage compiles Codex and bubblewrap (`bwrap`) from the upstream Apache 2.0 source using the official Rust toolchain, producing auditable binaries with no opaque pre-built artifacts. Only the compiled binaries are copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js 16 and npm from UBI repos for npx-based MCP servers. + +#### Build requirements + +| Resource | Minimum | Notes | +|----------|---------|-------| +| RAM | 32 GB | Podman VM must be configured with `podman machine set --memory 32768`. The Rust linker requires this during the final linking step of the 130+ crate workspace. | +| Disk | 50 GB | The builder stage downloads ~1.5 GB of Rust crate sources and produces ~20 GB of intermediate compilation artifacts. These are cleaned up before the layer is committed, but the space is needed during the build. | +| Time | ~12 min | Cross-compiling from ARM64 to x86_64. Native x86_64 builds are faster. | +| Network | Required | The builder clones the Codex source from GitHub and downloads the Rust toolchain + crate registry. Not needed once the image is built. | + +On Apple Silicon Macs, the builder uses cross-compilation (not QEMU emulation) to produce an x86_64 binary. This avoids the SIGSEGV crashes that occur when running `rustc` under QEMU, but requires more RAM than a native build. The upstream release profile's thin LTO is disabled during cross-compilation to stay within memory limits — the binary is functionally identical, just slightly larger. + +Native x86_64 builds (e.g., on the cluster via `oc start-build` or in CI) do not have the 32 GB RAM constraint and can use the default podman/Docker settings. + +#### Podman machine setup (macOS only) + +```bash +# First time +podman machine init --memory 32768 --disk-size 100 + +# Or update an existing machine +podman machine stop +podman machine set --memory 32768 +podman machine start +``` + +#### Build commands ```bash podman build --platform linux/amd64 \ @@ -70,6 +97,8 @@ podman build --platform linux/amd64 \ ### Build the RHOAI flavor image +The flavor image extends the base with the RHOAI entrypoint. This is a fast build (~5 seconds) since it only adds shell scripts. + ```bash podman build --platform linux/amd64 \ -t codex:latest \ @@ -94,14 +123,10 @@ Available tags are listed at [github.com/openai/codex/releases](https://github.c | Image | Approximate Size | Contents | |-------|-----------------|----------| -| `codex-base` | ~579 MB | UBI 9 minimal + Codex Rust binary (compiled from source) + git, jq, tar, nodejs, npm | +| `codex-base` | ~579 MB | UBI 9 minimal + Codex + bwrap (compiled from source) + git, jq, tar, nodejs, npm | | `codex` (flavor) | ~579 MB | Base + entrypoint.sh, codex-run wrapper | -The flavor image adds only shell scripts, so its size is effectively the same as the base. The base image build takes ~12 minutes because it cross-compiles Codex from Rust source. Cross-compiling on Apple Silicon requires 32GB+ RAM allocated to the podman VM (`podman machine set --memory 32768`). Native x86_64 builds (e.g., on the cluster via `oc start-build`) do not have this constraint. - -### Build with `--platform linux/amd64` - -Always pass `--platform linux/amd64` when building on Apple Silicon. The builder stage cross-compiles the Codex Rust binary for `x86_64-unknown-linux-gnu`, which is the architecture used by OpenShift nodes. +The flavor image adds only shell scripts, so its size is effectively the same as the base. --- From f9efe0d98e050692cd16686045491244b9fa2561 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 10 Jul 2026 13:01:15 -0400 Subject: [PATCH 04/25] fix(codex): fix entrypoint arg handling and MCP dedup - Split CODEX_EXTRA_ARGS (global) from CODEX_EXEC_ARGS (exec-only). --skip-git-repo-check is only valid for `codex exec`, not interactive mode. The codex-run wrapper now detects the subcommand and applies exec-only args appropriately. - MCP server injection now deduplicates: servers already present in config.toml are skipped on restart. Prevents accumulating duplicate [mcp_servers.*] entries across pod restarts when config is persisted on PVC. Applies to all three injection methods (TOML file, JSON env var, mounted ConfigMap). Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/deployment/entrypoint.sh | 87 ++++++++++++++++++--------- 1 file changed, 57 insertions(+), 30 deletions(-) diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index e99b635e..6dc5148a 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -203,9 +203,13 @@ setup_mcp() { # Method 1: TOML file path (preferred — native Codex format) if [[ -n "${MCP_CONFIG_TOML:-}" ]]; then if [[ -f "${MCP_CONFIG_TOML}" ]]; then - log_info "Appending MCP config from: ${MCP_CONFIG_TOML}" - echo "" >> "${config_file}" - cat "${MCP_CONFIG_TOML}" >> "${config_file}" + while IFS= read -r server_name; do + if ! grep -q "^\[mcp_servers\.${server_name}\]" "${config_file}" 2>/dev/null; then + log_info "Adding MCP server from MCP_CONFIG_TOML: ${server_name}" + sed -n "/^\[mcp_servers\.${server_name}\]/,/^\[/{ /^\[mcp_servers\.${server_name}\]/p; /^\[mcp_servers\./!{ /^\[/!p; }; }" "${MCP_CONFIG_TOML}" >> "${config_file}" + echo "" >> "${config_file}" + fi + done < <(grep -oP '^\[mcp_servers\.\K[^\]]+' "${MCP_CONFIG_TOML}" 2>/dev/null) else log_error "MCP_CONFIG_TOML file not found: ${MCP_CONFIG_TOML}" fi @@ -215,34 +219,44 @@ setup_mcp() { # the Claude Code MCP_CONFIG_JSON pattern) if [[ -n "${MCP_CONFIG_JSON:-}" ]]; then log_info "Converting MCP JSON config to TOML" - local toml_block - toml_block=$(echo "${MCP_CONFIG_JSON}" | jq -r ' - .mcpServers // {} | to_entries[] | - "\n[mcp_servers.\(.key)]", - (if .value.url then "url = \"\(.value.url)\"" else empty end), - (if .value.command then "command = \"\(.value.command)\"" else empty end), - (if .value.args then "args = [\(.value.args | map("\"" + . + "\"") | join(", "))]" else empty end), - "" - ' 2>/dev/null) || { - log_warn "Failed to convert MCP JSON to TOML. Check MCP_CONFIG_JSON format." + local server_names + server_names=$(echo "${MCP_CONFIG_JSON}" | jq -r '.mcpServers // {} | keys[]' 2>/dev/null) || { + log_warn "Failed to parse MCP_CONFIG_JSON. Check format." return } - if [[ -n "${toml_block}" ]]; then - echo "${toml_block}" >> "${config_file}" - log_info "Appended MCP config from MCP_CONFIG_JSON" - fi + for server_name in ${server_names}; do + if grep -q "^\[mcp_servers\.${server_name}\]" "${config_file}" 2>/dev/null; then + log_info "MCP server '${server_name}' already in config.toml, skipping" + continue + fi + local server_toml + server_toml=$(echo "${MCP_CONFIG_JSON}" | jq -r --arg name "$server_name" ' + .mcpServers[$name] | + "\n[mcp_servers.\($name)]", + (if .url then "url = \"\(.url)\"" else empty end), + (if .command then "command = \"\(.command)\"" else empty end), + (if .args then "args = [\(.args | map("\"" + . + "\"") | join(", "))]" else empty end), + "" + ' 2>/dev/null) + if [[ -n "${server_toml}" ]]; then + echo "${server_toml}" >> "${config_file}" + log_info "Added MCP server from MCP_CONFIG_JSON: ${server_name}" + fi + done fi # Method 3: Mounted ConfigMap (TOML file at well-known path) local mounted_mcp="/etc/codex-mcp/mcp-servers.toml" if [[ -f "${mounted_mcp}" ]]; then - # Only append if the file has actual content (not just comments) - if grep -qE '^\[mcp_servers\.' "${mounted_mcp}" 2>/dev/null; then - log_info "Loading MCP config from: ${mounted_mcp}" - echo "" >> "${config_file}" - cat "${mounted_mcp}" >> "${config_file}" - fi + # Only append servers that aren't already in config.toml + while IFS= read -r server_name; do + if ! grep -q "^\[mcp_servers\.${server_name}\]" "${config_file}" 2>/dev/null; then + log_info "Adding MCP server from ConfigMap: ${server_name}" + sed -n "/^\[mcp_servers\.${server_name}\]/,/^\[/{ /^\[mcp_servers\.${server_name}\]/p; /^\[mcp_servers\./!{ /^\[/!p; }; }" "${mounted_mcp}" >> "${config_file}" + echo "" >> "${config_file}" + fi + done < <(grep -oP '^\[mcp_servers\.\K[^\]]+' "${mounted_mcp}" 2>/dev/null) fi } @@ -252,20 +266,22 @@ setup_mcp() { build_codex_args() { local args=() + local exec_args=() - # Model selection + # Model selection (global — works with all subcommands) if [[ -n "${OPENAI_MODEL:-}" ]]; then args+=("--model" "${OPENAI_MODEL}") fi - # Sandbox mode — container itself is the isolation boundary + # Sandbox mode — container itself is the isolation boundary (global) local sandbox="${CODEX_SANDBOX:-danger-full-access}" args+=("--sandbox" "${sandbox}") - # Skip git repo check — workspace may not be a git repo initially - args+=("--skip-git-repo-check") + # Skip git repo check — only valid for `codex exec`, not interactive mode + exec_args+=("--skip-git-repo-check") export CODEX_EXTRA_ARGS="${args[*]:-}" + export CODEX_EXEC_ARGS="${exec_args[*]:-}" # Persist args for oc exec sessions local codex_dir="${HOME}/.codex" @@ -275,6 +291,7 @@ build_codex_args() { # Generated by entrypoint.sh — source this in oc exec sessions export CODEX_HOME="${CODEX_HOME}" export CODEX_EXTRA_ARGS="${args[*]:-}" +export CODEX_EXEC_ARGS="${exec_args[*]:-}" $([ -n "${OPENAI_API_KEY:-}" ] && echo "export OPENAI_API_KEY=\"${OPENAI_API_KEY}\"") $([ -n "${CODEX_API_KEY:-}" ] && echo "export CODEX_API_KEY=\"${CODEX_API_KEY}\"") $([ -n "${OPENAI_BASE_URL:-}" ] && echo "export OPENAI_BASE_URL=\"${OPENAI_BASE_URL}\"") @@ -289,7 +306,12 @@ EOF # codex-run exec "fix the bug in main.py" # codex-run (interactive mode) source "${HOME}/.codex/env.sh" 2>/dev/null || true -exec codex ${CODEX_EXTRA_ARGS} "$@" +# Add exec-only args when the subcommand is "exec" +if [[ "${1:-}" == "exec" ]]; then + exec codex ${CODEX_EXTRA_ARGS} "$1" ${CODEX_EXEC_ARGS} "${@:2}" +else + exec codex ${CODEX_EXTRA_ARGS} "$@" +fi WRAPPER chmod +x "${codex_dir}/codex-run" @@ -332,8 +354,13 @@ main() { if [[ "$1" == "codex" ]]; then shift - log_info "Running: codex ${CODEX_EXTRA_ARGS} $*" - exec codex ${CODEX_EXTRA_ARGS} "$@" + if [[ "${1:-}" == "exec" ]]; then + log_info "Running: codex ${CODEX_EXTRA_ARGS} exec ${CODEX_EXEC_ARGS} ${*:2}" + exec codex ${CODEX_EXTRA_ARGS} "$1" ${CODEX_EXEC_ARGS} "${@:2}" + else + log_info "Running: codex ${CODEX_EXTRA_ARGS} $*" + exec codex ${CODEX_EXTRA_ARGS} "$@" + fi fi log_info "Running: $*" From b2a193e1233e6df75a849c926af856df6e47835d Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 10 Jul 2026 13:03:12 -0400 Subject: [PATCH 05/25] fix(codex): address CodeRabbit review findings MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Remove API keys from env.sh persisted to PVC — secrets stay in container env vars only, not written to disk - Require OPENAI_MODEL when OPENAI_BASE_URL is set instead of falling back to invalid "default" model name - Deduplicate model entry in config.toml when staged ConfigMap already sets it - Replace --tls-verify=false with oc registry login in README - Remove cluster-build Option A from README (oc new-build without dockerfilePath doesn't find Containerfile.base) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 27 +++++++++------------------ agents/codex/deployment/entrypoint.sh | 16 ++++++++++++---- 2 files changed, 21 insertions(+), 22 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index ddb6679f..6a12d42e 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -160,22 +160,13 @@ The deployment manifest contains all required resources: ImageStream, BuildConfi # Apply all resources (ImageStream, BuildConfig, ConfigMaps, PVC, Deployment) oc apply -f deployment.yaml -# Option A: Build on the cluster (requires internet for docker.io/rust:1-bookworm) +# Build locally and push to the internal registry (recommended) # -# Create a BuildConfig for the base image -oc new-build --strategy=docker --binary --name=codex-base -oc start-build codex-base --from-dir=. --follow -# -# Then build the RHOAI flavor -oc start-build codex --from-dir=. --follow - -# Option B: Build locally and push (recommended — avoids Docker Hub rate limits) -# -# Build locally (see Step 1), then push to the internal registry: -REGISTRY=$(oc get route -n openshift-image-registry image-registry -o jsonpath='{.spec.host}') -podman login "${REGISTRY}" -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false -podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" -podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false +# Build locally (see Step 1), then push: +oc registry login +podman tag codex:latest \ + $(oc registry info)/$(oc project -q)/codex:latest +podman push $(oc registry info)/$(oc project -q)/codex:latest # # Import into the ImageStream oc tag --source=docker \ @@ -305,9 +296,9 @@ After modifying the Containerfile or entrypoint, rebuild locally and push: podman build --platform linux/amd64 -t codex:latest -f Containerfile \ --build-arg BASE_IMAGE=codex-base:latest . -REGISTRY=$(oc get route -n openshift-image-registry image-registry -o jsonpath='{.spec.host}') -podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" -podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false +oc registry login +podman tag codex:latest $(oc registry info)/$(oc project -q)/codex:latest +podman push $(oc registry info)/$(oc project -q)/codex:latest oc import-image codex:latest --confirm oc rollout restart deployment/codex diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index 6dc5148a..8385b6b8 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -175,12 +175,21 @@ setup_model_provider() { return fi - local model="${OPENAI_MODEL:-default}" + if [[ -z "${OPENAI_MODEL:-}" ]]; then + log_error "OPENAI_BASE_URL is set but OPENAI_MODEL is not. Set OPENAI_MODEL to the model name served by vLLM (check GET /v1/models)." + return + fi + + local model="${OPENAI_MODEL}" + + # Set model if not already present (avoids duplicate keys) + if ! grep -q '^model\s*=' "${config_file}" 2>/dev/null; then + echo "model = \"${model}\"" >> "${config_file}" + fi cat >> "${config_file}" < "${codex_dir}/env.sh" < Date: Tue, 14 Jul 2026 14:52:51 -0400 Subject: [PATCH 06/25] docs(codex): document vLLM version requirements from testing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested Codex CLI against vLLM v0.19.1 (RHOAI EA) and v0.25.1 (upstream). v0.19.1 returns "tool type namespace not supported" on every request — Codex cannot function. v0.25.1 works end-to-end with Qwen3-8B and tool calling enabled. - Add version compatibility table to Troubleshooting section - Document required vLLM flags (--enable-auto-tool-choice, --tool-call-parser) and model-specific parser values - Update Known Limitations with concrete version guidance - Fix entrypoint: env vars now override staged ConfigMap values for model/provider config (sed replace instead of skip) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 25 +++++++++++++++++++++---- agents/codex/deployment/deployment.yaml | 3 ++- agents/codex/deployment/entrypoint.sh | 14 ++++---------- 3 files changed, 27 insertions(+), 15 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 6a12d42e..588e210e 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -36,9 +36,9 @@ Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/c - `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) - `oc` CLI installed and logged into your OpenShift cluster -- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` (vLLM v0.10.0+) +- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `) -Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses`. See [Responses API Compatibility](#responses-api-compatibility) in Troubleshooting if you encounter errors. +Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. See [Responses API Compatibility](#responses-api-compatibility) in Troubleshooting for version requirements. All shell commands in this guide assume you are in the `deployment/` subdirectory, where the Containerfile and deployment manifests live: @@ -555,7 +555,24 @@ curl -s -o /dev/null -w "%{http_code}" \ }' ``` -The Responses API requires vLLM v0.10.0 or later. Full tool calling support (including namespace tool types used by Codex) may require newer versions — check [vLLM releases](https://github.com/vllm-project/vllm/releases) for the latest compatibility. +**vLLM version requirements:** + +| vLLM Version | Codex Compatibility | +|---|---| +| v0.19.1 (RHOAI 3.5 EA) | `/v1/responses` endpoint exists but returns `"tool type namespace not supported"` — Codex cannot function | +| v0.25.1 (upstream) | Fully functional — Codex connects, sends tool definitions, receives responses | + +The RHOAI early-access vLLM build (`registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.5.0-ea.1`) ships vLLM v0.19.1 which does not support the namespace tool types Codex sends with every request. This is resolved in upstream vLLM v0.25.x. Until a newer RHOAI-certified vLLM is available, use an upstream vLLM image (`docker.io/vllm/vllm-openai:v0.25.1`) for Codex workloads. + +Ensure the vLLM server is started with tool calling enabled: + +```bash +vllm serve \ + --enable-auto-tool-choice \ + --tool-call-parser +``` + +The `--tool-call-parser` value depends on the model: `qwen3_coder` for Qwen3, `hermes` for Llama/Granite/Mistral, `gemma4` for Gemma. See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list. ### Network Connectivity @@ -662,7 +679,7 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Your vLLM server must be v0.10.0+ with Responses API support enabled. Full tool calling support may require newer versions. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends namespace tool types with every request, which requires vLLM v0.25.x or later. The RHOAI early-access vLLM (v0.19.1) returns `"tool type namespace not supported"` and cannot be used with Codex. See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open Source Model Quality diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index 798111a9..4d27314e 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -5,7 +5,8 @@ # # Prerequisites: # - oc CLI logged into an OpenShift cluster -# - A vLLM endpoint serving /v1/responses (vLLM v0.10.0+) +# - A vLLM endpoint serving /v1/responses with tool calling enabled +# (vLLM v0.25.x+ required — RHOAI EA v0.19.1 does not support Codex) # # Usage: # oc new-project diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index 8385b6b8..120ed53f 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -169,12 +169,6 @@ setup_model_provider() { return fi - # Skip if provider config already exists (e.g., from staged ConfigMap) - if grep -q 'model_provider' "${config_file}" 2>/dev/null; then - log_info "Provider config already exists in config.toml, skipping" - return - fi - if [[ -z "${OPENAI_MODEL:-}" ]]; then log_error "OPENAI_BASE_URL is set but OPENAI_MODEL is not. Set OPENAI_MODEL to the model name served by vLLM (check GET /v1/models)." return @@ -182,12 +176,12 @@ setup_model_provider() { local model="${OPENAI_MODEL}" - # Set model if not already present (avoids duplicate keys) - if ! grep -q '^model\s*=' "${config_file}" 2>/dev/null; then - echo "model = \"${model}\"" >> "${config_file}" - fi + # Remove any existing model/provider lines so env vars always win + sed -i '/^model\s*=/d; /^model_provider\s*=/d' "${config_file}" 2>/dev/null || true + sed -i '/^\[model_providers\./,/^$/d' "${config_file}" 2>/dev/null || true cat >> "${config_file}" < Date: Wed, 15 Jul 2026 13:16:46 -0400 Subject: [PATCH 07/25] =?UTF-8?q?fix(codex):=20correct=20vLLM=20compatibil?= =?UTF-8?q?ity=20=E2=80=94=20parser=20issue,=20not=20version?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Testing gpt-oss-120b on upstream vLLM v0.25.1 with --tool-call-parser openai reproduces the same "tool type namespace not supported" error as RHOAI v0.21.0. The error is a parser limitation, not a version issue: the openai parser does not support the namespace tool type Codex sends. Using qwen3_coder or hermes parsers with compatible models works on v0.24.0+. Tested configurations: - RHOAI v0.21.0 + gpt-oss-120b + parser=openai → fails - Upstream v0.25.1 + gpt-oss-120b + parser=openai → fails (same error) - Upstream v0.25.1 + Qwen3-8B + parser=qwen3_coder → works - Upstream v0.24.0 + Qwen3-8B + parser=qwen3_coder → works Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 32 +++++++++++++++---------- agents/codex/deployment/deployment.yaml | 2 +- 2 files changed, 20 insertions(+), 14 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 588e210e..a6f0f767 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -36,9 +36,9 @@ Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/c - `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) - `oc` CLI installed and logged into your OpenShift cluster -- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `) +- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** the `--tool-call-parser openai` does not support the `namespace` tool type that Codex sends — use a model-specific parser (e.g., `qwen3_coder`, `hermes`). See [Responses API Compatibility](#responses-api-compatibility). -Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. See [Responses API Compatibility](#responses-api-compatibility) in Troubleshooting for version requirements. +Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. All shell commands in this guide assume you are in the `deployment/` subdirectory, where the Containerfile and deployment manifests live: @@ -555,24 +555,30 @@ curl -s -o /dev/null -w "%{http_code}" \ }' ``` -**vLLM version requirements:** +**Tool call parser compatibility:** -| vLLM Version | Codex Compatibility | -|---|---| -| v0.19.1 (RHOAI 3.5 EA) | `/v1/responses` endpoint exists but returns `"tool type namespace not supported"` — Codex cannot function | -| v0.25.1 (upstream) | Fully functional — Codex connects, sends tool definitions, receives responses | +Codex sends `namespace` tool types with every request. The `--tool-call-parser` flag determines whether vLLM accepts or rejects them. **The `openai` parser does not support namespace tools** — this is a parser limitation, not a vLLM version issue. It occurs on all vLLM versions (tested on v0.19.1, v0.21.0, and v0.25.1). -The RHOAI early-access vLLM build (`registry.redhat.io/rhaii-early-access/vllm-cuda-rhel9:3.5.0-ea.1`) ships vLLM v0.19.1 which does not support the namespace tool types Codex sends with every request. This is resolved in upstream vLLM v0.25.x. Until a newer RHOAI-certified vLLM is available, use an upstream vLLM image (`docker.io/vllm/vllm-openai:v0.25.1`) for Codex workloads. +| Parser | Namespace Tools | Codex Works? | Models | +|---|---|---|---| +| `openai` | Rejected: `"tool type namespace not supported"` | No | gpt-oss family | +| `qwen3_coder` | Accepted | Yes (v0.24.0+) | Qwen3 | +| `hermes` | Accepted | Yes (v0.24.0+) | Llama, Granite, Mistral | +| `gemma4` | Accepted | Yes (v0.24.0+) | Gemma 4 | -Ensure the vLLM server is started with tool calling enabled: +To use Codex with vLLM, choose a model with a supported parser: ```bash -vllm serve \ +# Example: Qwen3-8B (tested, works with Codex) +vllm serve Qwen/Qwen3-8B \ --enable-auto-tool-choice \ - --tool-call-parser + --tool-call-parser qwen3_coder \ + --reasoning-parser qwen3 ``` -The `--tool-call-parser` value depends on the model: `qwen3_coder` for Qwen3, `hermes` for Llama/Granite/Mistral, `gemma4` for Gemma. See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list. +The `openai` parser is specific to OpenAI's gpt-oss model family and does not implement the namespace tool type that Codex requires. This is unrelated to the vLLM version or the RHOAI build. See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list of supported parsers. + +**Note:** While namespace tools are accepted on vLLM v0.19.1–v0.20.x with compatible parsers, full end-to-end Codex functionality (tool execution with visible output) requires vLLM v0.24.0 or later. ### Network Connectivity @@ -679,7 +685,7 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends namespace tool types with every request, which requires vLLM v0.25.x or later. The RHOAI early-access vLLM (v0.19.1) returns `"tool type namespace not supported"` and cannot be used with Codex. See [Responses API Compatibility](#responses-api-compatibility) for details. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. The `--tool-call-parser openai` in vLLM does not support namespace tools and returns `"tool type namespace not supported"` — this affects all vLLM versions, not just older ones. Use a model-specific parser instead (e.g., `qwen3_coder`, `hermes`). Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open Source Model Quality diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index 4d27314e..f3c5c07f 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -6,7 +6,7 @@ # Prerequisites: # - oc CLI logged into an OpenShift cluster # - A vLLM endpoint serving /v1/responses with tool calling enabled -# (vLLM v0.25.x+ required — RHOAI EA v0.19.1 does not support Codex) +# (use --tool-call-parser qwen3_coder or hermes, NOT openai) # # Usage: # oc new-project From 18cd001ec08962fedb94c6d9c3a7429ff298cb0c Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 16 Jul 2026 10:13:53 -0400 Subject: [PATCH 08/25] =?UTF-8?q?fix(codex):=20correct=20root=20cause=20?= =?UTF-8?q?=E2=80=94=20gpt-oss=20model=20rejects=20namespace=20tools,=20no?= =?UTF-8?q?t=20parser?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Testing gpt-oss-120b with hermes parser on upstream v0.25.1 still returns "tool type namespace not supported". The rejection is in a gpt-oss-specific code path in vLLM's Responses API layer, before the parser is invoked. Open-source models (Qwen3, Llama) work fine. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 27 +++++++++++++++------------ 1 file changed, 15 insertions(+), 12 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index a6f0f767..2b8d6c61 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -36,7 +36,7 @@ Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/c - `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) - `oc` CLI installed and logged into your OpenShift cluster -- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** the `--tool-call-parser openai` does not support the `namespace` tool type that Codex sends — use a model-specific parser (e.g., `qwen3_coder`, `hermes`). See [Responses API Compatibility](#responses-api-compatibility). +- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** gpt-oss models reject namespace tools on all vLLM versions and parsers — use an open-source model (e.g., Qwen3, Llama) instead. See [Responses API Compatibility](#responses-api-compatibility). Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. @@ -555,18 +555,21 @@ curl -s -o /dev/null -w "%{http_code}" \ }' ``` -**Tool call parser compatibility:** +**Namespace tool type compatibility:** -Codex sends `namespace` tool types with every request. The `--tool-call-parser` flag determines whether vLLM accepts or rejects them. **The `openai` parser does not support namespace tools** — this is a parser limitation, not a vLLM version issue. It occurs on all vLLM versions (tested on v0.19.1, v0.21.0, and v0.25.1). +Codex sends `namespace` tool types with every request. Whether vLLM accepts them depends on the **model architecture**, not the parser or vLLM version. -| Parser | Namespace Tools | Codex Works? | Models | +| Model | Parser | vLLM Version | Result | |---|---|---|---| -| `openai` | Rejected: `"tool type namespace not supported"` | No | gpt-oss family | -| `qwen3_coder` | Accepted | Yes (v0.24.0+) | Qwen3 | -| `hermes` | Accepted | Yes (v0.24.0+) | Llama, Granite, Mistral | -| `gemma4` | Accepted | Yes (v0.24.0+) | Gemma 4 | +| gpt-oss-120b | `openai` | RHOAI v0.21.0 | `"tool type namespace not supported"` | +| gpt-oss-120b | `openai` | Upstream v0.25.1 | `"tool type namespace not supported"` | +| gpt-oss-120b | `hermes` | Upstream v0.25.1 | `"tool type namespace not supported"` | +| Qwen3-8B | `qwen3_coder` | Upstream v0.24.0 | Works | +| Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Works | -To use Codex with vLLM, choose a model with a supported parser: +The `"tool type namespace not supported"` error is specific to the **gpt-oss model family** (`GptOssForCausalLM` architecture). It occurs on all vLLM versions and all parsers — the rejection happens in a gpt-oss-specific code path in vLLM's Responses API layer, before the tool call parser is invoked. Namespace tool support for gpt-oss may be restricted to OpenAI's hosted infrastructure. + +**To use Codex with vLLM, use an open-source model:** ```bash # Example: Qwen3-8B (tested, works with Codex) @@ -576,9 +579,9 @@ vllm serve Qwen/Qwen3-8B \ --reasoning-parser qwen3 ``` -The `openai` parser is specific to OpenAI's gpt-oss model family and does not implement the namespace tool type that Codex requires. This is unrelated to the vLLM version or the RHOAI build. See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list of supported parsers. +See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list of supported models and parsers. -**Note:** While namespace tools are accepted on vLLM v0.19.1–v0.20.x with compatible parsers, full end-to-end Codex functionality (tool execution with visible output) requires vLLM v0.24.0 or later. +**Note:** Full end-to-end Codex functionality (tool execution with visible output) requires vLLM v0.24.0 or later. ### Network Connectivity @@ -685,7 +688,7 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. The `--tool-call-parser openai` in vLLM does not support namespace tools and returns `"tool type namespace not supported"` — this affects all vLLM versions, not just older ones. Use a model-specific parser instead (e.g., `qwen3_coder`, `hermes`). Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. The gpt-oss model family rejects namespace tools with `"tool type namespace not supported"` on all vLLM versions and parsers — use open-source models (Qwen3, Llama, etc.) instead. Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open Source Model Quality From fad96b9391086da62a0d4bb3bae451ccaf4106bd Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Fri, 17 Jul 2026 14:46:31 -0400 Subject: [PATCH 09/25] =?UTF-8?q?docs(codex):=20confirmed=20=E2=80=94=20na?= =?UTF-8?q?mespace=20tools=20unsupported=20for=20harmony=20models=20in=20v?= =?UTF-8?q?LLM?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per vLLM team: namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony (gpt-oss) models. This is a known limitation. Updated docs to use correct terminology (harmony vs non-harmony) and remove speculation. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 2b8d6c61..6b612ee4 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -36,7 +36,7 @@ Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/c - `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) - `oc` CLI installed and logged into your OpenShift cluster -- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** gpt-oss models reject namespace tools on all vLLM versions and parsers — use an open-source model (e.g., Qwen3, Llama) instead. See [Responses API Compatibility](#responses-api-compatibility). +- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** harmony models (gpt-oss) do not yet support namespace tools in vLLM — use a non-harmony model (e.g., Qwen3, Llama). See [Responses API Compatibility](#responses-api-compatibility). Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. @@ -567,7 +567,7 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | Qwen3-8B | `qwen3_coder` | Upstream v0.24.0 | Works | | Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Works | -The `"tool type namespace not supported"` error is specific to the **gpt-oss model family** (`GptOssForCausalLM` architecture). It occurs on all vLLM versions and all parsers — the rejection happens in a gpt-oss-specific code path in vLLM's Responses API layer, before the tool call parser is invoked. Namespace tool support for gpt-oss may be restricted to OpenAI's hosted infrastructure. +The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error occurs on all vLLM versions and all parsers. **To use Codex with vLLM, use an open-source model:** @@ -688,7 +688,7 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. The gpt-oss model family rejects namespace tools with `"tool type namespace not supported"` on all vLLM versions and parsers — use open-source models (Qwen3, Llama, etc.) instead. Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models (Qwen3, Llama, etc.) instead. Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open Source Model Quality From d773f6076418afe793aac8f0a52ce2e7953a7714 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Sat, 18 Jul 2026 10:30:08 -0400 Subject: [PATCH 10/25] fix(codex): address code review findings Security fixes: - Sanitize OPENAI_BASE_URL and OPENAI_MODEL before interpolating into config.toml and env.sh (prevents TOML/shell injection) - Use single-quoted printf in env.sh instead of double-quoted heredoc - Use secretKeyRef for OPENAI_API_KEY in deployment.yaml Supply chain fixes: - Pin Rust builder image to 1.87.0-bookworm (was rolling :1-bookworm) - Document that quay.io/hmoghani base image will move to org-owned registry; users can build locally with Containerfile.base Documentation fixes: - Fix Node.js version reference (UBI9 ships 18, not 16) - Add codex/deployment to AGENTS.md non-standard agents section Co-Authored-By: Claude Opus 4.6 (1M context) --- AGENTS.md | 2 ++ agents/codex/README.md | 2 +- agents/codex/deployment/Containerfile | 4 +++ agents/codex/deployment/Containerfile.base | 2 +- agents/codex/deployment/deployment.yaml | 7 ++++- agents/codex/deployment/entrypoint.sh | 32 ++++++++++++++-------- 6 files changed, 34 insertions(+), 15 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index e516f8d2..b5750e6c 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,6 +74,8 @@ Several agents diverge significantly from the standard pattern: **openclaw/deployment** - Kustomize-based deployment of OpenClaw on OpenShift. Uses pre-built image (`ghcr.io/openclaw/openclaw:latest`), not a Dockerfile. Kustomize overlays for customization (model endpoint, storage class). No Makefile, no FastAPI, no src/. Port 18789 (gateway). +**codex/deployment** - CLI-based coding agent (OpenAI Codex CLI, Apache 2.0). Two-stage Containerfile: `Containerfile.base` cross-compiles the Rust binary from source, `Containerfile` extends with RHOAI entrypoint. No Makefile, no pyproject.toml, no FastAPI, no `/chat/completions`, no tests/. Uses `entrypoint.sh` for runtime config, `sleep infinity` for `oc exec` access. Requires vLLM with Responses API (`/v1/responses`) and non-harmony models. + ## Common gotchas - `agent.yaml` is read at runtime by Makefile -- if malformed, all make targets break diff --git a/agents/codex/README.md b/agents/codex/README.md index 6b612ee4..7b8ad042 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -54,7 +54,7 @@ Codex uses a two-stage image build: a **base image** containing the Codex Rust b ### Build the base image -The base image uses a multi-stage Dockerfile. The builder stage compiles Codex and bubblewrap (`bwrap`) from the upstream Apache 2.0 source using the official Rust toolchain, producing auditable binaries with no opaque pre-built artifacts. Only the compiled binaries are copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js 16 and npm from UBI repos for npx-based MCP servers. +The base image uses a multi-stage Dockerfile. The builder stage compiles Codex and bubblewrap (`bwrap`) from the upstream Apache 2.0 source using the official Rust toolchain, producing auditable binaries with no opaque pre-built artifacts. Only the compiled binaries are copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js and npm from UBI repos for npx-based MCP servers. #### Build requirements diff --git a/agents/codex/deployment/Containerfile b/agents/codex/deployment/Containerfile index 6504e1a3..6a0cf88f 100644 --- a/agents/codex/deployment/Containerfile +++ b/agents/codex/deployment/Containerfile @@ -18,6 +18,10 @@ # - Git credential injection via GITHUB_PAT # - Non-root user compatible with restricted-v2 SCC +# Base image built from source (see Containerfile.base). +# Default points to quay.io/hmoghani — this will move to an org-owned +# registry (e.g., quay.io/opendatahub) once access is provisioned. +# To build the base image yourself: see Containerfile.base instructions. ARG BASE_IMAGE=quay.io/hmoghani/codex-base:latest FROM ${BASE_IMAGE} diff --git a/agents/codex/deployment/Containerfile.base b/agents/codex/deployment/Containerfile.base index 6c2131da..7f4039b2 100644 --- a/agents/codex/deployment/Containerfile.base +++ b/agents/codex/deployment/Containerfile.base @@ -28,7 +28,7 @@ # $BUILDPLATFORM ensures this stage runs natively (no QEMU). # We cross-compile to the $TARGETPLATFORM using Rust's --target flag. -FROM --platform=$BUILDPLATFORM docker.io/library/rust:1-bookworm AS builder +FROM --platform=$BUILDPLATFORM docker.io/library/rust:1.87.0-bookworm AS builder ARG CODEX_VERSION=rust-v0.144.0 ARG TARGETARCH diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index f3c5c07f..cbbe07ce 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -169,10 +169,15 @@ spec: # ── vLLM connection ────────────────────────────────────── # UPDATE THESE to match your vLLM endpoint. + # Create the secret first: + # oc create secret generic codex-api-key --from-literal=api-key=dummy - name: OPENAI_BASE_URL value: "http://vllm-svc.your-namespace.svc.cluster.local/v1" - name: OPENAI_API_KEY - value: "dummy" + valueFrom: + secretKeyRef: + name: codex-api-key + key: api-key - name: OPENAI_MODEL value: "your-model-id" diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index 120ed53f..a711346d 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -56,6 +56,11 @@ log_info() { echo "[entrypoint] $*" >&2; } log_warn() { echo "[entrypoint] WARNING: $*" >&2; } log_error() { echo "[entrypoint] ERROR: $*" >&2; } +sanitize_value() { + # Strip characters that could cause shell or TOML injection + printf '%s' "$1" | tr -d '"'"'"'`$\\' | tr -d '\n' +} + # ============================================================================= # Environment Validation # ============================================================================= @@ -174,7 +179,10 @@ setup_model_provider() { return fi - local model="${OPENAI_MODEL}" + local model + local base_url + model=$(sanitize_value "${OPENAI_MODEL}") + base_url=$(sanitize_value "${OPENAI_BASE_URL}") # Remove any existing model/provider lines so env vars always win sed -i '/^model\s*=/d; /^model_provider\s*=/d' "${config_file}" 2>/dev/null || true @@ -188,12 +196,12 @@ model_provider = "vllm" [model_providers.vllm] name = "vLLM" -base_url = "${OPENAI_BASE_URL}" +base_url = "${base_url}" env_key = "OPENAI_API_KEY" wire_api = "responses" EOF - log_info "Configured vLLM provider: ${OPENAI_BASE_URL} (model: ${model})" + log_info "Configured vLLM provider: ${base_url} (model: ${model})" } # ============================================================================= @@ -290,15 +298,15 @@ build_codex_args() { local codex_dir="${HOME}/.codex" mkdir -p "${codex_dir}" 2>/dev/null || true - cat > "${codex_dir}/env.sh" < "${codex_dir}/env.sh" # Create wrapper script for oc exec convenience cat > "${codex_dir}/codex-run" <<'WRAPPER' From a19d9cff138249e10a26d7bec48caeefe698451a Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Tue, 21 Jul 2026 14:23:57 -0400 Subject: [PATCH 11/25] =?UTF-8?q?fix(codex):=20address=20reviewer=20feedba?= =?UTF-8?q?ck=20=E2=80=94=20npm=20image,=20registry=20docs,=20CODEX=5FAPI?= =?UTF-8?q?=5FKEY?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Containerfile changes: - Switch to npm-based Containerfile (single stage, no base image needed). Added upstream disclaimer matching OpenClaw pattern. - Containerfile.base: add codex-code-mode-host binary (required for tool execution, reported by msager27) Deployment fixes: - Add CODEX_API_KEY to deployment.yaml (required for sleep infinity + oc exec mode, reported by msager27) - Use secretKeyRef for both OPENAI_API_KEY and CODEX_API_KEY Documentation fixes: - Fix registry push commands: use external route via oc get route instead of oc registry info (reported by bardenr) - Fix codex-run examples: use bash -l (login shell) so .bashrc PATH is loaded (reported by bardenr) - Update build instructions to reflect single-stage npm build - Remove stale two-stage / BASE_IMAGE references - Add disclaimer about pre-built npm binary Security fixes (shricharan-ks review): - Sanitize env vars before TOML/shell interpolation - Single-quoted printf in env.sh - secretKeyRef for API keys - Pin Rust builder image in Containerfile.base - Add codex to AGENTS.md non-standard agents Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 124 +++++++-------------- agents/codex/deployment/Containerfile | 78 +++++++++---- agents/codex/deployment/Containerfile.base | 7 +- agents/codex/deployment/README.md | 38 ++++--- agents/codex/deployment/deployment.yaml | 8 ++ 5 files changed, 127 insertions(+), 128 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 7b8ad042..de762842 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -50,83 +50,35 @@ cd deployment/ ## Step 1: Build the Container Image -Codex uses a two-stage image build: a **base image** containing the Codex Rust binary on UBI 9 minimal, and a **flavor image** that extends it with the RHOAI entrypoint, MCP injection, and session persistence. - -### Build the base image - -The base image uses a multi-stage Dockerfile. The builder stage compiles Codex and bubblewrap (`bwrap`) from the upstream Apache 2.0 source using the official Rust toolchain, producing auditable binaries with no opaque pre-built artifacts. Only the compiled binaries are copied into the clean UBI 9 minimal runtime. The runtime stage installs Node.js and npm from UBI repos for npx-based MCP servers. - -#### Build requirements - -| Resource | Minimum | Notes | -|----------|---------|-------| -| RAM | 32 GB | Podman VM must be configured with `podman machine set --memory 32768`. The Rust linker requires this during the final linking step of the 130+ crate workspace. | -| Disk | 50 GB | The builder stage downloads ~1.5 GB of Rust crate sources and produces ~20 GB of intermediate compilation artifacts. These are cleaned up before the layer is committed, but the space is needed during the build. | -| Time | ~12 min | Cross-compiling from ARM64 to x86_64. Native x86_64 builds are faster. | -| Network | Required | The builder clones the Codex source from GitHub and downloads the Rust toolchain + crate registry. Not needed once the image is built. | - -On Apple Silicon Macs, the builder uses cross-compilation (not QEMU emulation) to produce an x86_64 binary. This avoids the SIGSEGV crashes that occur when running `rustc` under QEMU, but requires more RAM than a native build. The upstream release profile's thin LTO is disabled during cross-compilation to stay within memory limits — the binary is functionally identical, just slightly larger. - -Native x86_64 builds (e.g., on the cluster via `oc start-build` or in CI) do not have the 32 GB RAM constraint and can use the default podman/Docker settings. - -#### Podman machine setup (macOS only) - -```bash -# First time -podman machine init --memory 32768 --disk-size 100 - -# Or update an existing machine -podman machine stop -podman machine set --memory 32768 -podman machine start -``` - -#### Build commands +The Codex CLI binary installed in this image is a pre-built artifact published by OpenAI via npm (`@openai/codex`). It has not been built, scanned, or validated according to Red Hat standards. Use it at your own discretion. To build from source instead, see `Containerfile.base`. ```bash podman build --platform linux/amd64 \ - -t codex-base:latest \ - -f Containerfile.base . + -t codex:latest \ + -f Containerfile . -# Build with a specific version (use rust-v* git tags) +# Build with a specific version podman build --platform linux/amd64 \ - --build-arg CODEX_VERSION=rust-v0.144.0 \ - -t codex-base:0.144.0 \ - -f Containerfile.base . + --build-arg CODEX_VERSION=0.144.0 \ + -t codex:0.144.0 \ + -f Containerfile . ``` -### Build the RHOAI flavor image - -The flavor image extends the base with the RHOAI entrypoint. This is a fast build (~5 seconds) since it only adds shell scripts. - -```bash -podman build --platform linux/amd64 \ - -t codex:latest \ - -f Containerfile \ - --build-arg BASE_IMAGE=codex-base:latest . -``` +Always pass `--platform linux/amd64` when building on Apple Silicon — the npm package downloads a platform-specific binary, and x86_64 is required for OpenShift nodes. ### Version pinning -The `CODEX_VERSION` build arg controls which git tag to build from. It defaults to `rust-v0.144.0`. Pin it explicitly for reproducible builds: - -```bash -podman build --platform linux/amd64 \ - --build-arg CODEX_VERSION=rust-v0.144.0 \ - -t codex-base:0.144.0 \ - -f Containerfile.base . -``` +The `CODEX_VERSION` build arg controls which npm package version to install. It defaults to `0.144.0`. Available versions are listed at [npmjs.com/package/@openai/codex](https://www.npmjs.com/package/@openai/codex). -Available tags are listed at [github.com/openai/codex/releases](https://github.com/openai/codex/releases). Use the `rust-v*` tags. - -### Image sizes +### Image size | Image | Approximate Size | Contents | |-------|-----------------|----------| -| `codex-base` | ~579 MB | UBI 9 minimal + Codex + bwrap (compiled from source) + git, jq, tar, nodejs, npm | -| `codex` (flavor) | ~579 MB | Base + entrypoint.sh, codex-run wrapper | +| `codex` | ~579 MB | UBI 9 minimal + Codex CLI (via npm) + git, jq, tar, nodejs, npm + entrypoint | + +### Building from source (alternative) -The flavor image adds only shell scripts, so its size is effectively the same as the base. +For supply chain trust, `Containerfile.base` compiles Codex from the upstream Apache 2.0 source using the Rust toolchain. This produces an auditable binary with no pre-built artifacts. See the header comments in `Containerfile.base` for build instructions. Building from source requires 32 GB+ RAM in the podman VM for cross-compilation on Apple Silicon. --- @@ -162,11 +114,13 @@ oc apply -f deployment.yaml # Build locally and push to the internal registry (recommended) # -# Build locally (see Step 1), then push: -oc registry login -podman tag codex:latest \ - $(oc registry info)/$(oc project -q)/codex:latest -podman push $(oc registry info)/$(oc project -q)/codex:latest +# Get the external registry route and login: +REGISTRY=$(oc get route -n openshift-image-registry -o jsonpath='{.items[0].spec.host}') +podman login "${REGISTRY}" -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false +# +# Tag and push: +podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" +podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false # # Import into the ImageStream oc tag --source=docker \ @@ -223,10 +177,10 @@ The deployment runs `sleep infinity` so the pod stays alive for `oc exec` sessio ### The codex-run Wrapper -The entrypoint generates a `codex-run` wrapper script at `~/.codex/codex-run` that includes all container-configured arguments (model, sandbox mode, provider config). Use it for all `oc exec` sessions: +The entrypoint generates a `codex-run` wrapper script at `~/.codex/codex-run` that includes all container-configured arguments (model, sandbox mode, provider config). Use it for all `oc exec` sessions. Use `bash -l` (login shell) so `.bashrc` adds `~/.codex` to PATH: ```bash -oc exec deployment/codex -- bash -c ' +oc exec deployment/codex -- bash -lc ' codex-run exec "Your prompt here" ' ``` @@ -245,9 +199,7 @@ oc exec deployment/codex -- bash -c ' For multi-turn conversations with a TTY: ```bash -oc exec -it deployment/codex -- bash -c ' - codex-run -' +oc exec -it deployment/codex -- bash -lc 'codex-run' ``` Interactive mode requires the `-it` flags on `oc exec`. Without them, the TUI will not render and Codex will exit immediately. @@ -258,12 +210,12 @@ The `codex exec` subcommand runs a prompt non-interactively and exits when compl ```bash # Single prompt -oc exec deployment/codex -- bash -c ' +oc exec deployment/codex -- bash -lc ' codex-run exec "Explain the structure of this project" ' # With a specific working directory -oc exec deployment/codex -- bash -c ' +oc exec deployment/codex -- bash -lc ' cd /workspace/projects/my-repo && codex-run exec "Run the tests and fix any failures" ' ``` @@ -293,12 +245,12 @@ oc exec deployment/codex -- ls -la /workspace/projects/ After modifying the Containerfile or entrypoint, rebuild locally and push: ```bash -podman build --platform linux/amd64 -t codex:latest -f Containerfile \ - --build-arg BASE_IMAGE=codex-base:latest . +podman build --platform linux/amd64 -t codex:latest -f Containerfile . -oc registry login -podman tag codex:latest $(oc registry info)/$(oc project -q)/codex:latest -podman push $(oc registry info)/$(oc project -q)/codex:latest +REGISTRY=$(oc get route -n openshift-image-registry -o jsonpath='{.items[0].spec.host}') +podman login "${REGISTRY}" -u $(oc whoami) -p $(oc whoami -t) --tls-verify=false +podman tag codex:latest "${REGISTRY}/$(oc project -q)/codex:latest" +podman push "${REGISTRY}/$(oc project -q)/codex:latest" --tls-verify=false oc import-image codex:latest --confirm oc rollout restart deployment/codex @@ -456,14 +408,14 @@ The JSON-to-TOML conversion supports `url`, `command`, and `args` fields. For MC | Remote HTTP | `url` | Remote MCP servers | Network access to endpoint | | Local process | `command` + `args` | Process-based MCP servers | Executable must exist in container | -The base image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, and `npm`. Command-based MCP servers using `npx` work out of the box. For servers requiring other runtimes, see [Extending the Container Image](#extending-the-container-image). +The image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, and `npm`. Command-based MCP servers using `npx` work out of the box. For servers requiring other runtimes, see [Extending the Container Image](#extending-the-container-image). ### Extending the Container Image -The base image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, `npm`, and `bash`. For real coding workflows, add language runtimes so the agent can run tests, lint, and build. Including the runtimes your project uses significantly improves agent output quality since the agent can run tests and catch its own mistakes. +The image includes `git`, `curl-minimal`, `jq`, `tar`, `nodejs`, `npm`, and `bash`. For real coding workflows, add language runtimes so the agent can run tests, lint, and build. Including the runtimes your project uses significantly improves agent output quality since the agent can run tests and catch its own mistakes. ```dockerfile -# In the "System dependencies" section of Containerfile.base +# In the "System dependencies" section of Containerfile RUN microdnf install -y --nodocs \ git \ jq \ @@ -667,7 +619,7 @@ Common causes: - `CODEX_HOME` directory not writable (PVC not mounted or permissions issue) - Invalid TOML in ConfigMap (syntax error in config.toml or mcp-servers.toml) -- Missing base image (flavor Containerfile references a base image that has not been built) +- npm registry unreachable (Codex CLI downloaded from npm at build time) --- @@ -675,8 +627,8 @@ Common causes: ```bash oc delete deployment codex -oc delete buildconfig codex codex-base 2>/dev/null -oc delete imagestream codex codex-base 2>/dev/null +oc delete buildconfig codex 2>/dev/null +oc delete imagestream codex 2>/dev/null oc delete configmap codex-config codex-mcp-config oc delete pvc codex-workspace oc delete project my-codex-project @@ -704,4 +656,4 @@ Codex CLI uses TOML for all configuration (`config.toml`). There is no JSON sett ### OpenShell Sandbox Variant -The `Containerfile.openshell` builds a separate image for the OpenShell sandbox environment. This variant uses a different base image (`openshell-base`), installs Codex via npm (not the extracted Rust binary), and does not include the RHOAI entrypoint or session persistence. It is intended for ephemeral sandbox sessions, not persistent OpenShift deployments. +The `Containerfile.openshell` builds a separate image for the OpenShell sandbox environment. This variant uses a different base image (`openshell-base`) and does not include the RHOAI entrypoint or session persistence. It is intended for ephemeral sandbox sessions, not persistent OpenShift deployments. diff --git a/agents/codex/deployment/Containerfile b/agents/codex/deployment/Containerfile index 6a0cf88f..470d9c7b 100644 --- a/agents/codex/deployment/Containerfile +++ b/agents/codex/deployment/Containerfile @@ -1,45 +1,79 @@ # Codex CLI — RHOAI Flavor # -# Extends the Codex base image with RHOAI-specific configuration: -# - Entrypoint with env-var-driven setup (auth, model provider, MCP, git) -# - codex-run wrapper for oc exec convenience -# - Session persistence via PVC-backed CODEX_HOME +# Container image for running OpenAI's Codex CLI on RHOAI. +# +# ⚠ Important: The Codex CLI binary installed in this image is a pre-built +# artifact published by OpenAI via npm (@openai/codex). It has not been +# built, scanned, or validated according to Red Hat standards. Use it at +# your own discretion. To build from source instead, use Containerfile.base. # -# Build the base image first, then this flavor: -# podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . -# podman build --platform linux/amd64 -t codex:latest -f Containerfile \ -# --build-arg BASE_IMAGE=codex-base:latest . +# Build: +# podman build --platform linux/amd64 -t codex:latest -f Containerfile . # # Features: +# - Codex CLI installed via npm (version-pinned, Apache 2.0) +# - Entrypoint with env-var-driven setup (auth, model provider, MCP, git) +# - codex-run wrapper for oc exec convenience +# - Session persistence via PVC-backed CODEX_HOME # - Configurable inference endpoint (vLLM, OpenAI, OpenRouter) -# - MCP server configuration injectable at runtime via environment variables: -# MCP_CONFIG_TOML - path to a TOML file with [mcp_servers.*] tables -# MCP_CONFIG_JSON - inline JSON string with MCP server definitions -# - Git credential injection via GITHUB_PAT +# - MCP server configuration injectable at runtime via environment variables # - Non-root user compatible with restricted-v2 SCC -# Base image built from source (see Containerfile.base). -# Default points to quay.io/hmoghani — this will move to an org-owned -# registry (e.g., quay.io/opendatahub) once access is provisioned. -# To build the base image yourself: see Containerfile.base instructions. -ARG BASE_IMAGE=quay.io/hmoghani/codex-base:latest -FROM ${BASE_IMAGE} +FROM registry.access.redhat.com/ubi9/ubi-minimal@sha256:8201445bebcb5bd4fe23fcc2a76cd5fec029ab401d270926a1563c03b36f0137 + +ARG CODEX_VERSION=0.144.0 +ARG USER_NAME=codex-agent +ARG USER_UID=1001 +ARG USER_GID=0 LABEL name="codex" \ + version="${CODEX_VERSION}" \ vendor="Red Hat" \ summary="Codex CLI agent for OpenShift AI" \ description="RHOAI-flavored Codex CLI with entrypoint, MCP injection, and session persistence" \ license="Apache-2.0" -USER 0 +# System dependencies +# - git: Version control (agent clones repos, makes commits) +# - jq: JSON processor (used by entrypoint for MCP config conversion) +# - tar: Required by oc cp for file retrieval +# - nodejs + npm: Install Codex CLI and run npx-based MCP servers +# Note: curl-minimal is pre-installed in ubi-minimal; full curl conflicts +RUN microdnf install -y --nodocs \ + git \ + jq \ + tar \ + nodejs \ + npm \ + && microdnf clean all \ + && rm -rf /var/cache/yum + +# Create non-root user (OpenShift restricted-v2 SCC compatible) +RUN useradd -u ${USER_UID} -g ${USER_GID} -d /home/${USER_NAME} -m -s /bin/bash ${USER_NAME} + +# Workspace and config directories +RUN mkdir -p /workspace/projects /home/${USER_NAME}/.codex \ + && chown -R ${USER_UID}:${USER_GID} /workspace /home/${USER_NAME} \ + && chmod -R 775 /workspace /home/${USER_NAME} + +# Install Codex CLI via npm (pre-built binary, Apache 2.0) +# The npm package is a thin wrapper that downloads the platform-specific +# Rust binary. For a source-built alternative, see Containerfile.base. +RUN npm install -g @openai/codex@${CODEX_VERSION} + +# Verify +RUN codex --version + +ENV HOME="/home/${USER_NAME}" \ + PATH="/home/${USER_NAME}/.local/bin:${PATH}" -# Remove ~/.codex created during base build — entrypoint will create a symlink +# Remove ~/.codex created during install — entrypoint will create a symlink # to /workspace/.codex for PVC-backed persistence -RUN rm -rf /home/codex-agent/.codex +RUN rm -rf /home/${USER_NAME}/.codex COPY --chmod=755 entrypoint.sh /usr/local/bin/entrypoint.sh -USER 1001 +USER ${USER_UID} WORKDIR /workspace/projects ENTRYPOINT ["entrypoint.sh"] diff --git a/agents/codex/deployment/Containerfile.base b/agents/codex/deployment/Containerfile.base index 7f4039b2..cfd7b6b0 100644 --- a/agents/codex/deployment/Containerfile.base +++ b/agents/codex/deployment/Containerfile.base @@ -82,10 +82,12 @@ WORKDIR /src/codex-rs # during cross-compilation linking. Disable it for cross builds — # the binary is functionally identical, just slightly larger. RUN CARGO_PROFILE_RELEASE_LTO=false \ - cargo build --release --target x86_64-unknown-linux-gnu --bin codex --bin bwrap \ + cargo build --release --target x86_64-unknown-linux-gnu \ + --bin codex --bin bwrap --bin codex-code-mode-host \ && cp target/x86_64-unknown-linux-gnu/release/codex /opt/codex \ && cp target/x86_64-unknown-linux-gnu/release/bwrap /opt/bwrap \ - && chmod +x /opt/codex /opt/bwrap \ + && cp target/x86_64-unknown-linux-gnu/release/codex-code-mode-host /opt/codex-code-mode-host \ + && chmod +x /opt/codex /opt/bwrap /opt/codex-code-mode-host \ && rm -rf target /src /usr/local/cargo/registry # ============================================================================= @@ -126,6 +128,7 @@ RUN microdnf install -y --nodocs \ # Copy the compiled binaries from the builder COPY --from=builder /opt/codex /usr/local/bin/codex COPY --from=builder /opt/bwrap /usr/local/bin/bwrap +COPY --from=builder /opt/codex-code-mode-host /usr/local/bin/codex-code-mode-host # Create non-root user (OpenShift restricted-v2 SCC compatible) RUN useradd -u ${USER_UID} -g ${USER_GID} -d /home/${USER_NAME} -m -s /bin/bash ${USER_NAME} diff --git a/agents/codex/deployment/README.md b/agents/codex/deployment/README.md index 51a7bdae..c097b357 100644 --- a/agents/codex/deployment/README.md +++ b/agents/codex/deployment/README.md @@ -1,5 +1,11 @@ # Codex Deployment Files +> **Disclaimer:** The Codex CLI binary installed in this image is a pre-built +> artifact published by [OpenAI](https://github.com/openai/codex) via npm +> (`@openai/codex`), **not by Red Hat**. It has not been built, scanned, or +> validated according to Red Hat standards. Use it at your own discretion. +> To build from source instead, see `Containerfile.base`. + This directory contains Containerfiles, entrypoint, and deployment manifests for running [OpenAI Codex CLI](https://github.com/openai/codex) on OpenShift. For the full deployment guide, see the [Codex on OpenShift README](../README.md). @@ -8,8 +14,8 @@ For the full deployment guide, see the [Codex on OpenShift README](../README.md) | File | Description | |------|-------------| -| `Containerfile.base` | Base image build (multi-stage: compiles Codex from source, UBI 9 minimal runtime) | -| `Containerfile` | RHOAI flavor (extends base with entrypoint, MCP injection, session persistence) | +| `Containerfile` | RHOAI image (UBI 9 minimal, Codex via npm, entrypoint, MCP, session persistence) | +| `Containerfile.base` | Alternative: compiles Codex from source (Rust cross-compilation) | | `Containerfile.openshell` | OpenShell sandbox variant (extends openshell-base) | | `entrypoint.sh` | Container entrypoint (auth, model provider, MCP, git credential setup) | | `deployment.yaml` | OpenShift manifests (ImageStream, BuildConfig, ConfigMaps, PVC, Deployment) | @@ -20,30 +26,26 @@ For the full deployment guide, see the [Codex on OpenShift README](../README.md) |---------|---------------------|---------| | API key | `OPENAI_API_KEY` | (required) | | Inference endpoint | `OPENAI_BASE_URL` | (required for vLLM) | -| Model | `OPENAI_MODEL` | (provider default) | +| Model | `OPENAI_MODEL` | (required for vLLM) | | Config directory | `CODEX_HOME` | `/workspace/.codex` | | Sandbox mode | `CODEX_SANDBOX` | `danger-full-access` | | MCP config (TOML file) | `MCP_CONFIG_TOML` | (none) | | MCP config (inline JSON) | `MCP_CONFIG_JSON` | (none) | | GitHub PAT | `GITHUB_PAT` | (none) | -## Build Order +## Build ```bash -# 1. Build base image -podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . - -# 2. Build RHOAI flavor -podman build --platform linux/amd64 -t codex:latest -f Containerfile \ - --build-arg BASE_IMAGE=codex-base:latest . +podman build --platform linux/amd64 -t codex:latest -f Containerfile . ``` -## Comparison: RHOAI vs OpenShell +## Comparison: RHOAI vs Source vs OpenShell -| | RHOAI (Containerfile) | OpenShell (Containerfile.openshell) | -|--|----------------------|-------------------------------------| -| **Base** | UBI 9 minimal (multi-stage) | openshell-base | -| **Codex install** | Compiled from source (Rust) | npm install in image | -| **Runtime** | Standalone pod, `oc exec` | OpenShell gateway sandbox | -| **Auth** | Env vars, entrypoint setup | Manual inside sandbox | -| **Use case** | Production RHOAI deployment | Sandboxed experimentation | +| | RHOAI (Containerfile) | Source (Containerfile.base) | OpenShell (Containerfile.openshell) | +|--|----------------------|---------------------------|-------------------------------------| +| **Base** | UBI 9 minimal | UBI 9 minimal (multi-stage) | openshell-base | +| **Codex install** | npm (pre-built binary) | Compiled from source (Rust) | npm install in image | +| **Build time** | ~1 min | ~12 min (32GB RAM required) | ~1 min | +| **Runtime** | Standalone pod, `oc exec` | Standalone pod, `oc exec` | OpenShell gateway sandbox | +| **Auth** | Env vars, entrypoint setup | Env vars, entrypoint setup | Manual inside sandbox | +| **Use case** | Production RHOAI deployment | Supply chain auditable build | Sandboxed experimentation | diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index cbbe07ce..18b43c22 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -180,6 +180,14 @@ spec: key: api-key - name: OPENAI_MODEL value: "your-model-id" + # CODEX_API_KEY is required for codex exec (headless mode). + # The entrypoint propagates OPENAI_API_KEY, but only when + # running codex directly — not in sleep infinity + oc exec mode. + - name: CODEX_API_KEY + valueFrom: + secretKeyRef: + name: codex-api-key + key: api-key # ── Sandbox ────────────────────────────────────────────── # Container itself is the isolation boundary From 51457741bdf4527424349f7c7fcd58f2773c3dd2 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Tue, 21 Jul 2026 14:33:31 -0400 Subject: [PATCH 12/25] docs: update AGENTS.md codex entry to reflect npm-based build The primary Containerfile uses npm (not two-stage source build). Containerfile.base remains as a source-built alternative. Co-Authored-By: Claude Opus 4.6 (1M context) --- AGENTS.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/AGENTS.md b/AGENTS.md index b5750e6c..7974df44 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,7 +74,7 @@ Several agents diverge significantly from the standard pattern: **openclaw/deployment** - Kustomize-based deployment of OpenClaw on OpenShift. Uses pre-built image (`ghcr.io/openclaw/openclaw:latest`), not a Dockerfile. Kustomize overlays for customization (model endpoint, storage class). No Makefile, no FastAPI, no src/. Port 18789 (gateway). -**codex/deployment** - CLI-based coding agent (OpenAI Codex CLI, Apache 2.0). Two-stage Containerfile: `Containerfile.base` cross-compiles the Rust binary from source, `Containerfile` extends with RHOAI entrypoint. No Makefile, no pyproject.toml, no FastAPI, no `/chat/completions`, no tests/. Uses `entrypoint.sh` for runtime config, `sleep infinity` for `oc exec` access. Requires vLLM with Responses API (`/v1/responses`) and non-harmony models. +**codex/deployment** - CLI-based coding agent (OpenAI Codex CLI, Apache 2.0). `Containerfile` installs Codex via npm (pre-built upstream binary with disclaimer); `Containerfile.base` available as source-built alternative. No Makefile, no pyproject.toml, no FastAPI, no `/chat/completions`, no tests/. Uses `entrypoint.sh` for runtime config, `sleep infinity` for `oc exec` access. Requires vLLM with Responses API (`/v1/responses`) and non-harmony models. ## Common gotchas From 910756d15a04166737de4d3487fd6998cdf119de Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Tue, 21 Jul 2026 14:44:32 -0400 Subject: [PATCH 13/25] fix(codex): fix stale references in deployment.yaml and Containerfile.base - deployment.yaml: remove two-stage build instructions from BuildConfig comment, fix harmony model note, use bash -l in usage example - Containerfile.base: update header to describe it as alternative to npm-based build, remove reference to nonexistent Containerfile.mlflow Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/deployment/Containerfile.base | 4 ++-- agents/codex/deployment/deployment.yaml | 8 +++----- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/agents/codex/deployment/Containerfile.base b/agents/codex/deployment/Containerfile.base index cfd7b6b0..2e71a674 100644 --- a/agents/codex/deployment/Containerfile.base +++ b/agents/codex/deployment/Containerfile.base @@ -1,8 +1,8 @@ # Codex CLI Base Image # # Multi-stage build that compiles Codex CLI from source (Apache 2.0) -# and produces a clean UBI 9 minimal runtime image. Extended by flavor -# images (Containerfile, Containerfile.mlflow, etc.). +# and produces a clean UBI 9 minimal runtime image. This is an alternative +# to the npm-based Containerfile for supply chain auditable builds. # # The builder uses cross-compilation so it runs natively on the host # (e.g., ARM64 on Apple Silicon) while producing an x86_64 binary. diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index 18b43c22..8d43ab38 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -6,7 +6,7 @@ # Prerequisites: # - oc CLI logged into an OpenShift cluster # - A vLLM endpoint serving /v1/responses with tool calling enabled -# (use --tool-call-parser qwen3_coder or hermes, NOT openai) +# (non-harmony models only — harmony/gpt-oss models do not support namespace tools) # # Usage: # oc new-project @@ -15,7 +15,7 @@ # oc rollout status deployment/codex # # Then connect: -# oc exec -it deployment/codex -- bash +# oc exec -it deployment/codex -- bash -l # codex-run exec "What files are in the current directory?" # # Update the OPENAI_BASE_URL, OPENAI_MODEL, and OPENAI_API_KEY below @@ -42,9 +42,7 @@ spec: # Recommended: build locally and push to the internal registry. # See the README for full instructions. # -# podman build --platform linux/amd64 -t codex-base:latest -f Containerfile.base . -# podman build --platform linux/amd64 -t codex:latest -f Containerfile \ -# --build-arg BASE_IMAGE=codex-base:latest . +# podman build --platform linux/amd64 -t codex:latest -f Containerfile . # podman push //codex:latest apiVersion: build.openshift.io/v1 kind: BuildConfig From 2e8904770ef748936aa702f8acf782c7d39b4d1f Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Wed, 22 Jul 2026 19:44:22 -0400 Subject: [PATCH 14/25] docs(codex): document vLLM tool calling limitation with Qwen3 models Tested Qwen3-8B and Qwen3-32B on vLLM v0.25.1: simple prompts work but tool execution fails. The models emit tool calls as text ( XML tags) instead of structured function_call objects. vLLM's Responses API parser doesn't convert them. Tool-dependent workflows require an OpenAI backend until vLLM fixes this. Added compatibility table, upstream issue links, and updated Known Limitations section. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 16 +++++++++++++--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index de762842..3c4fb4d4 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -533,7 +533,17 @@ vllm serve Qwen/Qwen3-8B \ See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/integrations/codex/) for the full list of supported models and parsers. -**Note:** Full end-to-end Codex functionality (tool execution with visible output) requires vLLM v0.24.0 or later. +**Tool calling on vLLM with open-source models:** + +Codex connects and receives responses from open-source models on vLLM, but tool execution (file creation, shell commands) does not work reliably. The model generates tool calls as **text** (e.g., `` XML tags inside `output_text`) instead of structured `function_call` output items. vLLM's Responses API parser does not convert these into the structured format Codex expects. This was tested with both Qwen3-8B and Qwen3-32B on vLLM v0.25.1 — simple prompts work, but tool execution fails silently. + +| Model | Simple Prompts | Tool Execution | Notes | +|---|---|---|---| +| Qwen3-8B | Works | No | Model reasons but rarely emits tool calls; 8B is borderline for agentic tasks | +| Qwen3-32B | Works | No | Model emits tool calls as text (`` tags), but vLLM doesn't parse them into structured function calls on the Responses API path | +| OpenAI models (via OpenAI API) | Works | Works | Native Responses API with proper tool call parsing | + +This is a vLLM Responses API limitation — the `qwen3_coder` parser likely works for the Chat Completions API but not the Responses API path. Upstream Codex issues tracking this: [#32318](https://github.com/openai/codex/issues/32318), [#31875](https://github.com/openai/codex/issues/31875), [#31882](https://github.com/openai/codex/issues/31882). ### Network Connectivity @@ -642,9 +652,9 @@ oc delete project my-codex-project Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models (Qwen3, Llama, etc.) instead. Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. -### Open Source Model Quality +### Open Source Model Tool Calling -Codex CLI is designed for OpenAI's models. Open source models may produce lower quality results, particularly for complex multi-step tasks and tool use chains. Including language runtimes in the container helps because the agent can run tests and iterate on its own output. +Codex CLI is designed for OpenAI's models. When using open-source models on vLLM, simple prompts (Q&A, reasoning) work, but **tool execution does not**. The models generate tool calls as text rather than structured function call objects, and vLLM's Responses API parser does not convert them. This was tested with Qwen3-8B and Qwen3-32B on vLLM v0.25.1. Until vLLM's Responses API parser supports Qwen3's tool call format, tool-dependent workflows (file creation, shell commands) require an OpenAI backend. ### No Built-in Permission System From dfc9029aa930e96696a392e5f335937c558ce0a1 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Wed, 22 Jul 2026 19:52:50 -0400 Subject: [PATCH 15/25] docs(codex): add upstream vLLM bug reference for harmony namespace tools Link to vllm-project/vllm#49493 in the Responses API Compatibility section so users can track the fix. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 3c4fb4d4..69b05bc9 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -519,7 +519,7 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | Qwen3-8B | `qwen3_coder` | Upstream v0.24.0 | Works | | Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Works | -The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error occurs on all vLLM versions and all parsers. +The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error occurs on all vLLM versions and all parsers. Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). **To use Codex with vLLM, use an open-source model:** From ed5d5d4650a0ea555d5004e464a708bb3fc6faa6 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 12:04:19 -0400 Subject: [PATCH 16/25] =?UTF-8?q?docs(codex):=20update=20compatibility=20?= =?UTF-8?q?=E2=80=94=20Qwen3.6-27B=20and=20Qwen3-32B=20tool=20calling=20wo?= =?UTF-8?q?rks?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Tested findings: - Qwen3.6-27B + qwen3_coder: full tool calling on upstream vLLM v0.25.1 - Qwen3-32B + hermes: full tool calling on upstream vLLM v0.25.1 - qwen3_coder is for Qwen3.6+; older Qwen3 models need hermes parser - RHOAI 3.5 EA (v0.21.0) fails with "Unexpected message role" — needs upstream v0.25.1+ or RHOAI 3.6 EA (expected v0.26+) Updated compatibility tables, parser guidance, vLLM version requirements, example commands, and Known Limitations. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 53 ++++++++++++++++++------- agents/codex/deployment/deployment.yaml | 4 +- 2 files changed, 40 insertions(+), 17 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 69b05bc9..1736bc04 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -36,7 +36,7 @@ Codex CLI is released under the [Apache 2.0 license](https://github.com/openai/c - `podman` installed locally (on macOS, also run `podman machine init` and `podman machine start`) - `oc` CLI installed and logged into your OpenShift cluster -- A vLLM endpoint serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** harmony models (gpt-oss) do not yet support namespace tools in vLLM — use a non-harmony model (e.g., Qwen3, Llama). See [Responses API Compatibility](#responses-api-compatibility). +- A vLLM endpoint (upstream v0.25.1+) serving the OpenAI Responses API at `/v1/responses` with tool calling enabled (`--enable-auto-tool-choice --tool-call-parser `). **Important:** harmony models (gpt-oss) do not support namespace tools — use a non-harmony model. See [Responses API Compatibility](#responses-api-compatibility) for tested model/parser combinations. Codex CLI uses the **Responses API exclusively**. It does not use the Chat Completions API (`/v1/chat/completions`). Your vLLM server must support `/v1/responses` and tool calling with namespace tool types. @@ -308,6 +308,7 @@ The entrypoint auto-generates model provider configuration from the `OPENAI_BASE ```toml # Auto-configured by entrypoint.sh model = "your-model-id" +model_supports_reasoning_summaries = false model_provider = "vllm" [model_providers.vllm] @@ -315,6 +316,7 @@ name = "vLLM" base_url = "http://vllm-svc.namespace.svc.cluster.local/v1" env_key = "OPENAI_API_KEY" wire_api = "responses" +supports_websockets = false ``` **Manual config.toml example** (for advanced setups, place in the ConfigMap): @@ -516,18 +518,28 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | gpt-oss-120b | `openai` | RHOAI v0.21.0 | `"tool type namespace not supported"` | | gpt-oss-120b | `openai` | Upstream v0.25.1 | `"tool type namespace not supported"` | | gpt-oss-120b | `hermes` | Upstream v0.25.1 | `"tool type namespace not supported"` | -| Qwen3-8B | `qwen3_coder` | Upstream v0.24.0 | Works | -| Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Works | +| Qwen3.6-27B | `qwen3_coder` | RHOAI v0.21.0 | `"Unexpected message role"` (needs v0.25.1+) | +| Qwen3.6-27B | `qwen3_coder` | Upstream v0.25.1 | Works (full tool calling) | +| Qwen3-32B | `hermes` | Upstream v0.25.1 | Works (full tool calling) | +| Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Namespace accepted, but no tool execution (8B too small) | The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error occurs on all vLLM versions and all parsers. Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). -**To use Codex with vLLM, use an open-source model:** +**To use Codex with vLLM, use an open-source model with the correct parser:** ```bash -# Example: Qwen3-8B (tested, works with Codex) -vllm serve Qwen/Qwen3-8B \ +# Qwen3.6-27B (tested, full tool calling works) +vllm serve Qwen/Qwen3.6-27B \ + --trust-remote-code \ --enable-auto-tool-choice \ --tool-call-parser qwen3_coder \ + --reasoning-parser qwen3 \ + --mm-encoder-tp-mode data + +# Qwen3-32B (use hermes parser, not qwen3_coder) +vllm serve Qwen/Qwen3-32B \ + --enable-auto-tool-choice \ + --tool-call-parser hermes \ --reasoning-parser qwen3 ``` @@ -535,15 +547,20 @@ See the [vLLM Codex integration docs](https://docs.vllm.ai/en/latest/serving/int **Tool calling on vLLM with open-source models:** -Codex connects and receives responses from open-source models on vLLM, but tool execution (file creation, shell commands) does not work reliably. The model generates tool calls as **text** (e.g., `` XML tags inside `output_text`) instead of structured `function_call` output items. vLLM's Responses API parser does not convert these into the structured format Codex expects. This was tested with both Qwen3-8B and Qwen3-32B on vLLM v0.25.1 — simple prompts work, but tool execution fails silently. +Tool execution depends on the model and parser combination. The entrypoint sets `supports_websockets = false` and `model_supports_reasoning_summaries = false` in the provider config, which are required for vLLM backends. -| Model | Simple Prompts | Tool Execution | Notes | -|---|---|---|---| -| Qwen3-8B | Works | No | Model reasons but rarely emits tool calls; 8B is borderline for agentic tasks | -| Qwen3-32B | Works | No | Model emits tool calls as text (`` tags), but vLLM doesn't parse them into structured function calls on the Responses API path | -| OpenAI models (via OpenAI API) | Works | Works | Native Responses API with proper tool call parsing | +| Model | Parser | Simple Prompts | Tool Execution | Notes | +|---|---|---|---|---| +| Qwen3.6-27B | `qwen3_coder` | Works | **Works** | Tested on upstream vLLM v0.25.1 — full tool calling including file creation and shell commands | +| Qwen3-32B | `hermes` | Works | **Works** | Tested on upstream vLLM v0.25.1 — `hermes` parser extracts `` tags correctly | +| Qwen3-32B | `qwen3_coder` | Works | No | `qwen3_coder` parser does not extract Qwen3-32B's `` format — use `hermes` instead | +| Qwen3-8B | `qwen3_coder` | Works | No | 8B is too small for reliable tool calling | +| Nemotron-3-Super-120B | — | Works | Works | Reported working by vLLM engineers | +| OpenAI models (via OpenAI API) | — | Works | Works | Native Responses API | -This is a vLLM Responses API limitation — the `qwen3_coder` parser likely works for the Chat Completions API but not the Responses API path. Upstream Codex issues tracking this: [#32318](https://github.com/openai/codex/issues/32318), [#31875](https://github.com/openai/codex/issues/31875), [#31882](https://github.com/openai/codex/issues/31882). +**Parser selection matters:** Newer Qwen models (3.6+) work with `qwen3_coder`. Older Qwen3 models (32B, 8B) need `hermes`. Using the wrong parser results in tool calls emitted as text instead of structured function calls. + +**vLLM version requirements:** Upstream vLLM v0.25.1+ is required. RHOAI vLLM 3.5 EA (v0.21.0) returns `"Unexpected message role"` with newer models like Qwen3.6-27B because it doesn't support the `developer` message role. RHOAI 3.6 EA (expected to be based on v0.26+) should include this fix. ### Network Connectivity @@ -650,11 +667,17 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models (Qwen3, Llama, etc.) instead. Full end-to-end Codex functionality requires vLLM v0.24.0+. See [Responses API Compatibility](#responses-api-compatibility) for details. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models instead. Requires upstream vLLM v0.25.1+ (RHOAI 3.5 EA v0.21.0 does not support the `developer` message role). See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open Source Model Tool Calling -Codex CLI is designed for OpenAI's models. When using open-source models on vLLM, simple prompts (Q&A, reasoning) work, but **tool execution does not**. The models generate tool calls as text rather than structured function call objects, and vLLM's Responses API parser does not convert them. This was tested with Qwen3-8B and Qwen3-32B on vLLM v0.25.1. Until vLLM's Responses API parser supports Qwen3's tool call format, tool-dependent workflows (file creation, shell commands) require an OpenAI backend. +Tool calling with open-source models works but requires the correct model + parser combination. Tested configurations: +- **Qwen3.6-27B + `qwen3_coder`**: Full tool calling (file creation, shell commands) — tested and working +- **Qwen3-32B + `hermes`**: Full tool calling — tested and working +- **Qwen3-32B + `qwen3_coder`**: Tool calls emitted as text, not parsed — use `hermes` instead +- **Qwen3-8B**: Too small for reliable agentic tool calling + +See [Responses API Compatibility](#responses-api-compatibility) for the full compatibility table. ### No Built-in Permission System diff --git a/agents/codex/deployment/deployment.yaml b/agents/codex/deployment/deployment.yaml index 8d43ab38..75a08b0f 100644 --- a/agents/codex/deployment/deployment.yaml +++ b/agents/codex/deployment/deployment.yaml @@ -5,8 +5,8 @@ # # Prerequisites: # - oc CLI logged into an OpenShift cluster -# - A vLLM endpoint serving /v1/responses with tool calling enabled -# (non-harmony models only — harmony/gpt-oss models do not support namespace tools) +# - Upstream vLLM v0.25.1+ serving /v1/responses with tool calling enabled +# (non-harmony models only — tested: Qwen3.6-27B + qwen3_coder, Qwen3-32B + hermes) # # Usage: # oc new-project From 001b26e095d30055138f1f8b8b105c996372978c Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 12:06:46 -0400 Subject: [PATCH 17/25] docs(codex): remove untested Nemotron-3 from compatibility table Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 1 - 1 file changed, 1 deletion(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 1736bc04..7c9049cf 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -555,7 +555,6 @@ Tool execution depends on the model and parser combination. The entrypoint sets | Qwen3-32B | `hermes` | Works | **Works** | Tested on upstream vLLM v0.25.1 — `hermes` parser extracts `` tags correctly | | Qwen3-32B | `qwen3_coder` | Works | No | `qwen3_coder` parser does not extract Qwen3-32B's `` format — use `hermes` instead | | Qwen3-8B | `qwen3_coder` | Works | No | 8B is too small for reliable tool calling | -| Nemotron-3-Super-120B | — | Works | Works | Reported working by vLLM engineers | | OpenAI models (via OpenAI API) | — | Works | Works | Native Responses API | **Parser selection matters:** Newer Qwen models (3.6+) work with `qwen3_coder`. Older Qwen3 models (32B, 8B) need `hermes`. Using the wrong parser results in tool calls emitted as text instead of structured function calls. From 9c886ace717a4bac00575b86d6ba2583d0a6b40b Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 12:10:27 -0400 Subject: [PATCH 18/25] fix(codex): clarify RHOAI developer role issue applies to all models The "Unexpected message role" error on RHOAI v0.21.0 is a Codex + vLLM compatibility issue (Codex sends developer role messages), not model-specific. Updated tables and version requirements accordingly. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 7c9049cf..9f9b706a 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -518,7 +518,7 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | gpt-oss-120b | `openai` | RHOAI v0.21.0 | `"tool type namespace not supported"` | | gpt-oss-120b | `openai` | Upstream v0.25.1 | `"tool type namespace not supported"` | | gpt-oss-120b | `hermes` | Upstream v0.25.1 | `"tool type namespace not supported"` | -| Qwen3.6-27B | `qwen3_coder` | RHOAI v0.21.0 | `"Unexpected message role"` (needs v0.25.1+) | +| Any model | any | RHOAI v0.21.0 | `"Unexpected message role"` — Codex sends `developer` role which this version doesn't support | | Qwen3.6-27B | `qwen3_coder` | Upstream v0.25.1 | Works (full tool calling) | | Qwen3-32B | `hermes` | Upstream v0.25.1 | Works (full tool calling) | | Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Namespace accepted, but no tool execution (8B too small) | @@ -559,7 +559,7 @@ Tool execution depends on the model and parser combination. The entrypoint sets **Parser selection matters:** Newer Qwen models (3.6+) work with `qwen3_coder`. Older Qwen3 models (32B, 8B) need `hermes`. Using the wrong parser results in tool calls emitted as text instead of structured function calls. -**vLLM version requirements:** Upstream vLLM v0.25.1+ is required. RHOAI vLLM 3.5 EA (v0.21.0) returns `"Unexpected message role"` with newer models like Qwen3.6-27B because it doesn't support the `developer` message role. RHOAI 3.6 EA (expected to be based on v0.26+) should include this fix. +**vLLM version requirements:** Upstream vLLM v0.25.1+ is required. RHOAI vLLM 3.5 EA (v0.21.0) returns `"Unexpected message role"` because Codex sends `developer` role messages which that version doesn't support (fixed upstream in [PR #43590](https://github.com/vllm-project/vllm/pull/43590)). RHOAI 3.6 EA (expected to be based on v0.26+) should include this fix. ### Network Connectivity From defff39daeb748ced7e8d46042c33ddf6c65fa9f Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 12:11:53 -0400 Subject: [PATCH 19/25] =?UTF-8?q?fix(codex):=20address=20CodeRabbit=20?= =?UTF-8?q?=E2=80=94=20qualify=20version=20claim,=20fix=20markdown=20lint?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Qualify "all vLLM versions" to "all tested vLLM versions" for the harmony namespace tool error - Hyphenate "Open-Source" in heading - Add blank line before list (MD032) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 9f9b706a..4f9c5fb2 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -523,7 +523,7 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | Qwen3-32B | `hermes` | Upstream v0.25.1 | Works (full tool calling) | | Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Namespace accepted, but no tool execution (8B too small) | -The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error occurs on all vLLM versions and all parsers. Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). +The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error was observed on all tested vLLM versions (RHOAI v0.21.0, upstream v0.25.1) and parsers (`openai`, `hermes`). Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). **To use Codex with vLLM, use an open-source model with the correct parser:** @@ -668,9 +668,10 @@ oc delete project my-codex-project Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models instead. Requires upstream vLLM v0.25.1+ (RHOAI 3.5 EA v0.21.0 does not support the `developer` message role). See [Responses API Compatibility](#responses-api-compatibility) for details. -### Open Source Model Tool Calling +### Open-Source Model Tool Calling Tool calling with open-source models works but requires the correct model + parser combination. Tested configurations: + - **Qwen3.6-27B + `qwen3_coder`**: Full tool calling (file creation, shell commands) — tested and working - **Qwen3-32B + `hermes`**: Full tool calling — tested and working - **Qwen3-32B + `qwen3_coder`**: Tool calls emitted as text, not parsed — use `hermes` instead From 2ef0cdc72d24f1e7246544d41659554901eda268 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 12:14:34 -0400 Subject: [PATCH 20/25] fix(codex): remove unused temp file cleanup from entrypoint The TEMP_FILES array and cleanup_temp_files trap were unused dead code from an earlier MCP injection approach that wrote temp files. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/deployment/entrypoint.sh | 13 ++----------- 1 file changed, 2 insertions(+), 11 deletions(-) diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index a711346d..db89db4b 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -37,17 +37,6 @@ set -euo pipefail -# Temp files to clean up on exit -TEMP_FILES=() - -cleanup_temp_files() { - for f in "${TEMP_FILES[@]}"; do - rm -f "$f" 2>/dev/null || true - done -} - -trap cleanup_temp_files EXIT - # ============================================================================= # Helper Functions # ============================================================================= @@ -190,6 +179,7 @@ setup_model_provider() { cat >> "${config_file}" < Date: Thu, 23 Jul 2026 12:16:14 -0400 Subject: [PATCH 21/25] fix(codex): replace npmjs.com link with GitHub releases (403 on link-check) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 4f9c5fb2..eea52b42 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -68,7 +68,7 @@ Always pass `--platform linux/amd64` when building on Apple Silicon — the npm ### Version pinning -The `CODEX_VERSION` build arg controls which npm package version to install. It defaults to `0.144.0`. Available versions are listed at [npmjs.com/package/@openai/codex](https://www.npmjs.com/package/@openai/codex). +The `CODEX_VERSION` build arg controls which npm package version to install. It defaults to `0.144.0`. Available versions are listed at [github.com/openai/codex/releases](https://github.com/openai/codex/releases). ### Image size From 4de326353c036909de992043db57426e6f97c2bd Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Thu, 23 Jul 2026 14:40:03 -0400 Subject: [PATCH 22/25] fix(codex): insert model/provider keys at top of config.toml, not end When Codex writes runtime sections like [tui.model_availability_nux] or [projects."..."] to config.toml, appending model= at the end places it inside those sections instead of at the top level, causing TOML parse errors on restart. Use sed -i '1i' to insert top-level keys at the beginning. The [model_providers.vllm] section block is still appended since it's a section header. (Reported by msager27) Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/deployment/entrypoint.sh | 16 ++++++++++------ 1 file changed, 10 insertions(+), 6 deletions(-) diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index db89db4b..f52aac3b 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -174,15 +174,19 @@ setup_model_provider() { base_url=$(sanitize_value "${OPENAI_BASE_URL}") # Remove any existing model/provider lines so env vars always win - sed -i '/^model\s*=/d; /^model_provider\s*=/d' "${config_file}" 2>/dev/null || true + sed -i '/^model\s*=/d; /^model_provider\s*=/d; /^model_supports_reasoning_summaries\s*=/d' "${config_file}" 2>/dev/null || true sed -i '/^\[model_providers\./,/^$/d' "${config_file}" 2>/dev/null || true - cat >> "${config_file}" <> "${config_file}" < Date: Thu, 23 Jul 2026 15:46:43 -0400 Subject: [PATCH 23/25] =?UTF-8?q?fix(codex):=20address=20msager27=20feedba?= =?UTF-8?q?ck=20=E2=80=94=20OGX=20note,=20consistent=20RHOAI=20naming?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add OGX to namespace tool error description (same error occurs) - Use consistent "RHOAI 3.5 EA (vLLM v0.21.0)" naming throughout instead of bare "RHOAI v0.21.0" Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index eea52b42..80045758 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -515,15 +515,15 @@ Codex sends `namespace` tool types with every request. Whether vLLM accepts them | Model | Parser | vLLM Version | Result | |---|---|---|---| -| gpt-oss-120b | `openai` | RHOAI v0.21.0 | `"tool type namespace not supported"` | +| gpt-oss-120b | `openai` | RHOAI 3.5 EA (vLLM v0.21.0) | `"tool type namespace not supported"` | | gpt-oss-120b | `openai` | Upstream v0.25.1 | `"tool type namespace not supported"` | | gpt-oss-120b | `hermes` | Upstream v0.25.1 | `"tool type namespace not supported"` | -| Any model | any | RHOAI v0.21.0 | `"Unexpected message role"` — Codex sends `developer` role which this version doesn't support | +| Any model | any | RHOAI 3.5 EA (vLLM v0.21.0) | `"Unexpected message role"` — Codex sends `developer` role which this version doesn't support | | Qwen3.6-27B | `qwen3_coder` | Upstream v0.25.1 | Works (full tool calling) | | Qwen3-32B | `hermes` | Upstream v0.25.1 | Works (full tool calling) | | Qwen3-8B | `qwen3_coder` | Upstream v0.25.1 | Namespace accepted, but no tool execution (8B too small) | -The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error was observed on all tested vLLM versions (RHOAI v0.21.0, upstream v0.25.1) and parsers (`openai`, `hermes`). Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). +The `"tool type namespace not supported"` error is specific to **harmony models** (gpt-oss family / `GptOssForCausalLM` architecture). This is a known vLLM limitation — namespace tool types are supported for non-harmony (open-source) models but not yet implemented for harmony models. The error was observed on all tested vLLM versions (RHOAI 3.5 EA (vLLM v0.21.0), upstream v0.25.1) and parsers (`openai`, `hermes`). The same error occurs when using OGX as the inference provider. Tracked upstream: [vllm-project/vllm#49493](https://github.com/vllm-project/vllm/issues/49493). **To use Codex with vLLM, use an open-source model with the correct parser:** @@ -666,7 +666,7 @@ oc delete project my-codex-project ### Responses API Only -Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models instead. Requires upstream vLLM v0.25.1+ (RHOAI 3.5 EA v0.21.0 does not support the `developer` message role). See [Responses API Compatibility](#responses-api-compatibility) for details. +Codex CLI uses the OpenAI Responses API (`/v1/responses`) exclusively. It does not support the Chat Completions API (`/v1/chat/completions`) or the Anthropic Messages API (`/v1/messages`). Codex sends `namespace` tool types with every request. Namespace tools are a known unsupported feature for harmony models (gpt-oss) in vLLM — use non-harmony models instead. Requires upstream vLLM v0.25.1+ (RHOAI 3.5 EA (vLLM v0.21.0) does not support the `developer` message role). See [Responses API Compatibility](#responses-api-compatibility) for details. ### Open-Source Model Tool Calling From c710d1155bd26c18e9fac104cc6c83c2f6e4553c Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Mon, 27 Jul 2026 15:25:35 -0400 Subject: [PATCH 24/25] feat(codex): add OpenShell sandbox guide, fix remaining review findings OpenShell sandbox: - Update Containerfile.openshell to use org-owned base image (quay.io/aipcc/agentic-ci/openshell:0.3.27, RHEL 10) - Use dnf instead of microdnf (RHEL 10) - Add comprehensive OpenShell setup guide to README: Helm install, mTLS cert extraction, sandbox creation, Codex testing - Tested on OpenShell v0.0.86, OpenShift 4.21, Qwen3.6-27B Review fixes: - Sanitize OPENAI_MODEL in fallback path (shricharan-ks #1) - Fix MCP JSON-to-TOML quoting with jq @json (shricharan-ks #2) - Update deployment/README.md references Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 132 +++++++++++++++++- .../codex/deployment/Containerfile.openshell | 11 +- agents/codex/deployment/README.md | 4 +- agents/codex/deployment/entrypoint.sh | 4 +- 4 files changed, 139 insertions(+), 12 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index 80045758..afb02d2a 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -23,6 +23,7 @@ A guide to deploying [OpenAI Codex CLI](https://github.com/openai/codex) as a co - [Troubleshooting](#troubleshooting) - [Cleanup](#cleanup) - [Known Limitations](#known-limitations) +- [OpenShell Sandbox](#openshell-sandbox) --- @@ -687,6 +688,133 @@ Unlike Claude Code, Codex CLI does not have a granular permission system for fil Codex CLI uses TOML for all configuration (`config.toml`). There is no JSON settings file equivalent. The entrypoint's `MCP_CONFIG_JSON` support converts JSON to TOML at startup for compatibility, but only a subset of fields (`url`, `command`, `args`) are converted. For full MCP server configuration (e.g., `bearer_token_env_var`, `env` blocks), use native TOML. -### OpenShell Sandbox Variant +--- + +## OpenShell Sandbox + +[NVIDIA OpenShell](https://github.com/NVIDIA/OpenShell) is an open-source sandbox runtime for AI agents. The `Containerfile.openshell` builds a Codex sandbox image based on `quay.io/aipcc/agentic-ci/openshell` (RHEL 10). This variant does not include the RHOAI entrypoint or session persistence — it is intended for ephemeral sandbox sessions. + +> Tested on: OpenShell gateway v0.0.86, OpenShift 4.21, Codex CLI v0.144.0, Qwen3.6-27B. + +#### Prerequisites + +- [Helm](https://helm.sh/) 3+ installed locally +- [OpenShell CLI](https://docs.nvidia.com/openshell/get-started/quickstart) installed locally (`openshell --version` should print `0.0.58` or later) +- The Kubernetes Agent Sandbox CRD must be installed: `oc get crd sandboxes.agents.x-k8s.io` +- A vLLM endpoint reachable from within the cluster + +#### Step 1: Install the OpenShell gateway + +```bash +export OPENSHELL_NAME=openshell + +# On shared clusters, use a unique name to avoid ClusterRole conflicts: +# export OPENSHELL_NAME=openshell- + +helm install openshell oci://ghcr.io/nvidia/openshell/helm-chart \ + --version 0.0.86 \ + --namespace \ + --set podSecurityContext.fsGroup=null \ + --set securityContext.runAsUser=null \ + --set server.auth.allowUnauthenticatedUsers=true \ + --set fullnameOverride=${OPENSHELL_NAME} + +oc rollout status statefulset/${OPENSHELL_NAME} --timeout=120s +``` + +Grant the privileged SCC to the sandbox service account: + +```bash +oc adm policy add-scc-to-user privileged -z ${OPENSHELL_NAME}-sandbox -n +``` + +#### Step 2: Connect the OpenShell CLI + +Extract the mTLS certificates and register the gateway: + +```bash +(umask 077 && mkdir -p ~/.config/openshell/gateways/codex-cluster/mtls) + +oc get secret openshell-client-tls \ + -o jsonpath='{.data.ca\.crt}' | base64 -d > ~/.config/openshell/gateways/codex-cluster/mtls/ca.crt + +oc get secret openshell-client-tls \ + -o jsonpath='{.data.tls\.crt}' | base64 -d > ~/.config/openshell/gateways/codex-cluster/mtls/tls.crt + +(umask 077 && oc get secret openshell-client-tls \ + -o jsonpath='{.data.tls\.key}' | base64 -d > ~/.config/openshell/gateways/codex-cluster/mtls/tls.key) -The `Containerfile.openshell` builds a separate image for the OpenShell sandbox environment. This variant uses a different base image (`openshell-base`) and does not include the RHOAI entrypoint or session persistence. It is intended for ephemeral sandbox sessions, not persistent OpenShift deployments. +oc port-forward svc/${OPENSHELL_NAME} 18080:8080 & +openshell gateway add https://127.0.0.1:18080 --local --name codex-cluster +openshell status +``` + +#### Step 3: Build the sandbox image + +Build inside the cluster so the internal registry can serve it to sandbox pods: + +```bash +cd agents/codex/deployment + +oc create imagestream codex-sandbox + +cat <<'EOF' | oc apply -f - +apiVersion: build.openshift.io/v1 +kind: BuildConfig +metadata: + name: codex-sandbox +spec: + output: + to: + kind: ImageStreamTag + name: codex-sandbox:latest + source: + type: Binary + strategy: + dockerStrategy: + dockerfilePath: Containerfile.openshell + type: Docker +EOF + +oc start-build codex-sandbox --from-dir=. --follow +``` + +#### Step 4: Create a sandbox and test + +```bash +openshell sandbox create \ + --name codex \ + --from image-registry.openshift-image-registry.svc:5000//codex-sandbox:latest \ + -- sleep infinity + +openshell sandbox list +``` + +Test Codex inside the sandbox: + +```bash +oc exec codex -c agent -- bash -c ' + export OPENAI_BASE_URL="http://..svc.cluster.local/v1" + export OPENAI_API_KEY="dummy" + export CODEX_API_KEY="dummy" + codex exec \ + -c model_provider=vllm \ + -c model="" \ + -c model_providers.vllm.name="vLLM" \ + -c model_providers.vllm.base_url="${OPENAI_BASE_URL}" \ + -c model_providers.vllm.supports_websockets=false \ + -c model_supports_reasoning_summaries=false \ + --sandbox danger-full-access \ + --skip-git-repo-check \ + "What is 2+2? Reply with just the number." +' +``` + +#### Cleanup + +```bash +openshell sandbox delete codex +helm uninstall openshell -n +oc delete imagestream codex-sandbox +oc delete buildconfig codex-sandbox +``` diff --git a/agents/codex/deployment/Containerfile.openshell b/agents/codex/deployment/Containerfile.openshell index a315ba10..0b0331f8 100644 --- a/agents/codex/deployment/Containerfile.openshell +++ b/agents/codex/deployment/Containerfile.openshell @@ -1,19 +1,18 @@ # Codex agent for OpenShell sandbox # -# Flavor image built on the shared openshell-base. +# Flavor image built on the OpenShell base (RHEL 10). # Adds Node.js and Codex CLI (Apache 2.0). # # Build: # podman build --platform linux/amd64 -t codex-sandbox:latest -f Containerfile.openshell . -FROM quay.io/hmoghani/openshell-base:latest +FROM quay.io/aipcc/agentic-ci/openshell:0.3.27 USER 0 -# Node.js runtime (from UBI repos) -RUN microdnf install -y --nodocs nodejs npm && \ - microdnf clean all && \ - rm -rf /var/cache/yum +# Node.js runtime +RUN dnf install -y --nodocs nodejs npm && \ + dnf clean all # Install Codex CLI (Apache 2.0) ARG CODEX_VERSION=0.144.0 diff --git a/agents/codex/deployment/README.md b/agents/codex/deployment/README.md index c097b357..0bb0eac0 100644 --- a/agents/codex/deployment/README.md +++ b/agents/codex/deployment/README.md @@ -16,7 +16,7 @@ For the full deployment guide, see the [Codex on OpenShift README](../README.md) |------|-------------| | `Containerfile` | RHOAI image (UBI 9 minimal, Codex via npm, entrypoint, MCP, session persistence) | | `Containerfile.base` | Alternative: compiles Codex from source (Rust cross-compilation) | -| `Containerfile.openshell` | OpenShell sandbox variant (extends openshell-base) | +| `Containerfile.openshell` | OpenShell sandbox variant (RHEL 10, extends `quay.io/aipcc/agentic-ci/openshell`) | | `entrypoint.sh` | Container entrypoint (auth, model provider, MCP, git credential setup) | | `deployment.yaml` | OpenShift manifests (ImageStream, BuildConfig, ConfigMaps, PVC, Deployment) | @@ -43,7 +43,7 @@ podman build --platform linux/amd64 -t codex:latest -f Containerfile . | | RHOAI (Containerfile) | Source (Containerfile.base) | OpenShell (Containerfile.openshell) | |--|----------------------|---------------------------|-------------------------------------| -| **Base** | UBI 9 minimal | UBI 9 minimal (multi-stage) | openshell-base | +| **Base** | UBI 9 minimal | UBI 9 minimal (multi-stage) | RHEL 10 (OpenShell) | | **Codex install** | npm (pre-built binary) | Compiled from source (Rust) | npm install in image | | **Build time** | ~1 min | ~12 min (32GB RAM required) | ~1 min | | **Runtime** | Standalone pod, `oc exec` | Standalone pod, `oc exec` | OpenShell gateway sandbox | diff --git a/agents/codex/deployment/entrypoint.sh b/agents/codex/deployment/entrypoint.sh index f52aac3b..77e5610e 100644 --- a/agents/codex/deployment/entrypoint.sh +++ b/agents/codex/deployment/entrypoint.sh @@ -157,7 +157,7 @@ setup_model_provider() { # Just set the model if specified if [[ -n "${OPENAI_MODEL:-}" ]]; then if ! grep -q '^model\s*=' "${config_file}" 2>/dev/null; then - echo "model = \"${OPENAI_MODEL}\"" >> "${config_file}" + echo "model = \"$(sanitize_value "${OPENAI_MODEL}")\"" >> "${config_file}" fi fi return @@ -242,7 +242,7 @@ setup_mcp() { "\n[mcp_servers.\($name)]", (if .url then "url = \"\(.url)\"" else empty end), (if .command then "command = \"\(.command)\"" else empty end), - (if .args then "args = [\(.args | map("\"" + . + "\"") | join(", "))]" else empty end), + (if .args then "args = [\(.args | map(@json) | join(", "))]" else empty end), "" ' 2>/dev/null) if [[ -n "${server_toml}" ]]; then From 67387f9adf4429da2b1840b7d3bb9e192259e8e4 Mon Sep 17 00:00:00 2001 From: Hamid Moghani Date: Mon, 27 Jul 2026 15:27:37 -0400 Subject: [PATCH 25/25] fix(codex): fix markdown heading levels in OpenShell section (MD001) Change #### to ### under ## OpenShell Sandbox to avoid skipping h3. Co-Authored-By: Claude Opus 4.6 (1M context) --- agents/codex/README.md | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/agents/codex/README.md b/agents/codex/README.md index afb02d2a..6dc4d7b1 100644 --- a/agents/codex/README.md +++ b/agents/codex/README.md @@ -696,14 +696,14 @@ Codex CLI uses TOML for all configuration (`config.toml`). There is no JSON sett > Tested on: OpenShell gateway v0.0.86, OpenShift 4.21, Codex CLI v0.144.0, Qwen3.6-27B. -#### Prerequisites +### Prerequisites - [Helm](https://helm.sh/) 3+ installed locally - [OpenShell CLI](https://docs.nvidia.com/openshell/get-started/quickstart) installed locally (`openshell --version` should print `0.0.58` or later) - The Kubernetes Agent Sandbox CRD must be installed: `oc get crd sandboxes.agents.x-k8s.io` - A vLLM endpoint reachable from within the cluster -#### Step 1: Install the OpenShell gateway +### Step 1: Install the OpenShell gateway ```bash export OPENSHELL_NAME=openshell @@ -728,7 +728,7 @@ Grant the privileged SCC to the sandbox service account: oc adm policy add-scc-to-user privileged -z ${OPENSHELL_NAME}-sandbox -n ``` -#### Step 2: Connect the OpenShell CLI +### Step 2: Connect the OpenShell CLI Extract the mTLS certificates and register the gateway: @@ -749,7 +749,7 @@ openshell gateway add https://127.0.0.1:18080 --local --name codex-cluster openshell status ``` -#### Step 3: Build the sandbox image +### Step 3: Build the sandbox image Build inside the cluster so the internal registry can serve it to sandbox pods: @@ -779,7 +779,7 @@ EOF oc start-build codex-sandbox --from-dir=. --follow ``` -#### Step 4: Create a sandbox and test +### Step 4: Create a sandbox and test ```bash openshell sandbox create \ @@ -810,7 +810,7 @@ oc exec codex -c agent -- bash -c ' ' ``` -#### Cleanup +### Cleanup ```bash openshell sandbox delete codex