diff --git a/AGENTS.md b/AGENTS.md index 44b5d985..76cecb14 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -49,12 +49,14 @@ make dry-run # preview Helm manifests ## Non-standard agents -Two agents diverge significantly from the standard pattern: +Several agents diverge significantly from the standard pattern: **langflow/simple_tool_calling_agent** - Podman Compose flow-import deployment, not standalone FastAPI. No Dockerfile, pyproject.toml, main.py, src/, or tests/. Different Makefile targets (`init`, `ollama`, `run`, `stop`, `clean`). Uses infra-only env vars (PostgreSQL, Langfuse, Ollama) -- not `API_KEY`/`BASE_URL`/`MODEL_ID`. **a2a/langgraph_crewai_agent** - Uses `python:3.12-slim` (not UBI9), PYTHONPATH `/app` (not `/opt/app-root/src`), `charts/a2a-langgraph-crewai/` chart, `template.env` (not `.env.example`), `entrypoint.sh` (not `main.py`), Starlette (not FastAPI). No tests/ directory. +**openclaw/deployment** - Kustomize-based deployment of OpenClaw on OpenShift. Uses pre-built image (`ghcr.io/openclaw/openclaw:latest`), not a Dockerfile. Kustomize overlays for customization (model endpoint, storage class). No Makefile, no FastAPI, no src/. Port 18789 (gateway). + ## Common gotchas - `agent.yaml` is read at runtime by Makefile -- if malformed, all make targets break diff --git a/agents/openclaw/deployment/README.md b/agents/openclaw/deployment/README.md new file mode 100644 index 00000000..523e7bb1 --- /dev/null +++ b/agents/openclaw/deployment/README.md @@ -0,0 +1,104 @@ +# OpenClaw on OpenShift + +> Tested: 2026-04-13 on OpenShift 4.19 (ROSA) with vLLM model serving + +Deploy [OpenClaw](https://github.com/openclaw/openclaw) on Red Hat OpenShift with vLLM model serving, OAuth SSO, and production-grade security — no cluster-admin required. + +## Prerequisites + +- OpenShift 4.17+ with namespace-scoped access (`oc login`) +- A model serving endpoint (vLLM, KServe, or external API) +- Block storage class (gp3-csi, managed-csi, thin-csi) — not NFS + +## Quick Start + +### Option A: Using claw-installer (recommended) + +The [claw-installer](https://github.com/sallyom/claw-installer) handles OAuth proxy, Routes, ServiceAccounts, and lifecycle management automatically. + +```bash +git clone https://github.com/sallyom/claw-installer.git +cd claw-installer +npm install && npm run build && npm run dev +# Open http://localhost:3000, select OpenShift, fill in the form +``` + +See [docs/installer-deployment.md](docs/installer-deployment.md) for the full walkthrough, validation, and rollback procedures. + +### Option B: Kustomize + +```bash +oc new-project my-openclaw + +# Copy and customize the example overlay +cp -r overlays/example overlays/my-env +# Edit overlays/my-env/configmap-patch.yaml with your vLLM endpoint +# Edit overlays/my-env/kustomization.yaml with your namespace and storage class + +oc apply -k overlays/my-env +``` + +### Option C: Direct YAML apply + +```bash +oc new-project my-openclaw + +# Edit manifests/02-configmap.yaml with your model endpoint +# Edit manifests/01-secret.yaml with your gateway token + +oc apply -k manifests/ +``` + +## Architecture + +```text + +-----------------------+ + | OpenShift Route | + | (TLS edge) | + +-----------+-----------+ + | + v + +------------------+------------------+ + | Pod | + | +---------------+ +-------------+ | + | | oauth-proxy | | gateway | | + | | port 8443 +->| port 18789 | | + | | (OpenShift | | (loopback) | | + | | OAuth SSO) | | | | + | +---------------+ +------+------+ | + | | | + | +------+------+ | + | | PVC (5Gi) | | + | +-------------+ | + +--------------------------------------+ + | + | OpenAI-compatible API + v + +---------------------------+ + | vLLM / KServe / API | + +---------------------------+ +``` + +## Customization + +| What | Where | +|------|-------| +| Model endpoint URL | `overlays//configmap-patch.yaml` | +| Storage class | `overlays//kustomization.yaml` (patch) | +| Namespace | `overlays//kustomization.yaml` (`namespace:` field) | +| Gateway token | `manifests/01-secret.yaml` | +| Resource limits | Patch `manifests/04-deployment.yaml` | + +## Docs + +| Document | Description | +|----------|-------------| +| [docs/installer-deployment.md](docs/installer-deployment.md) | Full deployment guide: prerequisites, validation, rollback, appendix | +| [docs/troubleshooting.md](docs/troubleshooting.md) | Common issues and fixes (route 503, model override, heartbeat, config clobber) | +| [docs/model-compatibility.md](docs/model-compatibility.md) | Model testing results for agentic tool-calling | + +## Related Projects + +- [claw-installer](https://github.com/sallyom/claw-installer) — Web-based deployment tool with OpenShift plugin +- [openclaw-on-openshift](https://github.com/aakankshaduggal/openclaw-on-openshift) — Source repo with full docs +- [OpenClaw](https://github.com/openclaw/openclaw) — Upstream project diff --git a/agents/openclaw/deployment/docs/installer-deployment.md b/agents/openclaw/deployment/docs/installer-deployment.md new file mode 100644 index 00000000..c6b1027c --- /dev/null +++ b/agents/openclaw/deployment/docs/installer-deployment.md @@ -0,0 +1,447 @@ +# Deploying OpenClaw on OpenShift with openclaw-installer + +> Tested: 2026-04-13 on OpenShift 4.19 (ROSA) with vLLM model serving + +Step-by-step guide for deploying OpenClaw on an OpenShift cluster using the [openclaw-installer](https://github.com/sallyom/claw-installer). This is the recommended deployment method — it handles OAuth proxy, ServiceAccounts, Routes, and lifecycle management automatically. + +## Table of Contents + +- [Prerequisites](#prerequisites) +- [Architecture Overview](#architecture-overview) +- [Deployment Steps](#deployment-steps) +- [Validation](#validation) +- [Instance Management](#instance-management) +- [Rollback Instructions](#rollback-instructions) +- [Appendix](#appendix) + +## Prerequisites + +### Environment Requirements + +- **OpenShift Version:** 4.17+ (tested on 4.19.18) +- **Access Level:** Namespace-scoped — no cluster-admin needed +- **CLI Tools:** `oc` (OpenShift CLI), `node` (Node.js 22+), `git` +- **Storage:** Block storage class (gp3-csi, managed-csi, thin-csi) — avoid NFS (SQLite requires POSIX file locking) +- **Model Endpoint:** vLLM, KServe, or external API (OpenAI, Anthropic, Vertex AI) + +### Verify Prerequisites + +```bash +# Verify oc CLI +oc version +``` + +Expected output: + +```text +Client Version: 4.17.0 +... +``` + +```bash +# Verify cluster access +oc whoami +``` + +Expected output: + +```text + +``` + +```bash +# Verify Node.js +node --version +``` + +Expected output: + +```text +v22.x.x +``` + +```bash +# Check available storage classes +oc get storageclass +``` + +Expected output (look for a block storage class): + +```text +NAME PROVISIONER RECLAIMPOLICY ... +gp3-csi ebs.csi.aws.com Delete ... +``` + +## Architecture Overview + +```text + +-----------------------+ + | OpenShift Route | + | (TLS edge) | + +-----------+-----------+ + | + v + +------------------+------------------+ + | Pod | + | +---------------+ +-------------+ | + | | oauth-proxy | | gateway | | + | | port 8443 +->| port 18789 | | + | | (OpenShift | | (loopback) | | + | | OAuth SSO) | | | | + | +---------------+ +------+------+ | + | | | + | +------+------+ | + | | PVC (10Gi) | | + | +-------------+ | + +--------------------------------------+ + | + | OpenAI-compatible API + v + +---------------------------+ + | vLLM / KServe / API | + +---------------------------+ +``` + +## Deployment Steps + +### Step 1: Create your namespace + +```bash +oc login --token= --server=https://api.:443 +oc new-project -openclaw +``` + +Expected output: + +```text +Now using project "-openclaw" on server "https://api.:443". +``` + +### Step 2: Start the installer + +```bash +git clone https://github.com/sallyom/claw-installer.git +cd openclaw-installer +npm install && npm run build && npm run dev +``` + +Expected output: + +```text +> openclaw-installer@x.x.x dev +> next dev + + ▲ Next.js x.x.x + - Local: http://localhost:3000 +``` + +Open `http://localhost:3000`. The installer auto-detects your OpenShift cluster and loads the OpenShift provider plugin. + +### Step 3: Fill in the deploy form + +| Field | Value | Notes | +|-------|-------|-------| +| **Agent name** | your choice | e.g., `my-agent` | +| **Project / Namespace** | your namespace | e.g., `my-openclaw` | +| **Image** | `ghcr.io/openclaw/openclaw:latest` | | +| **Provider** | Self-hosted (vLLM) | or Anthropic/OpenAI/Vertex AI if using cloud | +| **Model endpoint** | your vLLM URL | e.g., `https://vllm-20b-gpt-oss.apps.rosa./v1` | + +Click **Deploy**. The installer streams logs as it creates each resource. + +Expected output (installer log panel): + +```text +Creating ServiceAccount... ✓ +Creating Secrets... ✓ +Creating ConfigMap... ✓ +Creating PVC... ✓ +Creating Deployment... ✓ +Creating Service... ✓ +Creating Route... ✓ +Deployment complete. +``` + +### Step 4: Verify pod startup + +```bash +oc get pods -n +``` + +Expected output: + +```text +NAME READY STATUS RESTARTS AGE +openclaw-xxxxxxxxxx-xxxxx 2/2 Running 0 60s +``` + +> **Note:** `2/2` indicates both the oauth-proxy and gateway containers are running. If you see `1/2` or `CrashLoopBackOff`, check [troubleshooting.md](troubleshooting.md). + +### Step 5: Fix the Route (if needed) + +Open the Route URL: + +```bash +oc get route openclaw -n -o jsonpath='{.spec.host}' +``` + +Navigate to `https://` in your browser. If you see "Application is not available", the TLS termination is misconfigured. See [troubleshooting.md](troubleshooting.md#route-returns-application-is-not-available-503). + +**Verification:** + +```bash +curl -s -o /dev/null -w "%{http_code}" -k https://$(oc get route openclaw -n -o jsonpath='{.spec.host}') +``` + +Expected output: + +```text +403 +``` + +> A `403` is correct — it means the OAuth proxy is running and will redirect you to OpenShift SSO. A `503` means the Route is misconfigured. + +### Step 6: Approve device pairing + +After SSO login, the Control UI requires a one-time device pairing. Get the request ID from the UI prompt, then: + +```bash +oc exec deployment/openclaw -n -c gateway -- \ + openclaw devices approve +``` + +Expected output: + +```text +Device approved: +``` + +Or use the **Open** action from the installer's **Instances** tab — it opens with the token pre-filled and may auto-pair. + +### Step 7: Verify model configuration + +Check the gateway logs to confirm the correct model is loaded: + +```bash +oc logs deployment/openclaw -c gateway -n | grep "agent model" +``` + +Expected output: + +```text +[gateway] agent model: openai-compat/gpt-oss-20b +``` + +If it shows a different model (e.g., `anthropic/claude-sonnet-4-6`), the gateway auto-detected a provider and overrode your config. See [troubleshooting.md](troubleshooting.md#gateway-uses-wrong-model--no-api-key-errors). + +## Validation + +After completing the deployment steps, validate each component: + +### 1. Pod Health + +```bash +oc get pods -n +``` + +Expected: All pods show `2/2 Running` status with zero restarts. + +### 2. Route Accessibility + +```bash +curl -s -o /dev/null -w "%{http_code}" -k \ + https://$(oc get route openclaw -n -o jsonpath='{.spec.host}') +``` + +Expected: `403` (OAuth redirect gate). + +### 3. Gateway Logs + +```bash +oc logs deployment/openclaw -c gateway -n | grep -E "(ready|agent model|heartbeat)" +``` + +Expected output: + +```text +[gateway] agent model: openai-compat/gpt-oss-20b +[gateway] ready (4 plugins: ...; 3.xs) +[heartbeat] disabled +``` + +### 4. OAuth Proxy Logs + +```bash +oc logs deployment/openclaw -c oauth-proxy -n --tail=5 +``` + +Expected: No error lines. Should show listening messages. + +### 5. PVC Bound + +```bash +oc get pvc -n +``` + +Expected: + +```text +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +openclaw-home-pvc Bound ... 10Gi RWO gp3-csi 5m +``` + +### 6. End-to-End Test + +After device pairing, send a test message in the Control UI. The agent should respond using the configured model. If you see "No API key found" errors, the model provider isn't configured correctly — see [troubleshooting.md](troubleshooting.md). + +## What the installer creates + +| Resource | Name | Purpose | +|----------|------|---------| +| ServiceAccount | `openclaw-oauth-proxy` | SA for OAuth with redirect annotation | +| Secret | `openclaw-oauth-config` | OAuth client-secret + cookie secret | +| Secret | `openclaw-secrets` | Gateway token + provider API keys | +| ConfigMap | `openclaw-config` | `openclaw.json` gateway configuration | +| ConfigMap | `openclaw-agent` | Agent workspace files (HEARTBEAT.md, SOUL.md, etc.) | +| PVC | `openclaw-home-pvc` | 10Gi persistent state (SQLite, agent sessions, logs) | +| Service | `openclaw` | ClusterIP: gateway (18789) + oauth-ui (8443) | +| Route | `openclaw` | TLS edge-terminated route to oauth-proxy | +| Deployment | `openclaw` | Init container + oauth-proxy sidecar + gateway | + +## Instance Management + +From the installer's **Instances** tab: + +| Action | Effect | +|--------|--------| +| **Open** | Opens Route URL with token pre-filled | +| **Re-deploy** | Syncs local agent workspace files + restarts pod | +| **Stop** | Scales replicas to 0 (preserves PVC data) | +| **Start** | Scales replicas back to 1 | +| **Approve Pairing** | Approves pending device pairing requests | + +## Rollback Instructions + +### Complete Cleanup + +To completely remove the OpenClaw deployment and all associated resources: + +```bash +# Delete the entire namespace (removes all resources including PVC data) +oc delete project +``` + +> **Warning:** This deletes all data, including agent sessions and SQLite state on the PVC. + +### Partial Rollback + +To remove specific components while preserving others: + +```bash +# Stop the deployment (preserves PVC and config) +oc scale deployment/openclaw --replicas=0 -n + +# Remove deployment only (preserves PVC, secrets, config) +oc delete deployment openclaw -n + +# Remove route only (pod keeps running, just not externally accessible) +oc delete route openclaw -n +``` + +### Config Rollback + +If a ConfigMap change broke the gateway: + +```bash +# Check deployment revision history +oc rollout history deployment/openclaw -n + +# Roll back to previous revision +oc rollout undo deployment/openclaw -n +``` + +### PVC Data Reset + +If the on-disk config is corrupted and the gateway won't start: + +```bash +# Scale down, delete PVC, redeploy +oc scale deployment/openclaw --replicas=0 -n +oc delete pvc openclaw-home-pvc -n + +# Re-deploy via installer or re-apply manifests +# The init container will reinitialize the config from the ConfigMap +oc scale deployment/openclaw --replicas=1 -n +``` + +## Appendix + +### A. Configuration Parameters + +| Parameter | Location | Default | Description | +|-----------|----------|---------|-------------| +| `agents.defaults.model.primary` | ConfigMap | varies | Model reference (e.g., `openai-compat/gpt-oss-20b`) | +| `agents.defaults.heartbeat.every` | ConfigMap | `30m` | Heartbeat interval (`0m` to disable) | +| `gateway.bind` | ConfigMap | `loopback` | Bind address (`loopback` recommended with OAuth proxy) | +| `gateway.controlUi.allowedOrigins` | ConfigMap | `[]` | Route URL for CORS — must match your Route host | +| `models.providers.*.baseUrl` | ConfigMap | none | Model serving endpoint URL | +| `tools.deny` | ConfigMap | `[]` | Tool deny list (e.g., `["group:web", "browser"]`) | +| `OPENCLAW_GATEWAY_TOKEN` | Secret | generated | Gateway auth token (auto-generated by installer) | +| `OPENCLAW_HOME` | Env var | `/data/.openclaw` | Data directory on PVC | + +### B. Image Sources + +| Component | Registry | Image | +|-----------|----------|-------| +| Gateway | GitHub Container Registry | `ghcr.io/openclaw/openclaw:latest` | +| OAuth Proxy | Red Hat Registry | `registry.redhat.io/openshift4/ose-oauth-proxy:latest` | +| Init Container | Same as gateway | `ghcr.io/openclaw/openclaw:latest` | + +### C. Network Ports + +| Component | Port | Protocol | Purpose | +|-----------|------|----------|---------| +| Gateway | 18789 | HTTP | API server + Control UI WebSocket | +| OAuth Proxy | 8443 | HTTPS | OAuth-gated frontend | +| Browser Control | 18791 | HTTP | Internal browser automation (loopback only) | + +### D. Resource Requirements + +| Container | CPU Request | Memory Request | Memory Limit | +|-----------|-------------|----------------|--------------| +| init-config | — | — | 128Mi | +| gateway | 250m | 256Mi | 1Gi | +| oauth-proxy | 100m | 64Mi | 256Mi | + +### E. Test Environment Specifications + +| Property | Value | +|----------|-------| +| **OpenShift Version** | 4.19.18 | +| **Platform** | AWS (ROSA) | +| **Worker Nodes** | 4 nodes | +| **Storage Class** | gp3-csi (EBS) | +| **Model Endpoint** | vLLM (gpt-oss-20b) on GPU node | +| **Node OS** | Red Hat Enterprise Linux CoreOS | +| **Kernel** | 5.14.0-570.x | + +### F. Known Limitations + +| Limitation | Impact | Workaround | +|------------|--------|------------| +| **Single replica only** | No HA — Recreate strategy required because SQLite does not support concurrent writers | Use PostgreSQL-backed model server for multi-replica; or accept single-replica for PoC | +| **NFS incompatible** | SQLite requires POSIX file locking; NFS advisory locks are unreliable | Use block storage (gp3-csi, managed-csi, thin-csi) | +| **Config auto-override** | Gateway may overwrite ConfigMap settings with auto-detected providers on first start | Verify model in logs after every rollout; patch ConfigMap if overridden | +| **Heartbeat on by default** | Fires every 30 min, consumes model API tokens even when idle | Set `agents.defaults.heartbeat.every: "0m"` in ConfigMap | +| **PVC data lost on delete** | Deleting the PVC removes all agent sessions, SQLite state, and logs | Back up PVC data before deletion; consider `Retain` reclaim policy | +| **Device pairing required** | Every new browser session needs a one-time device approval via CLI | Use installer's **Open** action, or approve via `oc exec` | + +### G. References + +| Resource | URL | +|----------|-----| +| OpenClaw Upstream | | +| openclaw-installer | | +| OpenClaw Docs | | +| OpenShift Docs | | +| Kustomize | | diff --git a/agents/openclaw/deployment/docs/model-compatibility.md b/agents/openclaw/deployment/docs/model-compatibility.md new file mode 100644 index 00000000..80d8c468 --- /dev/null +++ b/agents/openclaw/deployment/docs/model-compatibility.md @@ -0,0 +1,60 @@ +# Model Compatibility for OpenClaw Tool-Calling + +> Tested: 2026-04-13 + +OpenClaw relies on models that support structured tool-calling (function calling). Not all models work — some emit raw JSON instead of using the native tool API, and some hallucinate tool results when calls fail silently. + +## Test Results + +| Model | Provider | Tool-Calling | Notes | +|-------|----------|:------------:|-------| +| **gpt-oss-20b** | vLLM (cluster) | Working | `--tool-call-parser openai --enable-auto-tool-choice` flags required | +| **qwen2.5:7b** | Ollama (local) | Working | Best local option for 32GB machines (~8GB RAM) | +| **qwen3.5:27b** | Ollama (local) | Working | Tools work, but 41GB RAM usage makes it impractical locally | +| **llama3.2:3b** | Ollama (local) | Partial | Executes tools but gets confused by OpenClaw's internal system prompts | +| **phi4-mini:3.8b** | Ollama (local) | Broken | Emits raw JSON in response body instead of native tool calls | +| **codex/gpt-5.4** | Codex Harness | Expected | Routes through Codex app-server; requires sidecar deployment (see [codex-harness.md](codex-harness.md)) | + +## Critical Warning: Hallucinated Tool Results + +When phi4-mini's tool calls failed silently, the model **fabricated an entire file listing** — realistic-looking paths that did not exist on the machine, presented confidently as real data. + +This is the most dangerous failure mode for an agentic system: the user has no way to know the results are fake without independently verifying every output. + +## Provider Routing + +OpenClaw supports multiple model providers simultaneously. The model reference prefix determines the routing: + +| Prefix | Route | Harness | +|--------|-------|---------| +| `openai-compat/` | Self-hosted vLLM endpoint | OpenClaw PI runtime | +| `openai/` | Direct OpenAI API | OpenClaw PI runtime | +| `codex/` | Codex app-server (sidecar) | Codex harness | +| `anthropic/` | Anthropic API | OpenClaw PI runtime | +| `ollama/` | Local Ollama instance | OpenClaw PI runtime | + +Mixed deployments can use `runtime: "auto"` to let OpenClaw select the harness based on model prefix. + +## Recommendations + +- **On OpenShift (self-hosted):** Use gpt-oss-20b or larger via vLLM with tool-calling flags enabled +- **On OpenShift (Codex):** Use `codex/gpt-5.4` with the Codex Harness sidecar for thread management and guardian approvals +- **On OpenShift (Google):** Use `google/gemini-2.5-pro` via Google AI Studio with `GEMINI_API_KEY` (Vertex AI SA auth is not yet supported natively — see [vertex-ai-provider.md](vertex-ai-provider.md#known-limitation-vertex-ai-auth)) +- **Local testing:** Use qwen2.5:7b via Ollama with a 32k context window +- **Always verify:** Test tool-calling with a known-answer query (e.g., "list files on my Desktop") before trusting agent output + +## Valid API Types + +When configuring `models.providers.*.api` in the ConfigMap, use one of: + +| API Type | Use Case | +|----------|----------| +| `openai-completions` | vLLM, LM Studio, any OpenAI-compatible endpoint | +| `openai-responses` | Direct OpenAI API | +| `openai-codex-responses` | Codex app-server | +| `anthropic-messages` | Anthropic API or Anthropic on Vertex AI | +| `google-generative-ai` | Google Gemini (AI Studio or Vertex AI) | +| `ollama` | Local Ollama instance | +| `azure-openai-responses` | Azure OpenAI | +| `bedrock-converse-stream` | AWS Bedrock | +| `github-copilot` | GitHub Copilot | diff --git a/agents/openclaw/deployment/docs/troubleshooting.md b/agents/openclaw/deployment/docs/troubleshooting.md new file mode 100644 index 00000000..047272d7 --- /dev/null +++ b/agents/openclaw/deployment/docs/troubleshooting.md @@ -0,0 +1,349 @@ +# Troubleshooting + +Common issues and solutions for OpenClaw on OpenShift. Issues are listed from most to least frequently encountered. + +## Table of Contents + +- [Route returns "Application is not available" (503)](#route-returns-application-is-not-available-503) +- [Gateway uses wrong model / "No API key" errors](#gateway-uses-wrong-model--no-api-key-errors) +- [Heartbeat flooding the chat UI](#heartbeat-flooding-the-chat-ui) +- [Device pairing required after SSO login](#device-pairing-required-after-sso-login) +- [Pod stuck in CrashLoopBackOff](#pod-stuck-in-crashloopbackoff) +- [Config clobbered on restart](#config-clobbered-on-restart) +- [Device pairing rate limiter](#device-pairing-rate-limiter) +- [Diagnostic commands](#diagnostic-commands) + +--- + +## Route returns "Application is not available" (503) + +**Severity:** Blocking — users cannot access the UI. + +**Cause:** The oauth-proxy uses `--http-address` (plain HTTP) by default. If the Route has `tls.termination: reencrypt`, the router expects TLS on the backend and the connection fails silently. + +**Symptoms:** + +```text +Browser: "Application is not available" +curl returns: 503 +``` + +**Fix:** Recreate the Route with `edge` termination: + +```bash +oc delete route openclaw -n + +cat <<'EOF' | oc apply -f - +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: openclaw + namespace: + labels: + app: openclaw + annotations: + haproxy.router.openshift.io/timeout: 30m +spec: + to: + kind: Service + name: openclaw + weight: 100 + port: + targetPort: oauth-ui + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect +EOF +``` + +**Verification:** + +```bash +curl -s -o /dev/null -w "%{http_code}" -k https:// +``` + +Expected output: + +```text +403 +``` + +> A `403` is correct — it means the OAuth proxy is running and will redirect to OpenShift SSO. A `503` means the Route is still misconfigured. + +**Prevention:** When deploying via the installer, verify the Route TLS mode immediately after deployment: + +```bash +oc get route openclaw -n -o jsonpath='{.spec.tls.termination}' +# Should show: edge +``` + +--- + +## Gateway uses wrong model / "No API key" errors + +**Severity:** Blocking — agent cannot respond to messages. + +**Cause:** OpenClaw auto-generates its config on first start. If the gateway detects a provider (e.g., Anthropic) it will override the ConfigMap settings with its own defaults. This means the ConfigMap says vLLM but the gateway is actually trying to use Anthropic. + +**Symptoms in logs:** + +```text +No API key found for provider "anthropic" +model fallback decision: decision=candidate_failed requested=anthropic/claude-sonnet-4-6 +``` + +**Symptoms in UI:** The model selector in the Control UI shows a model name you did not configure (e.g., `gpt-oss-20b` instead of `gemini-2.5-pro`). + +**Fix:** Patch the ConfigMap with the correct model provider config, then restart: + +```bash +# Export current config +oc get configmap openclaw-config -n \ + -o jsonpath='{.data.openclaw\.json}' > /tmp/openclaw-config.json + +# Edit /tmp/openclaw-config.json: +# - Set agents.defaults.model.primary to "openai-compat/gpt-oss-20b" +# - Add models.providers.openai-compat with your vLLM endpoint + +# Apply and restart +oc create configmap openclaw-config \ + --from-file=openclaw.json=/tmp/openclaw-config.json \ + -n --dry-run=client -o yaml | oc apply -f - +oc rollout restart deployment/openclaw -n +``` + +**Verification:** + +```bash +oc logs deployment/openclaw -c gateway -n | grep "agent model" +``` + +Expected output: + +```text +[gateway] agent model: openai-compat/gpt-oss-20b +``` + +**Prevention:** Always check the gateway logs after every rollout to confirm the model matches your intent. + +--- + +## Heartbeat flooding the chat UI + +**Severity:** Moderate — consumes API tokens and clutters the chat history. + +**Cause:** OpenClaw's heartbeat scheduler fires every 30 minutes by default, sending `HEARTBEAT_OK` messages to the chat. If the model provider is misconfigured, each heartbeat triggers an error instead, producing dozens of error messages per day. + +**Symptoms in UI:** Repeated "HEARTBEAT_OK" messages or "Agent failed before reply" errors every 30 minutes. + +**Symptoms in logs:** + +```text +[heartbeat] started +``` + +**Fix:** Set `heartbeat.every` to `"0m"` in the ConfigMap: + +```bash +# Export config +oc get configmap openclaw-config -n \ + -o jsonpath='{.data.openclaw\.json}' > /tmp/openclaw-config.json + +# Add to agents.defaults: +# "heartbeat": { "every": "0m" } +# Also add to each agent in agents.list: +# "heartbeat": { "every": "0m" } + +# Apply and restart +oc create configmap openclaw-config \ + --from-file=openclaw.json=/tmp/openclaw-config.json \ + -n --dry-run=client -o yaml | oc apply -f - +oc rollout restart deployment/openclaw -n +``` + +**Verification:** + +```bash +oc logs deployment/openclaw -c gateway -n | grep heartbeat +``` + +Expected output: + +```text +[heartbeat] disabled +``` + +**Reference:** See [OpenClaw heartbeat docs](https://docs.openclaw.ai/gateway/heartbeat) for advanced options (`activeHours`, `target`, `lightContext`). + +--- + +## Device pairing required after SSO login + +**Severity:** Low — one-time setup per browser. + +**Cause:** The Control UI browser session needs a one-time device approval, even after OpenShift OAuth login. This is an OpenClaw security feature, not an OpenShift issue. + +**Fix:** + +```bash +# Get the request ID from the UI prompt, then: +oc exec deployment/openclaw -n -c gateway -- \ + openclaw devices approve +``` + +Expected output: + +```text +Device approved: +``` + +Alternatively, use the **Open** action from the installer's **Instances** tab — it opens with the token pre-filled and may auto-pair. + +--- + +## Pod stuck in CrashLoopBackOff + +**Severity:** Blocking — gateway is down. + +**Cause:** OpenClaw auto-generates config at startup. If the config on the PVC conflicts with the ConfigMap, the gateway detects a change, overwrites the file, and triggers a process restart that kills PID 1. + +**Diagnostic:** + +```bash +oc describe pod -l app=openclaw -n | grep -A 5 "Last State" +oc logs deployment/openclaw -c gateway -n --previous +``` + +Look for: `Config overwrite` or `Config observe anomaly` messages. + +**Fix:** Delete the PVC data and redeploy: + +```bash +oc scale deployment/openclaw --replicas=0 -n +oc delete pvc openclaw-state -n +# If deployed via claw-installer, the PVC is named openclaw-home-pvc instead +# Redeploy via installer or re-apply manifests +oc scale deployment/openclaw --replicas=1 -n +``` + +> **Warning:** This deletes all agent sessions and chat history. Back up the PVC first if needed: +> +> ```bash +> oc cp :/data/.openclaw /tmp/openclaw-backup -c gateway -n +> ``` + +--- + +## Config clobbered on restart + +**Severity:** Moderate — settings revert unexpectedly. + +**Cause:** OpenClaw's init container copies `openclaw.json` from the ConfigMap to the PVC on every pod start. However, the gateway also writes back to the same file at runtime (model discovery, plugin state). On the next restart, the init container overwrites these runtime changes with the original ConfigMap version. This produces `.clobbered.*` backup files on the PVC. + +**Symptoms:** + +```bash +oc exec deployment/openclaw -c gateway -n -- ls /home/node/.openclaw/ +# Shows: openclaw.json.clobbered.2026-04-15T14-45-55-777Z +``` + +**Fix:** Update the ConfigMap to include any runtime changes you want to preserve: + +```bash +# Export the running config (not the ConfigMap) +oc exec deployment/openclaw -c gateway -n -- \ + cat /home/node/.openclaw/openclaw.json > /tmp/openclaw-config.json + +# Review and apply +oc create configmap openclaw-config \ + --from-file=openclaw.json=/tmp/openclaw-config.json \ + -n --dry-run=client -o yaml | oc apply -f - +``` + +--- + +## Device Pairing Rate Limiter + +**Severity:** Blocking — browser cannot connect even with valid credentials. + +**Cause:** When a browser repeatedly sends a stale device token (e.g., after PVC data was deleted and device pairings were cleared), the gateway's in-memory rate limiter kicks in after several failed attempts. Once rate-limited, all connection attempts from that client are rejected — even after fixing the underlying token issue. + +**Symptoms in logs:** + +```text +unauthorized ... reason=device_token_mismatch +unauthorized ... reason=rate_limited +``` + +**Symptoms in browser:** "Too many failed authentication attempts (retry later)" or the connection silently fails. + +**Fix (both steps required):** + +1. **Clear browser site data** for the OpenClaw route URL — this removes the cached stale device token: + - Chrome: Click lock icon in URL bar → "Site settings" → "Clear data" + - Or: DevTools (F12) → Application → Storage → "Clear site data" + +2. **Restart the pod** to reset the in-memory rate limiter: + + ```bash + oc rollout restart deployment/openclaw -n + ``` + +3. **Refresh the browser** — you should now see a fresh device pairing prompt. Approve it: + + ```bash + oc exec deployment/openclaw -c gateway -n -- \ + openclaw devices approve + ``` + +**Verification:** The browser connects and shows the chat UI without errors. + +**Prevention:** When clearing device pairings (`openclaw devices clear`), always clear browser site data at the same time to prevent the stale token retry loop. + +--- + +## Diagnostic Commands + +```bash +# Pod status +oc get pods -n + +# Gateway logs (tail) +oc logs deployment/openclaw -c gateway -n --tail=50 + +# Gateway logs (full) +oc logs deployment/openclaw -c gateway -n + +# OAuth proxy logs +oc logs deployment/openclaw -c oauth-proxy -n + +# Previous pod logs (after crash) +oc logs deployment/openclaw -c gateway -n --previous + +# Check route config +oc get route openclaw -n -o jsonpath='{.spec.tls.termination}' + +# Check what model is active +oc logs deployment/openclaw -c gateway -n | grep "agent model" + +# Check ConfigMap vs on-disk config +oc get configmap openclaw-config -n \ + -o jsonpath='{.data.openclaw\.json}' | python3 -m json.tool + +# Check on-disk config (what gateway is actually using) +oc exec deployment/openclaw -c gateway -n -- \ + cat /home/node/.openclaw/openclaw.json | python3 -m json.tool + +# List clobbered config backups +oc exec deployment/openclaw -c gateway -n -- \ + ls -la /home/node/.openclaw/openclaw.json.clobbered.* 2>/dev/null + +# Check PVC status +oc get pvc -n + +# Check deployment events +oc describe deployment openclaw -n | tail -20 + +# Exec into the gateway +oc exec -it deployment/openclaw -c gateway -n -- sh +``` diff --git a/agents/openclaw/deployment/manifests/01-secret.yaml b/agents/openclaw/deployment/manifests/01-secret.yaml new file mode 100644 index 00000000..4ac66e8e --- /dev/null +++ b/agents/openclaw/deployment/manifests/01-secret.yaml @@ -0,0 +1,11 @@ +apiVersion: v1 +kind: Secret +metadata: + name: openclaw-secrets + labels: + app: openclaw +type: Opaque +stringData: + # Replace with your own token + OPENCLAW_GATEWAY_TOKEN: "CHANGE-ME" + VLLM_API_KEY: "not-needed" diff --git a/agents/openclaw/deployment/manifests/02-configmap.yaml b/agents/openclaw/deployment/manifests/02-configmap.yaml new file mode 100644 index 00000000..7489f56f --- /dev/null +++ b/agents/openclaw/deployment/manifests/02-configmap.yaml @@ -0,0 +1,55 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: openclaw-config + labels: + app: openclaw +data: + openclaw.json: | + { + "gateway": { + "port": 18789, + "bind": "0.0.0.0", + "mode": "local", + "auth": { + "mode": "token" + } + }, + "channels": {}, + "agents": { + "defaults": { + "heartbeat": {}, + "model": "openai-compat/gpt-oss-20b", + "sandbox": { + "mode": "off" + } + } + }, + "models": { + "mode": "replace", + "providers": { + "openai-compat": { + "baseUrl": "https://YOUR-VLLM-ENDPOINT/v1", + "api": "openai-completions", + "apiKey": "not-needed", + "models": [ + { + "id": "gpt-oss-20b", + "name": "gpt-oss-20b", + "reasoning": false, + "input": ["text"], + "contextWindow": 32768, + "maxTokens": 4096 + } + ] + } + } + }, + "plugins": {}, + "tools": { + "deny": [ + "group:web", + "browser" + ] + } + } diff --git a/agents/openclaw/deployment/manifests/03-pvc.yaml b/agents/openclaw/deployment/manifests/03-pvc.yaml new file mode 100644 index 00000000..ed3e7b7d --- /dev/null +++ b/agents/openclaw/deployment/manifests/03-pvc.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: openclaw-state + labels: + app: openclaw +spec: + accessModes: + - ReadWriteOnce + # Change to your cluster's block storage class (e.g., gp3-csi, managed-csi, thin-csi) + # Avoid NFS — OpenClaw uses SQLite which requires POSIX file locking + storageClassName: gp3-csi + resources: + requests: + storage: 5Gi diff --git a/agents/openclaw/deployment/manifests/04-deployment.yaml b/agents/openclaw/deployment/manifests/04-deployment.yaml new file mode 100644 index 00000000..00195f56 --- /dev/null +++ b/agents/openclaw/deployment/manifests/04-deployment.yaml @@ -0,0 +1,93 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: openclaw + labels: + app: openclaw +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: openclaw + template: + metadata: + labels: + app: openclaw + spec: + securityContext: + runAsNonRoot: true + initContainers: + - name: init-config + image: ghcr.io/openclaw/openclaw:latest + command: + - sh + - -c + - | + cp /config/openclaw.json /data/.openclaw/openclaw.json + chgrp -R 0 /data/.openclaw + chmod -R g=u /data/.openclaw + echo "Config initialized" + volumeMounts: + - name: config + mountPath: /config + - name: data + mountPath: /data/.openclaw + resources: + limits: + memory: 128Mi + cpu: 100m + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + containers: + - name: gateway + image: ghcr.io/openclaw/openclaw:latest + ports: + - containerPort: 18789 + protocol: TCP + env: + - name: OPENCLAW_GATEWAY_TOKEN + valueFrom: + secretKeyRef: + name: openclaw-secrets + key: OPENCLAW_GATEWAY_TOKEN + - name: OPENCLAW_HOME + value: /data/.openclaw + volumeMounts: + - name: data + mountPath: /data/.openclaw + resources: + requests: + memory: 256Mi + cpu: 250m + limits: + memory: 1Gi + cpu: "1" + livenessProbe: + httpGet: + path: / + port: 18789 + initialDelaySeconds: 15 + periodSeconds: 30 + readinessProbe: + httpGet: + path: / + port: 18789 + initialDelaySeconds: 10 + periodSeconds: 10 + securityContext: + allowPrivilegeEscalation: false + capabilities: + drop: + - ALL + volumes: + - name: config + configMap: + name: openclaw-config + - name: data + persistentVolumeClaim: + claimName: openclaw-state diff --git a/agents/openclaw/deployment/manifests/05-service.yaml b/agents/openclaw/deployment/manifests/05-service.yaml new file mode 100644 index 00000000..7c8faf07 --- /dev/null +++ b/agents/openclaw/deployment/manifests/05-service.yaml @@ -0,0 +1,15 @@ +apiVersion: v1 +kind: Service +metadata: + name: openclaw + labels: + app: openclaw +spec: + selector: + app: openclaw + ports: + - name: gateway + port: 18789 + targetPort: 18789 + protocol: TCP + type: ClusterIP diff --git a/agents/openclaw/deployment/manifests/06-route.yaml b/agents/openclaw/deployment/manifests/06-route.yaml new file mode 100644 index 00000000..71c4ead1 --- /dev/null +++ b/agents/openclaw/deployment/manifests/06-route.yaml @@ -0,0 +1,18 @@ +apiVersion: route.openshift.io/v1 +kind: Route +metadata: + name: openclaw + labels: + app: openclaw + annotations: + haproxy.router.openshift.io/timeout: 30m +spec: + to: + kind: Service + name: openclaw + weight: 100 + port: + targetPort: gateway + tls: + termination: edge + insecureEdgeTerminationPolicy: Redirect diff --git a/agents/openclaw/deployment/manifests/kustomization.yaml b/agents/openclaw/deployment/manifests/kustomization.yaml new file mode 100644 index 00000000..abc6201c --- /dev/null +++ b/agents/openclaw/deployment/manifests/kustomization.yaml @@ -0,0 +1,14 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +resources: + - 01-secret.yaml + - 02-configmap.yaml + - 03-pvc.yaml + - 04-deployment.yaml + - 05-service.yaml + - 06-route.yaml + +commonLabels: + app.kubernetes.io/name: openclaw + app.kubernetes.io/part-of: openclaw-on-openshift diff --git a/agents/openclaw/deployment/overlays/example/configmap-patch.yaml b/agents/openclaw/deployment/overlays/example/configmap-patch.yaml new file mode 100644 index 00000000..3c102bcf --- /dev/null +++ b/agents/openclaw/deployment/overlays/example/configmap-patch.yaml @@ -0,0 +1,53 @@ +apiVersion: v1 +kind: ConfigMap +metadata: + name: openclaw-config +data: + openclaw.json: | + { + "gateway": { + "port": 18789, + "bind": "0.0.0.0", + "mode": "local", + "auth": { + "mode": "token" + } + }, + "channels": {}, + "agents": { + "defaults": { + "heartbeat": {}, + "model": "openai-compat/my-model", + "sandbox": { + "mode": "off" + } + } + }, + "models": { + "mode": "replace", + "providers": { + "openai-compat": { + "baseUrl": "https://my-vllm-endpoint.apps.my-cluster.example.com/v1", + "api": "openai-completions", + "apiKey": "not-needed", + "models": [ + { + "id": "my-model", + "name": "my-model", + "reasoning": false, + "input": ["text"], + "contextWindow": 32768, + "maxTokens": 4096 + } + ] + } + } + }, + "plugins": {}, + "tools": { + "deny": [ + "group:web", + "browser" + ] + } + } diff --git a/agents/openclaw/deployment/overlays/example/kustomization.yaml b/agents/openclaw/deployment/overlays/example/kustomization.yaml new file mode 100644 index 00000000..40da6a1b --- /dev/null +++ b/agents/openclaw/deployment/overlays/example/kustomization.yaml @@ -0,0 +1,24 @@ +apiVersion: kustomize.config.k8s.io/v1beta1 +kind: Kustomization + +# Point to the base manifests +resources: + - ../../manifests + +# Set your namespace +namespace: my-openclaw + +# Patch the ConfigMap with your vLLM endpoint +patches: + - target: + kind: ConfigMap + name: openclaw-config + path: configmap-patch.yaml + + - target: + kind: PersistentVolumeClaim + name: openclaw-state + patch: | + - op: replace + path: /spec/storageClassName + value: managed-csi