Skip to content

fix: restart ollama proxy when token diverges from persisted token #2555

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
adrian-santos wants to merge 1 commit into from
Closed

fix: restart ollama proxy when token diverges from persisted token #2555

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 22 additions & 2 deletions src/lib/onboard-ollama-proxy.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,20 @@ function clearPersistedProxyPid(): void {

// ── Process management ───────────────────────────────────────────

/**
* Verify that the running proxy accepts the given token.
* /v1/models is an authenticated endpoint (unlike /api/tags which is exempt),
* so a 200 response confirms the token matches what the proxy was started with.
*/
function isProxyTokenValid(token: string): boolean {
const result = spawnSync(
"curl",
["-sf", "--max-time", "3", "-H", `Authorization: Bearer ${token}`, `http://localhost:${OLLAMA_PROXY_PORT}/v1/models`],
{ encoding: "utf8" },
);
return result.status === 0;
}
Comment on lines +102 to +109

@coderabbitai coderabbitai Bot Apr 27, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

cd /repo && find . -name "onboard-ollama-proxy.ts" -type f

Repository: NVIDIA/NemoClaw

Length of output: 115

🏁 Script executed:

cd /repo && cat -n src/lib/onboard-ollama-proxy.ts | head -120

Repository: NVIDIA/NemoClaw

Length of output: 115

🏁 Script executed:

cd /repo && cat -n src/lib/onboard-ollama-proxy.ts | sed -n '190,210p'

Repository: NVIDIA/NemoClaw

Length of output: 115

🏁 Script executed:

git ls-files | grep -E "onboard.*proxy" | head -20

Repository: NVIDIA/NemoClaw

Length of output: 91

🏁 Script executed:

find . -name "onboard-ollama-proxy.ts" -type f 2>/dev/null

Repository: NVIDIA/NemoClaw

Length of output: 93

🏁 Script executed:

cat -n src/lib/onboard-ollama-proxy.ts | sed -n '1,15p'

Repository: NVIDIA/NemoClaw

Length of output: 746

🏁 Script executed:

cat -n src/lib/onboard-ollama-proxy.ts | sed -n '100,115p'

Repository: NVIDIA/NemoClaw

Length of output: 833

🏁 Script executed:

cat -n src/lib/onboard-ollama-proxy.ts | sed -n '190,210p'

Repository: NVIDIA/NemoClaw

Length of output: 818

🌐 Web query:

curl -sf flag exit code behavior 401 403 5xx timeout

💡 Result:

With curl -sf (equivalent to --silent --fail), the exit code behavior for various scenarios is as follows: - HTTP 401 (Unauthorized) or 403 (Forbidden): Exits with code 22. The -f/--fail flag causes curl to fail silently (no output) and exit with error 22 for any HTTP response code >=400. Note that this is not always fail-safe, especially with authentication involved (e.g., 401/407 may sometimes slip through). - HTTP 5xx (server errors, e.g., 500): Exits with code 22, same as above for >=400. - Timeout: Exits with code 28 (CURLE_OPERATION_TIMEDOUT). This occurs if the operation exceeds the maximum allowed time (connect, transfer, etc.). Use --max-time or --connect-timeout to adjust. Without -f, curl exits with 0 even for HTTP errors >=400, as it considers the transfer successful if data was received. -s suppresses progress/output but does not affect exit codes directly.

Citations:

🌐 Web query:

curl --max-time timeout exit code semantics

💡 Result:

When using curl with --max-time (or -m), if the maximum time is exceeded, curl exits with exit code 28, corresponding to CURLE_OPERATION_TIMEDOUT. This applies to the overall operation timeout, including connection, transfer, etc. The error message typically reads "Operation timed out after X milliseconds" (distinguishing it from connect-timeout, which says "Connection timed out" or "Resolving timed out", but both use code 28). curl returns 0 on success. Non-zero exit codes indicate errors, with 28 specifically for timeouts triggered by --max-time. Official documentation confirms: "--max-time makes curl exit with a timeout error code (28)" and lists CURLE_OPERATION_TIMEDOUT (28) for operation timeouts.

Citations:

Differentiate auth rejection from transient probe failures before forcing restart.

Line 108 treats any non-zero curl result as token divergence. With -f, the exit code 22 is returned for both auth rejection (401/403) and server errors (5xx), while timeouts exit with code 28. This means a valid proxy experiencing transient failures will be unnecessarily killed and restarted. Only explicitly reject the token on 401/403 responses; treat other failures as indeterminate.

Suggested fix 
-function isProxyTokenValid(token: string): boolean {
-  const result = spawnSync(
-    "curl",
-    ["-sf", "--max-time", "3", "-H", `Authorization: Bearer ${token}`, `http://localhost:${OLLAMA_PROXY_PORT}/v1/models`],
-    { encoding: "utf8" },
-  );
-  return result.status === 0;
+function getProxyTokenValidationStatus(token: string): "valid" | "rejected" | "unknown" {
+  const result = spawnSync(
+    "curl",
+    [
+      "-sS",
+      "-o",
+      "/dev/null",
+      "-w",
+      "%{http_code}",
+      "--max-time",
+      "3",
+      "-H",
+      `Authorization: Bearer ${token}`,
+      `http://localhost:${OLLAMA_PROXY_PORT}/v1/models`,
+    ],
+    { encoding: "utf8" },
+  );
+  const code = String(result.stdout || "").trim();
+  if (code === "200") return "valid";
+  if (code === "401" || code === "403") return "rejected";
+  return "unknown";
 }

Also update line 198 to use the returned status instead of a boolean check.

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
function isProxyTokenValid(token: string): boolean {
const result = spawnSync(
"curl",
["-sf", "--max-time", "3", "-H", `Authorization: Bearer ${token}`, `http://localhost:${OLLAMA_PROXY_PORT}/v1/models`],
{ encoding: "utf8" },
);
return result.status === 0;
}
function getProxyTokenValidationStatus(token: string): "valid" | "rejected" | "unknown" {
const result = spawnSync(
"curl",
[
"-sS",
"-o",
"/dev/null",
"-w",
"%{http_code}",
"--max-time",
"3",
"-H",
`Authorization: Bearer ${token}`,
`http://localhost:${OLLAMA_PROXY_PORT}/v1/models`,
],
{ encoding: "utf8" },
);
const code = String(result.stdout || "").trim();
if (code === "200") return "valid";
if (code === "401" || code === "403") return "rejected";
return "unknown";
}
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@src/lib/onboard-ollama-proxy.ts` around lines 102 - 109, isProxyTokenValid
currently treats any non-zero curl exit as token invalid; change it to return
the HTTP status code (number) on a probe success and null on transient/errors:
call curl without -f and use --write-out "%{http_code}" (capture stdout/stderr
and exit code), return parsed numeric HTTP status when curl succeeds, return
null for timeouts/other non-probe failures; then update the caller that
currently does a boolean check (the usage around line 198) to treat status ===
401 or === 403 as token rejection (force restart) and treat null or any other
status as indeterminate (do not force restart).


function isOllamaProxyProcess(pid: number | null | undefined): boolean {
if (!Number.isInteger(pid) || pid <= 0) return false;
const cmdline = runCapture(["ps", "-p", String(pid), "-o", "args="], { ignoreError: true });
Expand Down Expand Up @@ -168,19 +182,25 @@ function startOllamaAuthProxy(): boolean {
/**
* Ensure the auth proxy is running — called on sandbox connect to recover
* from host reboots where the background proxy process was lost.
*
* Also handles token divergence: if the proxy is running but was started with
* a different token than what is persisted on disk (which can happen when
* multiple onboard runs occur without the proxy being restarted in between),
* the proxy is killed and restarted with the persisted token so the sandbox
* credential and the proxy stay in sync.
*/
function ensureOllamaAuthProxy(): void {
// Try to load persisted token first — if none, this isn't an Ollama setup.
const token = loadPersistedProxyToken();
if (!token) return;

const pid = loadPersistedProxyPid();
if (isOllamaProxyProcess(pid)) {
if (isOllamaProxyProcess(pid) && isProxyTokenValid(token)) {
ollamaProxyToken = token;
return;
}

// Proxy not running — restart it with the persisted token.
// Proxy not running, or running with a stale token — restart with the persisted token.
killStaleProxy();
ollamaProxyToken = token;
spawnOllamaAuthProxy(token);
Expand Down
Loading