Skip to content

fix(proxy): close restartless observability gaps - #1223

Merged
murdore merged 1 commit into
releasefrom
fix/proxy-completion-gaps
Jul 22, 2026
Merged

murdore merged 1 commit into
releasefrom
fix/proxy-completion-gaps

Conversation

@murdore

@murdore murdore commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • replace the active rolling worker on .env changes while preserving the public listener and draining existing streams
  • serialize environment, explicit, and crash-recovery replacements; invalidate stale timer callbacks; and harden listener cleanup, supervisor PID validation, and stale status metadata handling
  • extend socket-transfer acknowledgement to 30 seconds, retain exact worker exit/transfer failures after recovery, and keep unsafe descriptors non-replayable
  • guard parent and worker copies of pre-commit transferred sockets so client ECONNRESET cannot crash the supervisor or worker
  • preserve pending environment replacements across worker recovery and ignore comment/whitespace-only .env rewrites
  • persist successful upstream attempts, bound and abort request/body log I/O during shutdown, and preflight immediate SSE errors after OAuth refresh
  • retain current-day metadata indexes, report unattributed persisted counters, and expose per-stream/body-artifact data quality in proxy analyze

Validation

  • pnpm exec vitest run test/proxyAnalysis.test.ts test/proxyConfigHotReload.test.ts test/proxyObservabilityFoundation.test.ts test/proxyReliabilityHardening.test.ts test/proxyReplay.test.ts test/proxyRollingWorkerHandoff.test.ts test/proxyUpdaterFallback.test.ts test/proxyUsageStatsPersistence.test.ts (181 passed)
  • pnpm run test:bugfixes (234 passed)
  • pnpm run typecheck
  • changed-file ESLint (0 errors)
  • pnpm run build:cli
  • full pre-commit pipeline: check, format, lint, validate/security, package build, browser build, publint
  • pnpm run proxy:performance (all budgets passed; 0 dropped handoff requests; active stream preserved)
  • built CLI proxy analyze --help smoke test

Live safety

No repository test sent model traffic or used the live proxy as a development target.

Summary by CodeRabbit

  • New Features
    • Expanded offline proxy log analysis to include debug time windows and body-artifact quality (indexed, present/missing, invalid paths, truncation, write failures).
    • Runtime reload now better detects environment-only changes and reports environmentChanged.
    • Proxy /status now improves serving-state details and per-account attribution, including an “unattributed” entry.
  • Bug Fixes
    • Standardized attempt/telemetry completion logging across JSON, streaming, and auth-retry flows.
    • Improved shutdown/reload reliability by coordinating request-log flushing and strengthening log cleanup/retention safeguards.

Copilot AI review requested due to automatic review settings July 21, 2026 12:16
@vercel

vercel Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
neurolink Ready Ready Preview, Comment Jul 22, 2026 2:06pm

@github-actions

github-actions Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
Contributor

✅ Single Commit Policy - COMPLIANT

Status: Policy requirements met • 1 commit • Valid format • Ready for merge

📊 View validation details

📝 Commit Details

  • Hash: eb3feca4c620bd0e84cb9802c917cc51fc20c3a5
  • Message: fix(proxy): close restartless observability gaps
  • Author: Sachin Sharma

✅ Validation Results

  • Single commit requirement met
  • No merge commits in branch
  • Semantic commit message format verified
  • Ready for squash merge to release branch

🤖 Automated validation by NeuroLink Single Commit Enforcement

@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The proxy updates offline log analysis, request-log durability, environment-triggered rolling replacement, Anthropic retry telemetry, supervisor detection, and /status account and serving-state reporting. Tests cover artifact tracking, log flushing, hot reloads, worker handoff, retry behavior, and account attribution.

Changes

Proxy observability and reliability

Layer / File(s) Summary
Offline analysis and artifact quality
src/lib/proxy/proxyAnalysis.ts, src/cli/commands/proxyAnalyze.ts, src/lib/types/proxy.ts, test/proxyAnalysis.test.ts
Analysis reports per-stream observation windows, debug files, and body-artifact indexing, validation, presence, truncation, and write metrics.
Request log flushing and retention
src/lib/proxy/requestLogger.ts, src/cli/commands/proxy.ts, test/proxyObservabilityFoundation.test.ts
File writes can be flushed before shutdown, current-day metadata is protected from cleanup, and request-log flush failures are reported.
Environment reload and rolling replacement
src/lib/proxy/runtimeConfig.ts, src/lib/proxy/rollingWorker*.ts, src/lib/proxy/rollingProxyServer.ts, src/cli/commands/proxy.ts, tests
Environment-file hashes distinguish reloads, socket workers request environment replacements, and rolling servers serialize replacement and socket-failure handling.
Anthropic retry and attempt telemetry
src/lib/server/routes/claudeProxyRoutes.ts, test/proxyReliabilityHardening.test.ts
Success, retry, streaming, and OAuth refresh paths record attempt results and preserve transient rate-limit retry outcomes.
Status attribution and process identification
src/cli/commands/proxy.ts, src/lib/types/proxy.ts, tests
Supervisor detection requires a proxy-start command, status rows reconcile account totals including unattributed activity, and serving-state fields are preferred when available.
Socket handoff and shutdown validation
src/lib/proxy/socketWorkerRuntime.ts, test/fixtures/proxySocketWorker.cjs, test/proxyRollingWorkerHandoff.test.ts, test/proxyReliabilityHardening.test.ts
Socket cleanup handles resets before commit, delayed acknowledgements are exercised, and transfer, replacement, and shutdown invariants are verified.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

Suggested reviewers: tara-ag

Sequence Diagram(s)

sequenceDiagram
  participant ProxyRuntimeConfigStore
  participant SocketWorker
  participant RollingWorkerSupervisor
  participant RollingProxyServer
  ProxyRuntimeConfigStore->>SocketWorker: report environmentChanged
  SocketWorker->>RollingWorkerSupervisor: send replacement-requested(environment)
  RollingWorkerSupervisor->>RollingProxyServer: notify active generation
  RollingProxyServer->>RollingWorkerSupervisor: start same-version replacement
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 25.93% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is concise and matches the PR’s main theme: fixing proxy restartless replacement and observability/reporting gaps.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/proxy-completion-gaps

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the proxy’s restartless worker lifecycle and observability pipeline, focusing on safer rolling worker replacement triggers, more durable request/attempt logging during shutdown, and richer offline log analysis output for operators.

Changes:

  • Add “same-version” rolling worker replacements on .env (non-routing) changes, with debouncing/serialization and safer cleanup.
  • Improve request/attempt observability: attempt logging on success paths, flush admitted request/body logs on shutdown, and surface unattributed persisted counters in status.
  • Enhance proxy analyze to report per-stream coverage windows and body-artifact data quality without reading artifacts.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
test/proxyUpdaterFallback.test.ts Asserts status output includes an unattributed internal account row when persisted counters can’t be attributed.
test/proxyRollingWorkerHandoff.test.ts Adds coverage for worker-requested same-version replacements (environment change signal).
test/proxyReliabilityHardening.test.ts Covers SSE preflight after OAuth refresh and validates attempt logging is emitted for success paths.
test/proxyObservabilityFoundation.test.ts Verifies request-log flushing and log cleanup behavior (preserve current-day metadata, evict older artifacts).
test/proxyConfigHotReload.test.ts Ensures env-only changes are detected and published as replacement-worthy (environmentChanged).
test/proxyAnalysis.test.ts Adds analysis checks for per-stream observation windows and body-artifact presence/quality metrics.
test/continuous-test-suite-bugfixes.ts Tightens supervisor PID validation expectations to require neurolink proxy start.
src/lib/types/proxy.ts Extends proxy analysis/status types (debug stream, data-quality metrics, replacement-request message, env-change metadata).
src/lib/server/routes/claudeProxyRoutes.ts Logs attempt outcomes on success paths and preflights immediate SSE errors before client commit (especially post-refresh).
src/lib/proxy/runtimeConfig.ts Hashes env file content to detect env-only changes and reports environmentChanged on reload.
src/lib/proxy/rollingWorkerSupervisor.ts Forwards active-worker replacement requests to the rolling server via onReplacementRequested.
src/lib/proxy/rollingWorkerProtocol.ts Validates the new proxy-worker:replacement-requested status message shape.
src/lib/proxy/rollingProxyServer.ts Debounces/serializes worker-requested replacements and hardens listener/supervisor cleanup paths.
src/lib/proxy/requestLogger.ts Tracks pending log writes for flushing, and preserves current-day metadata during cleanup.
src/lib/proxy/proxyAnalysis.ts Adds debug-stream ingestion, stream observation windows, artifact presence checks, and more accurate “coverage” semantics.
src/cli/commands/proxyAnalyze.ts Prints new debug-file counts, per-stream observation windows, and body-artifact data quality summary.
src/cli/commands/proxy.ts Tightens supervisor process matching, flushes request logs on shutdown, improves stale status handling, and adds unattributed stats.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/lib/proxy/requestLogger.ts
Comment thread src/lib/types/proxy.ts
@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
src/lib/server/routes/claudeProxyRoutes.ts (1)

4311-4361: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Remove the dead stream branch from handleAnthropicSuccessfulRetryResponse. It is only called from the !body.stream path, so the if (body.stream && retryResp.body) block and its stream-handling code are unreachable and misleading.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/server/routes/claudeProxyRoutes.ts` around lines 4311 - 4361, Remove
the unreachable stream-handling branch from
handleAnthropicSuccessfulRetryResponse, including the body.stream/retryResp.body
condition and its associated streaming logic. Keep this helper focused on the
non-stream retry path used by the response construction shown, preserving its
existing non-stream response behavior.
src/cli/commands/proxy.ts (1)

3671-3695: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Reuse StatusStats["accounts"] for the live /status cast
This inline account shape is duplicated and already stale: it hard-codes cooling: boolean and omits status, while the shared StatusStats["accounts"] entry already models both. Reuse the canonical type here to avoid drift.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/cli/commands/proxy.ts` around lines 3671 - 3695, Update the live /status
response type cast in the statusData flow to reuse the canonical
StatusStats["accounts"] type for stats.accounts instead of the duplicated inline
account shape. Preserve the existing stats fields while removing the stale
inline account definition, including its hard-coded cooling property, so the
shared type remains authoritative.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/cli/commands/proxy.ts`:
- Around line 3671-3695: Update the live /status response type cast in the
statusData flow to reuse the canonical StatusStats["accounts"] type for
stats.accounts instead of the duplicated inline account shape. Preserve the
existing stats fields while removing the stale inline account definition,
including its hard-coded cooling property, so the shared type remains
authoritative.

In `@src/lib/server/routes/claudeProxyRoutes.ts`:
- Around line 4311-4361: Remove the unreachable stream-handling branch from
handleAnthropicSuccessfulRetryResponse, including the body.stream/retryResp.body
condition and its associated streaming logic. Keep this helper focused on the
non-stream retry path used by the response construction shown, preserving its
existing non-stream response behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 317cf5f6-899c-4116-9d0b-128dfecabde8

📥 Commits

Reviewing files that changed from the base of the PR and between 2b2797f and 4b5c013.

📒 Files selected for processing (17)
  • src/cli/commands/proxy.ts
  • src/cli/commands/proxyAnalyze.ts
  • src/lib/proxy/proxyAnalysis.ts
  • src/lib/proxy/requestLogger.ts
  • src/lib/proxy/rollingProxyServer.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/lib/proxy/rollingWorkerSupervisor.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/lib/server/routes/claudeProxyRoutes.ts
  • src/lib/types/proxy.ts
  • test/continuous-test-suite-bugfixes.ts
  • test/proxyAnalysis.test.ts
  • test/proxyConfigHotReload.test.ts
  • test/proxyObservabilityFoundation.test.ts
  • test/proxyReliabilityHardening.test.ts
  • test/proxyRollingWorkerHandoff.test.ts
  • test/proxyUpdaterFallback.test.ts

@murdore
murdore force-pushed the fix/proxy-completion-gaps branch from 3b536e1 to a34bd18 Compare July 21, 2026 12:41
@murdore

murdore commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Addressed both CodeRabbit nitpicks in a34bd18: removed the unreachable auth-retry streaming branch after SSE preflight unification, and reused the canonical StatusStats type for live status parsing. The focused and full proxy tests, typecheck, and complete pre-commit pipeline pass.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@murdore

murdore commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Tara-ag Tara-ag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Reviewed the 17 changed files in this proxy observability/lifecycle PR. The changes are well-scoped and the test coverage is thorough, but I found 3 MAJOR issues that should be addressed before merge, plus 3 MINOR suggestions.

Blocking issues (MAJOR)

  1. src/lib/server/routes/claudeProxyRoutes.ts — handleAnthropicSuccessfulRetryResponse logs logAttempt(retryResp.status) without attemptDurationMs, while the streaming retry path and main success paths do include it. This leaves a real observability gap in the "JSON, streaming, and auth-retry flows" consistency the PR targets.

  2. src/lib/proxy/rollingProxyServer.ts — The requestedReplacementRunning guard is set inside the async setTimeout callback, not before it. Between the timer firing and the flag being set, a concurrent replace() call can observe requestedReplacementRunning === false, skip awaiting, and invoke supervisor.replace() concurrently. The supervisor serializes the underlying spawn, but the rolling-server state (requestedReplacementPromise, requestedReplacementRunning) ends up tracking the wrong replacement.

  3. src/lib/proxy/requestLogger.ts — flushRequestLogs() can hang indefinitely if a tracked appendFile/writeFile promise never settles (stuck NFS, broken pipe, etc.). The while loop keeps awaiting Promise.allSettled([...pendingLogOperations]) without progress. The shutdown wrapper times out, but the underlying hung promise remains and can prevent clean process exit.

Non-blocking suggestions (MINOR)

  • rollingProxyServer.ts: clearTimeout does not remove a timer callback that has already been queued by the event loop, creating another race window between scheduled and explicit replacements.
  • proxyAnalysis.ts: the artifact path-traversal guard is reasonable but could be more robust; writeFailures is incremented when bodyPath is absent, which may represent an intentional redaction-only capture rather than a write failure.
  • runtimeConfig.ts: environmentChanged is only computed after the early-return branch; consider computing it earlier so a changed:false reload result can still report it.

Existing review comments

The two prior Copilot comments (lifecycle log matcher, StatusStats.accounts[].cooling required) appear resolved in the latest commit and are not re-raised.

Next steps

Please address the three MAJOR issues above. The replacement-serialization race and the flush hang are the highest-risk for production reliability.

Comment thread src/lib/server/routes/claudeProxyRoutes.ts
Comment thread src/lib/proxy/rollingProxyServer.ts
Comment thread src/lib/proxy/requestLogger.ts
Comment thread src/lib/proxy/rollingProxyServer.ts
Comment thread src/lib/proxy/proxyAnalysis.ts
Comment thread src/lib/proxy/runtimeConfig.ts
@murdore

murdore commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Fresh read-only live evidence exposed one additional handoff gap: the active v10.3.0 worker lost four socket transfers, the supervisor recovered to generation 2, but the 5-second acknowledgement threshold killed the worker and successful recovery cleared the root-cause field. Commit 5b8c23b raises the production acknowledgement deadline to 30 seconds, logs exact transfer/exit details (including code/signal), retains lastFailure across successful recovery, and keeps ambiguous transferred sockets non-replayable. Revalidated: 175 proxy tests, 234 bugfix tests, typecheck, ESLint, CLI build, full pre-commit, and all performance budgets.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/server/routes/claudeProxyRoutes.ts`:
- Around line 4029-4031: Update the non-streaming retry path around
handleAnthropicAuthRetry so its logAttempt call includes the retry attempt
duration. Modify retryLogAttempt to preserve extra.attemptDurationMs when
provided, falling back to elapsed time from retryAttemptStartedAt only when
absent, and retain the attempt number. Ensure this helper remains restricted to
non-streaming retries, either by renaming it to reflect that scope or asserting
!body.stream.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 21170c3b-4ca0-4601-aec7-b4cfc3fbbf89

📥 Commits

Reviewing files that changed from the base of the PR and between 3b536e1 and 5b8c23b.

📒 Files selected for processing (18)
  • src/cli/commands/proxy.ts
  • src/cli/commands/proxyAnalyze.ts
  • src/lib/proxy/proxyAnalysis.ts
  • src/lib/proxy/requestLogger.ts
  • src/lib/proxy/rollingProxyServer.ts
  • src/lib/proxy/rollingWorkerProcess.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/lib/proxy/rollingWorkerSupervisor.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/lib/server/routes/claudeProxyRoutes.ts
  • src/lib/types/proxy.ts
  • test/continuous-test-suite-bugfixes.ts
  • test/proxyAnalysis.test.ts
  • test/proxyConfigHotReload.test.ts
  • test/proxyObservabilityFoundation.test.ts
  • test/proxyReliabilityHardening.test.ts
  • test/proxyRollingWorkerHandoff.test.ts
  • test/proxyUpdaterFallback.test.ts
🚧 Files skipped from review as they are similar to previous changes (14)
  • test/proxyConfigHotReload.test.ts
  • test/proxyUpdaterFallback.test.ts
  • test/proxyRollingWorkerHandoff.test.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/cli/commands/proxyAnalyze.ts
  • test/proxyObservabilityFoundation.test.ts
  • src/lib/types/proxy.ts
  • src/lib/proxy/rollingProxyServer.ts
  • src/lib/proxy/requestLogger.ts
  • src/cli/commands/proxy.ts
  • test/proxyReliabilityHardening.test.ts
  • test/proxyAnalysis.test.ts
  • src/lib/proxy/proxyAnalysis.ts

Comment thread src/lib/server/routes/claudeProxyRoutes.ts Outdated
@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (1)
src/lib/proxy/runtimeConfig.ts (1)

337-340: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

envFileHash hashes raw file bytes, not parsed vars — cosmetic .env edits trigger real worker replacement.

envFileHash is derived from the raw file content (envFileContent) rather than the parsed key/value pairs. A whitespace-only or comment-only edit to .env will therefore still flip the hash, set environmentChanged: true, and (per the downstream consumer in proxy.ts) trigger an actual rolling worker replacement even though no effective environment variable changed.

Consider hashing a normalized, sorted representation of the parsed env vars (e.g. JSON.stringify(Object.entries(parse(envFileContent)).sort())) instead of the raw content, so only semantically meaningful changes trigger a replacement.

♻️ Proposed fix
-  const envFileHash = createHash("sha256")
-    .update(envFilePresent ? envFileContent : "[missing]")
-    .digest("hex")
-    .slice(0, 16);
+  const envFileHash = createHash("sha256")
+    .update(
+      envFilePresent
+        ? JSON.stringify(
+            Object.entries(effectiveEnv).sort(([a], [b]) => a.localeCompare(b)),
+          )
+        : "[missing]",
+    )
+    .digest("hex")
+    .slice(0, 16);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/proxy/runtimeConfig.ts` around lines 337 - 340, Update the
envFileHash calculation in the runtime configuration flow to hash a normalized,
deterministically sorted representation of the parsed environment variables
rather than raw envFileContent. Reuse the existing env parsing logic, ensuring
whitespace-only and comment-only edits produce the same hash while actual
key/value changes still update environmentChanged.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/proxy/rollingProxyServer.ts`:
- Around line 106-144: In scheduleRequestedReplacement, move the
requestedReplacementPending = false assignment to after the closing or
inactive-worker guard. Preserve the flag when the timer fires without an active
worker so queueReplacement’s recovery/self-heal path can retry the pending
environment replacement.

---

Nitpick comments:
In `@src/lib/proxy/runtimeConfig.ts`:
- Around line 337-340: Update the envFileHash calculation in the runtime
configuration flow to hash a normalized, deterministically sorted representation
of the parsed environment variables rather than raw envFileContent. Reuse the
existing env parsing logic, ensuring whitespace-only and comment-only edits
produce the same hash while actual key/value changes still update
environmentChanged.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: b2726221-d9e9-4d43-af64-0b03afe8c7ce

📥 Commits

Reviewing files that changed from the base of the PR and between 5b8c23b and 637e098.

📒 Files selected for processing (18)
  • src/cli/commands/proxy.ts
  • src/cli/commands/proxyAnalyze.ts
  • src/lib/proxy/proxyAnalysis.ts
  • src/lib/proxy/requestLogger.ts
  • src/lib/proxy/rollingProxyServer.ts
  • src/lib/proxy/rollingWorkerProcess.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/lib/proxy/rollingWorkerSupervisor.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/lib/server/routes/claudeProxyRoutes.ts
  • src/lib/types/proxy.ts
  • test/continuous-test-suite-bugfixes.ts
  • test/proxyAnalysis.test.ts
  • test/proxyConfigHotReload.test.ts
  • test/proxyObservabilityFoundation.test.ts
  • test/proxyReliabilityHardening.test.ts
  • test/proxyRollingWorkerHandoff.test.ts
  • test/proxyUpdaterFallback.test.ts
🚧 Files skipped from review as they are similar to previous changes (12)
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/cli/commands/proxyAnalyze.ts
  • test/proxyConfigHotReload.test.ts
  • src/lib/types/proxy.ts
  • test/proxyObservabilityFoundation.test.ts
  • test/continuous-test-suite-bugfixes.ts
  • test/proxyAnalysis.test.ts
  • src/lib/proxy/rollingWorkerSupervisor.ts
  • src/lib/proxy/requestLogger.ts
  • src/lib/server/routes/claudeProxyRoutes.ts
  • test/proxyReliabilityHardening.test.ts
  • src/cli/commands/proxy.ts

Comment thread src/lib/proxy/rollingProxyServer.ts
@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@murdore

murdore commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

Live evidence captured after the released 10.3.0 worker rotated:

  • the long-lived supervisor stayed on PID 14230 while the worker generation advanced 2 -> 3 -> 4
  • status recorded 26 rejected sockets and 26 failed transfers
  • generation 3 failed immediately, before serving lifecycle traffic
  • stderr proves the immediate worker failure was an unhandled Node Socket error: Error: read ECONNRESET, emitted while a transferred socket was waiting for the child-side commit
  • generation 2 likely entered recovery after the released 5-second transfer-ack window was exceeded under load, but 10.3.0 clears the root failure after recovery, so that first trigger cannot be stated as proven

Commit 2412be2 guards both parent and worker copies of pre-commit transferred sockets. A client reset now removes/destroys the pending descriptor instead of terminating either process. Coverage includes a deterministic pending-socket ECONNRESET unit test and a real child-process TCP RST test proving the same worker PID remains active and serves /health with zero transfer failures.

The same commit also preserves an environment-triggered replacement if the active worker exits during the debounce window, and hashes parsed/sorted env values so comment- or whitespace-only edits do not roll a worker.

Validation: proxy suites 181/181, bugfix suite 234/234, typecheck, ESLint, CLI/package/browser builds, publint, security validation, and all proxy performance budgets. No live model request was sent and no additional live restart was performed during this investigation.

@murdore
murdore requested a review from Tara-ag July 21, 2026 14:04
@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@murdore

murdore commented Jul 21, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 21, 2026 •

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@Tara-ag Tara-ag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Reviewed the proxy observability/lifecycle changes in this PR. The previously raised blocking issues have been addressed in the latest commits:

  • Non-stream OAuth retry duration: handleAnthropicSuccessfulNonStreamRetryResponse now logs attemptDurationMs, and retryLogAttempt preserves an explicitly passed duration.
  • Rolling replacement serialization: rollingProxyServer.ts uses a shared queueReplacement serializer and defers clearing requestedReplacementPending until after the active-worker guard.
  • Request-log flush deadline: flushRequestLogs now enforces a bounded deadline, drops timed-out tracked operations, and uses abortable writeFile calls.
  • Artifact path containment: proxyAnalysis.ts uses path.relative plus absolute/parent checks instead of a simple prefix match.
  • Env-hash normalization: runtimeConfig.ts hashes parsed, sorted env values so comment/whitespace edits do not trigger worker replacement.

No new blocking issues were found in this run. The test coverage is thorough (rolling handoff, observability foundation, reliability hardening, config hot-reload, and analysis suites), and the changes stay within the proxy subsystem without affecting the public SDK API.

Files reviewed: 20 changed files (focused on src/cli/commands/proxy.ts, src/lib/proxy/*, src/lib/server/routes/claudeProxyRoutes.ts, src/lib/types/proxy.ts, and test updates).

New issues raised this run: 0

Approving.

@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/lib/proxy/rollingWorkerProcess.ts (1)

143-149: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Use ErrorFactory for the transfer-exit error.

This new lifecycle error bypasses the repository’s typed-error contract. Use an existing lifecycle factory or add a dedicated factory case so callers can reliably classify the failure.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/proxy/rollingWorkerProcess.ts` around lines 143 - 149, Update the
child.once("exit") handler to create the socket-transfer failure through
ErrorFactory instead of directly constructing Error. Reuse an existing lifecycle
factory case when applicable; otherwise add a dedicated factory case that
preserves the exit code and signal context and allows callers to classify the
error reliably.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/proxy/rollingWorkerProcess.ts`:
- Around line 143-149: Update the child.once("exit") handler to create the
socket-transfer failure through ErrorFactory instead of directly constructing
Error. Reuse an existing lifecycle factory case when applicable; otherwise add a
dedicated factory case that preserves the exit code and signal context and
allows callers to classify the error reliably.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: d8fed8fd-e82e-442b-b796-89a052896221

📥 Commits

Reviewing files that changed from the base of the PR and between 637e098 and c347f31.

📒 Files selected for processing (20)
  • src/cli/commands/proxy.ts
  • src/cli/commands/proxyAnalyze.ts
  • src/lib/proxy/proxyAnalysis.ts
  • src/lib/proxy/requestLogger.ts
  • src/lib/proxy/rollingProxyServer.ts
  • src/lib/proxy/rollingWorkerProcess.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • src/lib/proxy/rollingWorkerSupervisor.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/lib/proxy/socketWorkerRuntime.ts
  • src/lib/server/routes/claudeProxyRoutes.ts
  • src/lib/types/proxy.ts
  • test/continuous-test-suite-bugfixes.ts
  • test/fixtures/proxySocketWorker.cjs
  • test/proxyAnalysis.test.ts
  • test/proxyConfigHotReload.test.ts
  • test/proxyObservabilityFoundation.test.ts
  • test/proxyReliabilityHardening.test.ts
  • test/proxyRollingWorkerHandoff.test.ts
  • test/proxyUpdaterFallback.test.ts
🚧 Files skipped from review as they are similar to previous changes (13)
  • test/proxyObservabilityFoundation.test.ts
  • src/lib/proxy/rollingWorkerProtocol.ts
  • test/proxyUpdaterFallback.test.ts
  • test/continuous-test-suite-bugfixes.ts
  • src/lib/types/proxy.ts
  • src/lib/proxy/rollingProxyServer.ts
  • test/proxyAnalysis.test.ts
  • src/lib/proxy/runtimeConfig.ts
  • src/lib/proxy/proxyAnalysis.ts
  • src/cli/commands/proxy.ts
  • src/lib/proxy/requestLogger.ts
  • test/proxyReliabilityHardening.test.ts
  • src/lib/server/routes/claudeProxyRoutes.ts

@Tara-ag Tara-ag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review Summary

Reviewed the proxy restartless-observability changes. The PR addresses real gaps, but I found 1 CRITICAL and 5 MAJOR new issues that need to be fixed before merge.

New issues raised

Severity Count Files
🔒 CRITICAL 1 src/lib/proxy/rollingWorkerProcess.ts
⚠️ MAJOR 5 src/lib/proxy/requestLogger.ts, src/lib/proxy/proxyAnalysis.ts, src/lib/proxy/rollingProxyServer.ts, src/lib/server/routes/claudeProxyRoutes.ts, src/lib/proxy/socketWorkerRuntime.ts
💡 MINOR 0 —
💬 SUGGESTION 0 —

Blocking reasons

  1. CRITICAL correctness bug in socket handoff (rollingWorkerProcess.ts): the supervisor can send socket-cancel to a worker after the worker has already accepted and committed the socket, which may destroy a live connection.
  2. ≥3 MAJOR issues: shutdown log-flush can drop newly-admitted operations; the offline analyzer's path containment is bypassable via symlinks; recovery backoff state is not reset on successful explicit replacements; pending socket event listeners leak after commit.

Next steps

Please address the inline comments and re-request review. Focus first on the CRITICAL socket-cancel race and the MAJOR shutdown-flush / path-traversal issues, as those have the highest operational/security impact. The existing resolved review threads were not re-raised.

Comment thread src/lib/proxy/rollingWorkerProcess.ts
Comment thread src/lib/proxy/socketWorkerRuntime.ts
Comment thread src/lib/proxy/rollingProxyServer.ts
Comment thread src/lib/server/routes/claudeProxyRoutes.ts
Comment thread src/lib/proxy/requestLogger.ts
Comment thread src/lib/proxy/proxyAnalysis.ts
@github-actions

Copy link
Copy Markdown
Contributor

🤖 AI Review & Build Compliance ✅

Status: AI analysis complete • Build rules validated • Ready for review

📊 View detailed analysis results

🛡️ Analysis Complete

  • ✅ Security scan (vulnerabilities, API keys)
  • ✅ TypeScript safety & code quality
  • ✅ Error handling & best practices
  • ✅ Build rule enforcement validated
  • ✅ Commit format & compliance checks

📋 Ready for Merge When

  • All CI checks passing
  • Manual review approved
  • Any AI-flagged issues resolved

🤖 AI analysis complete - check individual code comments for specific feedback

@murdore

murdore commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Final review update on eb3feca (rebased onto current origin/release v10.4.2):

  • Fixed the post-accept socket-cancel race.
  • Removed exact pending transfer listeners without disturbing HTTP/server listeners.
  • Preserved log operations admitted during a timed-out flush.
  • Added lexical and canonical realpath containment for analysis artifacts.
  • Switched the worker transfer-exit path to ErrorFactory with lifecycle context.
  • Verified the recovery counter and OAuth retry findings against current code; both paths already behave correctly.

Verification on the final rebased commit:

  • 182/182 focused proxy tests
  • 234/234 bugfix suite tests
  • full ./pre-commit.sh
  • all proxy performance budgets, with 0 concurrent handoff failures and 0 dropped requests

All review threads are answered and resolved. Fresh CI and Yama re-review are running.

@Tara-ag Tara-ag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review summary

Reviewed the proxy restartless-observability changes in PR #1223. This is a final verification pass over the latest commit (eb3feca4).

Previously raised blocking issues — verified fixed

  • CRITICAL socket-cancel race (src/lib/proxy/rollingWorkerProcess.ts): socket-cancel is now sent only while !pending.accepted, so a late parent timeout cannot destroy a socket the worker already owns.
  • MAJOR worker-side listener leak (src/lib/proxy/socketWorkerRuntime.ts): pending pre-commit error/close listeners are now removed on commit/cancel/drain/shutdown, while unrelated server listeners are preserved.
  • MAJOR shutdown log-flush accounting (src/lib/proxy/requestLogger.ts): flushRequestLogs now snapshots admitted operations and only settles/removes that snapshot; operations admitted mid-flush remain queued for the next flush instead of being silently dropped.
  • MAJOR offline analyzer path traversal (src/lib/proxy/proxyAnalysis.ts): artifact paths now reject NUL bytes and .. components lexically, then compare realpath-resolved paths against the canonical bodies root so symlink escapes are excluded.
  • MAJOR replacement serialization (src/lib/proxy/rollingProxyServer.ts): environment, explicit, and recovery replacements share the queueReplacement serializer; pending timer callbacks carry a monotonic schedule token and are invalidated by explicit replacement/close.
  • MAJOR non-stream OAuth retry duration (src/lib/server/routes/claudeProxyRoutes.ts): handleAnthropicSuccessfulNonStreamRetryResponse logs attemptDurationMs, and retryLogAttempt preserves an explicitly passed duration.
  • MINOR env-hash normalization (src/lib/proxy/runtimeConfig.ts): envFileHash now hashes parsed, sorted key/value pairs so comment/whitespace edits do not trigger worker replacement.

New issues raised this run

0 — no new blocking or major issues found in the current commit.

Verification performed

  • Re-read the changed source files at refs/pull/1223/head.
  • Cross-checked the fixes against the resolved review threads.
  • Confirmed test coverage additions in test/proxyRollingWorkerHandoff.test.ts, test/proxyObservabilityFoundation.test.ts, test/proxyReliabilityHardening.test.ts, test/proxyAnalysis.test.ts, and test/proxyConfigHotReload.test.ts.

The changes stay within the proxy subsystem, do not touch the public SDK API, and the previously identified blocking concerns are addressed. Approving.

@murdore
murdore merged commit b956836 into release Jul 22, 2026
17 checks passed
@murdore
murdore deleted the fix/proxy-completion-gaps branch July 22, 2026 14:47
@github-actions

Copy link
Copy Markdown
Contributor

🎉 This PR is included in version 10.4.3 🎉

The release is available on:

Your semantic-release bot 📦🚀

This branch was successfully deployed

1 active deployment
Preview — eb3feca4 Deployed Jul 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants