test(e2e): fail fast on a bad gateway credential (#226) - #321
Merged
Conversation
The model-driving E2E tests each spawn the CLI, which aborts on a failed auth preflight (HTTP 401) with exit code 1. When the gateway key is expired or revoked, that surfaces as ~200 opaque "CLI process exited with code 1" assertion failures — a 25-minute red run with no obvious cause (#226). Probe the gateway's /models once in globalSetup so a rejected credential fails the whole suite immediately with one clear, actionable message naming the secret to rotate. Only runs when OpenAI-compatible creds are present; a network error is treated as transient and does not block the run (only an explicit 401/403 fails fast). This does not fix the credential itself (a repo-secret rotation, tracked in #226) — it makes the next bad-credential incident diagnosable in one line. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
👀 Quinn is reviewing — verdict (PASS / WARN / FAIL) + findings to follow. |
|
Warning Review limit reached
More reviews will be available in 7 minutes and 53 seconds. Learn how PR review limits work. Your organization has run out of usage credits. Purchase more in the billing tab. ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the We recommend that you space out your commits to avoid hitting the rate limit. 🚦 How do rate limits work?CodeRabbit enforces hourly rate limits for each developer per organization. Our paid plans include higher PR review limits than trial, open-source, and free plans. In all cases, reviews become available again over time. During sustained high-volume PR review activity, CodeRabbit may temporarily slow when the next review becomes available. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![protoquinn[bot]](https://avatars.githubusercontent.com/in/3282126?s=60&v=4){kind=small}
There was a problem hiding this comment.
QA Audit — PR #321 | test(e2e): fail fast on a bad gateway credential (#226)
INTERIM REVIEW — CI not yet terminal; formal verdict held.
Diff Review
- Adds
authPreflight()tointegration-tests/globalSetup.ts, called at end ofsetup(). - Guards on
OPENAI_API_KEY+OPENAI_BASE_URLpresence (early-return if absent) — local runs without creds unaffected. - Probes
${baseUrl}/modelswith a 10sAbortSignal.timeoutand Bearer auth. - Network errors →
console.warn+ early return (transient infra); HTTP 401/403 → throws with a clear diagnostic message citingprotoCLI#226. - Error message correctly names both env vars and explains the downstream symptom ("CLI process exited with code 1"), making the root cause actionable rather than opaque.
Observations
- The logic is a clean, non-invasive addition. No production code touched; no test assertions changed; no new surface area beyond the preflight itself.
- The
console.warnpath for network errors is intentional and appropriate — transient DNS or proxy failures should not block a CI run. - CLAWPATCH unavailable (invalid SHA
mainon checkout cache); diff-based review is sufficient for this scope.
Will re-check CI and promote to APPROVED once checks are terminal.
— Quinn, QA Engineer
|
Submitted COMMENT review on |
Contributor
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
When the E2E gateway credential is expired/revoked, every model-driving test spawns the CLI, which aborts on a failed auth preflight (HTTP 401) and exits 1 — surfacing as ~200 opaque
Error: CLI process exited with code 1assertion failures and a 25-minute red run with no obvious cause. This is exactly the current state of #226.This adds a one-shot auth preflight in
integration-tests/globalSetup.tsthat probes the gateway's/modelsonce. A rejected credential (401/403) fails the whole suite immediately with a single clear message naming the secret to rotate.Behavior
OPENAI_API_KEY+OPENAI_BASE_URL) are present — a missing key is left to individual tests (some auth paths don't use these vars), and local runs without creds are unaffected.checkAuthPreflightlogic.Scope
This does not fix the credential itself — that's a repo-secret rotation (tracked in #226, where I posted the root-cause diagnosis). It makes the next bad-credential incident diagnosable in one line instead of a 20-minute red haystack.
Validation
Refs #226.
🤖 Generated with Claude Code