Skip to content

test(autoCombo): raise timeout on tieredRotation stress tests - #203

Merged
KooshaPari merged 1 commit into
mainfrom
fix/tiered-rotation-timeout
Jul 2, 2026
Merged

KooshaPari merged 1 commit into
mainfrom
fix/tiered-rotation-timeout

Conversation

@KooshaPari

Copy link
Copy Markdown
Owner

Problem

Three stress tests in tests/unit/autoCombo/tieredRotation.test.ts hit vitest's default 5000ms per-test limit and time out:

  1. smart combo rotates within top tier across many requests (50 iterations, 4-candidate pool)
  2. cheap combo pulls from rest tier (lower scores) more often than smart (200 iterations, 2-candidate pool)
  3. rotates across all 43 Cerebras connection IDs, not just one (43-candidate pool, 200 iterations)

The first getTaskFitness() call in the file boots the SQLite migration runner (~2.6s one-time DB init). Combined with vitest's collection/pool overhead and the per-test iteration work, each test blows past the 5s default.

Fix (Option A — per-test timeout)

Added { timeout: 30000 } to each of the three it(...) calls. Chosen over reducing iteration counts (Option C) because Option A preserves the sample sizes and cannot alter the tests' statistical validity — the tests still exercise the exact same rotation behavior they were written to verify. Tests are not skipped or disabled.

Validation

The file is vitest-native (vi.mock at the top), and the vitest runner is currently broken in worktrees (EXIT=194, zero output — tracked separately). Per Hard Rule #18, the rotation logic + timing were validated with a standalone tsx timing probe reproducing all three tests' bodies:

test3 connections seen: 43 (need >=10)  elapsed(incl DB init): 2640 ms
test1 seen.size: 4 (need >=2)  hasTop: true   elapsed(warm): 27 ms
test2 cheap count: 78 (need >0)  elapsed(warm): 115 ms

All three assertions pass; the DB-init cost (~2.6s) is what pushes the file over the 5s default. The 30000ms budget resolves it with no change to test behavior.

The 3 rotation stress tests pay a one-time ~2.6s DB init (first
getTaskFitness call boots the SQLite migrations) plus many
selectProvider iterations. Combined with vitest collection overhead
this exceeds vitest's default 5000ms per-test limit and the tests
time out. Raise each to a 30000ms budget rather than thinning the
sample, so iteration counts and statistical validity are unchanged.
@gemini-code-assist

Copy link
Copy Markdown

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

@KooshaPari
KooshaPari merged commit efe73f0 into main Jul 2, 2026
9 of 20 checks passed
@KooshaPari
KooshaPari deleted the fix/tiered-rotation-timeout branch July 2, 2026 09:44
@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown

Warning

Review limit reached

@KooshaPari, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 25 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 91a52bc0-f7cf-471f-b014-346c076e9c72

📥 Commits

Reviewing files that changed from the base of the PR and between c29654c and 8670fbe.

📒 Files selected for processing (1)
  • tests/unit/autoCombo/tieredRotation.test.ts

Note

.coderabbit.yaml has unrecognized properties

CodeRabbit is using all valid settings from your configuration. Unrecognized properties (listed below) have been ignored and may indicate typos or deprecated fields that can be removed.

⚠️ Parsing warnings (1)
Validation error: Unrecognized key: "review"
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/tiered-rotation-timeout

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Jul 2, 2026

Copy link
Copy Markdown

L17 Latency Budget Report

--- Latency Budget Summary ---
  Total endpoints checked: 0
  Passed: 0
  Warnings: 0
  Failures: 0

Checked against: budgets/rest-endpoints.yaml.

@sonarqubecloud

sonarqubecloud Bot commented Jul 2, 2026

Copy link
Copy Markdown

@kilo-code-bot

kilo-code-bot Bot commented Jul 4, 2026 •

Copy link
Copy Markdown

Code Review Summary

Status: No blocking issues in changed code | Recommendation: Not actionable — PR is already merged

Overview

Severity Count
CRITICAL 0
WARNING 0
SUGGESTION 0
Files Reviewed (1 file)
  • tests/unit/autoCombo/tieredRotation.test.ts — Timeout additions only; no issues in diffsed lines.

Note

This PR is already in state MERGED and cannot be un-merged via review feedback.
The change correctly preserves test statistical validity while working around the ~2.6s one-time SQLite migration init on first getTaskFitness() call. Worth tracking as a follow-up: the underlying slow-first-call cost is unaddressed; if that cost grows, tests will time out again. Consider future work to initialize the DB runner in test setup rather than lazily in production code.


Reviewed by step-3.7-flash-20260528 · Input: 54.4K · Output: 4.5K · Cached: 311.7K

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant