perf(onboard): calibrate provider validation timeouts

[HOYALIM]

Refs #3770.

Summary

• add an opt-in OpenAI-compatible provider validation calibration probe before onboard validation requests
• derive standard validation curl timeout budgets from observed transport latency, with a safe widened fallback when calibration cannot complete
• keep direct probe callers on existing defaults unless they opt in, preserve the WSL fallback path, and record the selected timing profile in existing NEMOCLAW_TRACE output

Notes

• calibration uses a secret-free /models request and treats any HTTP response as a useful transport timing sample
• special slow-model timeout budgets remain unchanged; the adaptive profile only covers the standard validation path

Validation

• npx @biomejs/biome check src/lib/inference/probe-http-helpers.ts src/lib/inference/probe-http-helpers.test.ts src/lib/inference/onboard-probes.ts src/lib/onboard/inference-selection-validation.ts src/lib/onboard/inference-selection-validation.test.ts test/wsl2-probe-timeout.test.ts
• npx vitest run --project cli --project integration src/lib/inference/probe-http-helpers.test.ts src/lib/inference/onboard-probes.test.ts src/lib/onboard/inference-selection-validation.test.ts test/wsl2-probe-timeout.test.ts
• npm run build:cli
• npm run test-size:check
• npm run test-conditionals:scan -- --top 25
• npm run check:diff

Summary by CodeRabbit

• New Features

  • Onboarding validation probes now use calibrated, profile-based connect/max timeouts, adjusting automatically from observed latency for OpenAI-like endpoints.
  • OpenAI-compatible endpoint validation now always enables timeout calibration during probing.

• Bug Fixes

  • Validation probe curl timing is more consistent and safely falls back when calibration data isn’t available.
  • Retry attempts now preserve the calibrated timing behavior.

• Tests

  • Expanded coverage for calibrated vs fallback timing profiles, WSL handling, and retry behavior, including updated probe request expectations.

Signed-off-by: Ho Lim subhoya@gmail.com

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: dbd36e85-14b0-4bdf-bb81-cf527b2cd739

📥 Commits

Reviewing files that changed from the base of the PR and between a8aaecc and b718e6b.

📒 Files selected for processing (8)

• src/lib/inference/onboard-probes.ts
• src/lib/inference/probe-http-helpers.test.ts
• src/lib/inference/probe-http-helpers.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• test/fixtures/strict-tool-call-probe-driver.ts
• test/onboard-selection.test.ts
• test/wsl2-probe-timeout.test.ts

💤 Files with no reviewable changes (3)

• test/onboard-selection.test.ts
• test/wsl2-probe-timeout.test.ts
• test/fixtures/strict-tool-call-probe-driver.ts

🚧 Files skipped from review as they are similar to previous changes (5)

• src/lib/onboard/inference-selection-validation.ts
• src/lib/inference/probe-http-helpers.test.ts
• src/lib/inference/probe-http-helpers.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/inference/onboard-probes.ts

---

📝 Walkthrough

Walkthrough

Adds adaptive validation timeout calibration for onboarding probes, threads the derived timing through probe execution paths, enables calibration in selection validation, and updates tests and fixtures for the new calibration request flow.

Changes

Adaptive Timeout Calibration

Layer / File(s)	Summary
Timing profile builder and curl arg helpers src/lib/inference/probe-http-helpers.ts, src/lib/inference/probe-http-helpers.test.ts	Adds ValidationProbeTimingProfile and buildValidationProbeTimingProfile, updates validation curl arg builders to use the derived timing profile, and adds tests for calibrated and fallback timing values.
Calibration probe and timing option threading src/lib/inference/onboard-probes.ts	Adds timing normalization and calibration helpers, records calibration trace events, and threads calibrated timing through Responses, Chat Completions, streaming, and retry probe paths.
Enable calibration in selection validation and tests src/lib/onboard/inference-selection-validation.ts, src/lib/onboard/inference-selection-validation.test.ts, test/wsl2-probe-timeout.test.ts, test/fixtures/strict-tool-call-probe-driver.ts, test/onboard-selection.test.ts	Forces calibrateTimeouts: true in selection validation probe options, updates expectations, and adds retry and request-accounting tests for calibration flows.

Estimated code review effort: 4 (Complex) | ~60 minutes

Suggested labels: enhancement: feature

Suggested reviewers: cv, jyaunches

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: calibrating provider validation timeouts during onboarding.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/inference/probe-http-helpers.ts`:
- Around line 74-91: The calibrated timing path in
buildValidationProbeTimingProfile is allowing validation maxTimeSeconds to fall
below the existing 15s baseline, which can make /responses and /chat/completions
probes timeout too early. Update the logic in buildValidationProbeTimingProfile
(and any related calibration constants) so the calibrated profile never reduces
validation below 15 seconds, either by raising the computed floor or by
preventing calibration from lowering the existing standard.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 1fa68fbf-13f8-442b-a298-31a829b698a9

📥 Commits

Reviewing files that changed from the base of the PR and between edf69f0 and ae046ad.

📒 Files selected for processing (6)

• src/lib/inference/onboard-probes.ts
• src/lib/inference/probe-http-helpers.test.ts
• src/lib/inference/probe-http-helpers.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• test/wsl2-probe-timeout.test.ts

[apurvvkumaria]

The calibration direction is sound, but the exact head has contributor-compliance and behavioral-test blockers:

1. dco-check is failing because the PR body has no contributor declaration. Please add Signed-off-by: Ho Lim <subhoya@gmail.com> to the PR description yourself; maintainers cannot do that on a contributor's behalf.
2. CLI shard 1 fails on the new behavior. Two onboard-selection assertions do not expect calibrateTimeouts: true, and strict-tool-call-probe-driver.ts still expects one request even though calibration adds another. Update the focused tests to assert the intended two-request/calibrated contract and make the strict probe pass without weakening it.
3. Fixes #3770 still lacks the acceptance item requiring #2001 implementation notes or before/after timing evidence. Add that evidence/update, or change the linkage to a partial/mitigation statement so the issue remains open.
4. The exact-head static gate is also red because test/langchain-deepagents-code-image.test.ts exceeds the 1500-line budget by one line. Rebase onto a green base or split/ratchet that test without raising the ceiling.

Please rerun the focused probe tests and the normal PR gate after these changes.

Resolved on exact head e9400cd: contributor DCO is present; calibrated request-contract tests and strict probe coverage pass; the PR now uses Refs #3770; current main removed the unrelated size-gate failure; all 30 checks are green; and the exact-head security audit is clean.

[cv]

Exact-head review at e9400cd43 is clean. The main sync was conflict-free and preserved the contributor patch byte-for-byte by stable patch ID. All 30 checks are green; DCO and both Verified commits pass; CodeRabbit has no unresolved major findings. The nine-category security review confirmed that calibration is secret-free, retains SSRF address pinning and proxy bypass, forbids redirects, preserves bounded curl/process timeouts, cleans temporary response data, and cannot make endpoint validation fail open. Focused evidence: 18 changed tests, 76 integration tests, and 119 curl/SSRF boundary tests passed, plus CLI build/typecheck.