feat(resilience): credential health check + adaptive circuit breaker

[herjarsa]

Two resilience enhancements:\n1. Fail-Fast Credential Health Check: background scheduler validates connections, eliminates stale credential latency\n2. Circuit Breaker v2: progressive DEGRADED state, adaptive backoff by failure kind, automatic HALF-OPEN probing

[gemini-code-assist]

Code Review

This pull request introduces Circuit Breaker v2.0, featuring a new 'DEGRADED' state, adaptive exponential backoff, and per-failure-kind thresholds. It also implements a background credential health check system with an in-memory cache and a 'Credential Gate' to skip targets with known-bad credentials. Key feedback includes addressing a ReferenceError in combo.ts due to block-scoped variables, fixing unreachable event emission code, and correcting an emit call mistakenly passed as a function argument. Additionally, the health check scheduler should be updated to ensure that backoff for a single failing connection does not globally delay checks for healthy connections.

[gemini-code-assist]

The variable requestBody is declared using const inside a nested if block (line 2174), which limits its scope to that block. Furthermore, if the condition at line 2172 is not met, the variable is never declared at all. This will cause a ReferenceError at line 2183 when attempting to access requestBody. You should declare requestBody with let outside the if block.

Suggested change

	// Universal handoff: inject existing handoff if model changed
	if (
	universalHandoffConfig.enabled &&
	relayOptions?.sessionId &&
	!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
	) {
	const lastModel = getLastSessionModel(relayOptions.sessionId, combo.name);
	if (lastModel && lastModel !== modelStr) {
	const existingHandoff = getHandoff(relayOptions.sessionId, combo.name);
	const requestBody = injectUniversalHandoffBody(
	body,
	lastModel,
	modelStr,
	`Model routing: ${lastModel} → ${modelStr}`,
	existingHandoff
	);
	}
	}
	const result = await handleSingleModelWrapped(requestBody ?? body, modelStr, {
	// Universal handoff: inject existing handoff if model changed
	let requestBody: any = null;
	if (
	universalHandoffConfig.enabled &&
	relayOptions?.sessionId &&
	!(body as Record<string, unknown>)?.[SKIP_UNIVERSAL_HANDOFF_FLAG]
	) {
	const lastModel = getLastSessionModel(relayOptions.sessionId, combo.name);
	if (lastModel && lastModel !== modelStr) {
	const existingHandoff = getHandoff(relayOptions.sessionId, combo.name);
	requestBody = injectUniversalHandoffBody(
	body,
	lastModel,
	modelStr,
	`Model routing: ${lastModel} → ${modelStr}`,
	existingHandoff
	);
	}
	}
	const result = await handleSingleModelWrapped(requestBody ?? body, modelStr, {

[gemini-code-assist]

The scheduleSweep function calculates the next sweep interval based on the maximum failure count across all connections. This causes a single failing connection to slow down the health checks for all other healthy connections (e.g., if one connection is in a 2-hour backoff, all connections will only be checked every 2 hours). The scheduler should maintain a constant base interval, and the sweep function should independently decide which connections are due for testing based on their individual backoff state.

[gemini-code-assist]

The emit("combo.target.failed", ...) call is unreachable because it is placed immediately after a break statement. It should be moved before the break to ensure the failure event is correctly recorded when quality validation fails.

Suggested change

	break; // move to next model
	emit("combo.target.failed", {
	comboName: combo.name,
	targetIndex: i,
	provider,
	model: modelStr,
	error: `Quality: ${quality.reason}`,
	latencyMs: Date.now() - startTime,
	});
	emit("combo.target.failed", {
	comboName: combo.name,
	targetIndex: i,
	provider,
	model: modelStr,
	error: `Quality: ${quality.reason}`,
	latencyMs: Date.now() - startTime,
	});
	break; // move to next model

[gemini-code-assist]

The emit("credential.health.changed", ...) call is incorrectly passed as an 8th argument to setCredentialHealth. Since the function only accepts 7 arguments, the return value of emit is passed as an extra argument and ignored. More importantly, emit is executed during argument evaluation rather than after the health status is updated. It should be moved to a separate statement.

Suggested change

	setCredentialHealth(
	connectionId,
	provider,
	"active",
	undefined,
	undefined,
	undefined,
	latencyMs,
	emit("credential.health.changed", {
	connectionId,
	provider,
	oldStatus: oldStatus || "unknown",
	newStatus: "active",
	timestamp: Date.now(),
	});
	);
	setCredentialHealth(
	connectionId,
	provider,
	"active",
	undefined,
	undefined,
	undefined,
	latencyMs,
	);
	emit("credential.health.changed", {
	connectionId,
	provider,
	oldStatus: oldStatus || "unknown",
	newStatus: "active",
	timestamp: Date.now(),
	});

[kilo-code-bot]

Kilo Code Review could not run — your account is out of credits.

Add credits or switch to a free model to enable reviews on this change.

[diegosouzapw]

Thank you! PR merged.