t4925: Model-level backoff in headless-runtime-helper.sh

[alex-solovyev]

Summary

• Change backoff granularity from provider-level to model-level so that rate-limiting one model (e.g. claude-sonnet) doesn't block other models from the same provider (e.g. claude-opus)
• Auth errors still back off at provider level since credentials are shared across models
• Fix file_mtime() stat order bug on Linux that caused auth signatures to change between calls, silently clearing backoff state

Changes

headless-runtime-helper.sh

• record_provider_backoff(): New 4th param model. For auth_error, keys on provider name. For rate_limit/provider_error, keys on full model ID (e.g. anthropic/claude-sonnet-4-6)
• backoff_active_for_key(): New internal function that checks a specific backoff key against the DB
• model_backoff_active(): New function that checks both model-level backoff (rate limits) and provider-level backoff (auth errors)
• choose_model(): Uses model_backoff_active() instead of provider_backoff_active() — checks per-model, not per-provider
• cmd_backoff CLI: Accepts both model IDs (anthropic/claude-sonnet-4-6) and provider names (anthropic)
• cmd_run(): Passes full model ID to record_provider_backoff()
• file_mtime(): Fix stat order — try stat -c '%Y' (Linux) before stat -f '%m' (macOS). On Linux, stat -f '%m' returns filesystem free blocks, not file mtime
• Sandbox bypass: AIDEVOPS_HEADLESS_SANDBOX_DISABLED=1 env var to skip sandbox (needed for tests)

tests/test-headless-runtime-helper.sh

• Set AIDEVOPS_HEADLESS_MODELS explicitly for self-contained test runs
• Disable sandbox via AIDEVOPS_HEADLESS_SANDBOX_DISABLED=1 (sandbox strips STUB_* env vars)
• 4 new tests: sonnet rate-limited with opus available, both models backed off, clear individual model backoff, auth error backs off all provider models
• All 16 tests pass (12 existing + 4 new)

Before/After

Before: AIDEVOPS_HEADLESS_MODELS="anthropic/claude-sonnet-4-6,anthropic/claude-opus-4-6" — sonnet gets 429 → backoff recorded for anthropic → opus skipped → worker fails

After: sonnet gets 429 → backoff recorded for anthropic/claude-sonnet-4-6 → opus checked separately → opus selected → worker runs

Closes #4925

Summary by CodeRabbit

Release Notes

• Improvements
  • Refined backoff logic to track models individually, allowing more granular control over model availability
  • Authentication errors continue to affect all models from a provider
  • Added environment variable to optionally disable sandboxed execution
  • Enhanced test coverage for backoff and authentication scenarios

[gemini-code-assist]

Warning

You have reached your daily quota limit. Please wait up to 24 hours and I will start processing your requests again!

[coderabbitai]

Caution

Review failed

Pull request was closed or merged during review

Walkthrough

Refactored backoff mechanism from provider-level to model-level granularity. Models from the same provider now have independent rate-limit backoffs, while auth errors still trigger provider-wide blocks. Added model-level backoff check function and updated storage to use model keys instead of provider keys.

Changes

Cohort / File(s)	Summary
Model-Level Backoff Refactoring .agents/scripts/headless-runtime-helper.sh	Introduced model_backoff_active() and refactored backoff_active_for_key(key, provider) to handle model-scoped checks. Updated record_provider_backoff() to accept a model/key parameter and derive backoff_key for DB operations. Modified choose_model(), cmd_backoff(), and cmd_run() to operate on model-level granularity. Added AIDEVOPS_HEADLESS_SANDBOX_DISABLED support for test sandbox control.
Model-Level Backoff Test Coverage tests/test-headless-runtime-helper.sh	Added test-time configuration for models and sandbox disablement. Introduced "Model-Level Backoff" test block validating per-model backoff independence (sonnet/opus). Added "Auth Error Backs Off Provider" test block ensuring auth errors trigger provider-wide blocks. Updated backoff validation patterns to reflect model-level semantics.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🎭 One model stumbles on rate-limit's wall,
Yet siblings from the same provider still stand tall,
No longer chained by a provider's plight—
Each model now backoffs just right,
Auth errors still bound them as one,
But their rate-limit struggles are done! ✨

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 10.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately and specifically describes the main change: introducing model-level backoff granularity in headless-runtime-helper.sh, referencing issue #4925.
Linked Issues check	✅ Passed	The PR fully implements all coding requirements from issue #4925: model-level backoff keying, provider-level auth error handling, model_backoff_active() checks, choose_model() integration, and file_mtime() stat fix.
Out of Scope Changes check	✅ Passed	All changes directly support model-level backoff objectives. The sandbox configuration change and test-only environment variables are necessary test infrastructure and not extraneous to PR scope.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

• 📝 Generate docstrings (stacked PR)
• 📝 Generate docstrings (commit on current branch)

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feature/model-level-backoff

📝 Coding Plan

• Generate coding plan for human review comments

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[github-actions]

🔍 Code Quality Report

�[0;35m[MONITOR]�[0m Code Review Monitoring Report

�[0;34m[INFO]�[0m Latest Quality Status:
SonarCloud: 0 bugs, 0 vulnerabilities, 361 code smells

�[0;34m[INFO]�[0m Recent monitoring activity:
Sun Mar 15 03:38:15 UTC 2026: Code review monitoring started
Sun Mar 15 03:38:15 UTC 2026: SonarCloud - Bugs: 0, Vulnerabilities: 0, Code Smells: 361

📈 Current Quality Metrics

• BUGS: 0
• CODE SMELLS: 361
• VULNERABILITIES: 0

Generated on: Sun Mar 15 03:38:17 UTC 2026

---

Generated by AI DevOps Framework Code Review Monitoring

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud