Skip to content

fix(agent): preemptively pace low RPM buckets - #78612

Open
konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/preemptive-rpm-throttling-followup
Open

konsisumer wants to merge 1 commit into
NousResearch:mainfrom
konsisumer:fix/preemptive-rpm-throttling-followup

Conversation

@konsisumer

Copy link
Copy Markdown
Contributor

What changed and why

Per the follow-up on #7489, this is a clean, smaller successor to Refs #7490. It captures rate-limit headers for non-streaming OpenAI-compatible calls, parses OpenAI compact durations and Anthropic RFC 3339 reset values, then delays the next request when a matching request bucket has at most two requests remaining.

The captured state is in-memory and is accepted only when both the active provider and normalized base URL still match, so a provider or route switch cannot inherit an unrelated bucket. OpenRouter is intentionally not enabled by default because successful routed responses do not reliably provide the necessary headers. A custom provider may opt in with rpm_throttle_threshold on its configured route.

Scope justification: the 5-file / 347-line diff is limited to the existing rate-limit state model (parsing plus a pure, interruptible wait), the two existing API entry paths (non-streaming capture and preflight), the one capture call that records route identity, custom-provider normalization/lookup, and focused regressions. It adds no new manager, provider integration, dependency, or configuration migration.

How to test

  • pytest -q --timeout=60 tests/agent/test_rate_limit_tracker.py tests/agent/test_cron_inline_api_call_62151.py tests/run_agent/test_anthropic_response_header_capture.py (16 passed)
  • ruff check agent/rate_limit_tracker.py agent/chat_completion_helpers.py run_agent.py hermes_cli/config.py tests/agent/test_rate_limit_tracker.py
  • python -m py_compile agent/rate_limit_tracker.py agent/chat_completion_helpers.py run_agent.py hermes_cli/config.py tests/agent/test_rate_limit_tracker.py

The requested full suite was attempted, but collection is blocked in this worktree because FastAPI is absent and its automatic install is rejected by the externally managed Homebrew Python.

What platforms tested on

  • macOS (Homebrew Python 3.14.6)

Related reference

Refs #7489

@alt-glitch alt-glitch added type/feature New feature or request P3 Low — cosmetic, nice to have comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint area/config Config system, migrations, profiles labels Aug 4, 2026
@konsisumer

Copy link
Copy Markdown
Contributor Author

Rebased onto current origin/main and resolved the shared streaming entrypoint by preserving upstream stream-lifecycle helpers alongside the RPM preflight. The PR remains its original five-file scope.

Focused regressions: 16 passed. Ruff, Windows-footgun scan, bytecode compilation, and git diff --check pass. The full suite cannot collect in this sandbox because FastAPI is absent and its automatic install is blocked.

@konsisumer
konsisumer force-pushed the fix/preemptive-rpm-throttling-followup branch from b135b22 to 14e4ee6 Compare August 13, 2026 07:27

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint P3 Low — cosmetic, nice to have type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants