Skip to content

feat(cli+tui): provider quota and rate-limit in status bar - #53375

Open
rafaumeu wants to merge 1 commit into
NousResearch:mainfrom
rafaumeu:feat/quota-cost-statusbar-clean
Open

feat(cli+tui): provider quota and rate-limit in status bar#53375
rafaumeu wants to merge 1 commit into
NousResearch:mainfrom
rafaumeu:feat/quota-cost-statusbar-clean

Conversation

@rafaumeu

Copy link
Copy Markdown

Summary

Closes #53306

Adds provider quota/cost tracking to both CLI and TUI status bars with a dual-source approach:

Changes

agent/account_usage.py (+87 lines)

  • Add _fetch_zai_account_usage() — fetches Z.AI (Zhipu) token quota via the monitoring endpoint /api/monitor/usage/quota/limit
  • Base URL heuristic in fetch_account_usage(): when the provider name is not directly recognised (e.g. xai-oauth hitting api.z.ai), resolves to zai by inspecting the host. Falls back to custom:zai for API key resolution.

cli.py (+46 lines)

  • Quota/cost segment in _get_status_bar_snapshot() using dual-source pattern:
    1. fetch_account_usage() as primary (queries provider quota APIs)
    2. rate_limit_tracker.format_rate_limit_compact() as fallback (parses rate-limit headers from last response)
  • Timezone-safe reset timer: uses datetime.now(timezone.utc) and forces timezone.utc on naive reset_at values, so the countdown is correct regardless of system timezone (tested across UTC-3, UTC, UTC+9).

tui_gateway/server.py (+41 lines)

  • Same quota block in _get_usage() for the TUI WebSocket payload (quota_pct, quota_reset, quota_rl_text fields).
  • Same timezone-safe UTC comparison.

Design decisions

  • All quota fetches are wrapped in try/except — failure is silent, status bar simply omits the quota segment.
  • The rate_limit_tracker fallback ensures providers without a quota API still show useful info.
  • Provider normalisation uses a whitelist of known provider names; unrecognized names with a matching base_url host get resolved automatically.

Tested

  • Z.AI quota endpoint returns TOKENS_LIMIT and TIME_LIMIT with percentage and nextResetTime (epoch ms).
  • Timezone simulation confirms identical countdown across São Paulo (UTC-3), UTC, and Tokyo (UTC+9).
  • Syntax check passes (ast.parse) on all 3 files.
  • Lint: no new pyright errors (pre-existing type issues only).

@alt-glitch alt-glitch added type/feature New feature or request comp/cli CLI entry point, hermes_cli/, setup wizard comp/tui Terminal UI (ui-tui/ + tui_gateway/) comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint provider/zai ZAI provider P3 Low — cosmetic, nice to have labels Jun 27, 2026
@alt-glitch

Copy link
Copy Markdown
Collaborator

This was generated by AI during triage.

Implements feature request #53306 (all-provider quota/rate-limit status-bar segment). Related to the Codex-only status-bar cluster (#18958, #34978); broader all-provider scope, not a duplicate. Maintainers may want to consolidate.

@tonydwb tonydwb left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review Summary

Verdict: Approved

Good addition of provider quota and rate-limit display in the status bar. The new _fetch_zai_account_usage function follows the established pattern for other providers (OpenRouter, etc.) with proper error handling and timeout.

Looks Good

  • Follows established pattern for provider usage fetching
  • Proper timeout and error handling
  • No security concerns
  • Single-concern feature addition

Reviewed by Hermes Agent

@rafaumeu
rafaumeu force-pushed the feat/quota-cost-statusbar-clean branch from 01c57b1 to 5e3f4fa Compare June 30, 2026 02:37
@rafaumeu

rafaumeu commented Jul 5, 2026

Copy link
Copy Markdown
Author

Multi-provider rate-limit support

Research findings

Tested all 8 providers in our config with real API calls. Results:

Provider Dedicated quota endpoint Rate-limit headers
Z.AI /api/paas/v4/charges/token No
OpenRouter /api/v1/credits No
Anthropic OAuth usage endpoint No
Codex Internal No
Groq None x-ratelimit-{limit,remaining,reset}-{requests,tokens}
Mistral None x-ratelimit-{limit,remaining}-{req,tokens}-minute
Cerebras None x-ratelimit-{limit,remaining}-{requests,tokens}-{minute,hour,day}
SambaNova None x-ratelimit-{limit,remaining,reset}-requests-day
NVIDIA None None
Google None None
Cloudflare None None

What was done (3rd commit in this PR)

Since Groq, Mistral, Cerebras and SambaNova don't have dedicated quota/usage endpoints but do send rate-limit headers on chat completions, the right approach is to make the existing rate_limit_tracker parser multi-provider aware:

  1. Extended parse_rate_limit_headers() — added _best_bucket() helper that tries canonical format first, then provider-specific header naming variants (req-minute, requests-minute, tokens-minute, requests-day, etc.)
  2. SambaNova daily buckets promoted to hour-level when no hourly data exists
  3. Fixed format_rate_limit_compact() call without argument — both cli.py and tui_gateway/server.py fallback paths now properly fetch agent.get_rate_limit_state() (or _rate_limit_state attribute) before formatting

Tested with real headers

All 5 provider formats parse correctly:

  • Groq: RPM: 14399/14400 | TPM: 6.0K/6.0K
  • Mistral: RPM: 187/188 | TPM: 625.0K/625.0K
  • Cerebras: RPM: 4/5 | RPH: 149/150 | TPM: 30.0K/30.0K | TPH: 1000.0K/1.0M
  • SambaNova: RPH: 19/20

NVIDIA, Google, and Cloudflare don't expose rate-limit data at all — they will show nothing in the status bar (no API to call).

- Add Z.AI, Cloudflare, Google, NVIDIA quota fetchers in account_usage.py
- Add quota/cost segment in CLI status bar with TTL cache (60s)
- Add quota/cost fields in TUI gateway usage payload
- Dual-source: fetch_account_usage primary, rate_limit_tracker fallback
- Timezone-safe reset timer using datetime.now(timezone.utc)

Closes NousResearch#53306
@rafaumeu
rafaumeu force-pushed the feat/quota-cost-statusbar-clean branch from fef7ecc to bd70f5a Compare July 8, 2026 00:53

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for extending the existing account-usage and rate-limit work. The gap is real: current main only surfaces these details through /usage (cli.py:9647-9710), not the status bars.

Problems

  • The TUI portion is payload-only: tui_gateway/server.py:3171 adds fields, but the PR changes no ui-tui/ files. Current Usage has no quota fields (ui-tui/src/types.ts:170-184) and StatusRule does not render them (ui-tui/src/components/appChrome.tsx:426-440).
  • The CLI renderer synchronously calls the fetcher through cli.py:4702; the new fetchers use 10-second httpx.Client calls (for example agent/account_usage.py:687-692). This conflicts with #53306's requirement that render paths avoid network I/O.
  • The cache is process-global rather than provider/session keyed (cli.py:52, tui_gateway/server.py:3063), so one session can receive another session's quota. The TUI also retries empty results every usage update because {} is falsy at tui_gateway/server.py:3063.
  • The new helpers infer provider from agent.model (cli.py:57, tui_gateway/server.py:3068) instead of the canonical agent.provider used by current /usage (cli.py:9691).

Suggested changes

  • Refresh keyed snapshots asynchronously and keep rendering read-only.
  • Use agent.provider, add the Ink contract/rendering, and cover parser, cache, and UI behavior with tests.

Automated hermes-sweeper review.

Comment thread cli.py
"""Fetch provider quota with a TTL cache to avoid blocking the render loop."""
global _quota_cache, _quota_cache_ts
now = time.monotonic()
if _quota_cache is not None and (now - _quota_cache_ts) < QUOTA_CACHE_TTL:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This cache is process-global and has no provider, credential, or session key. A second agent rendered within 60 seconds will receive the first agent's quota snapshot. Cache per active provider/account identity instead.

Comment thread cli.py
result: Dict[str, Any] = {}
try:
from agent.account_usage import fetch_account_usage
_provider = (getattr(agent, "model", None) or "").split("/")[0]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use agent.provider as the primary input here. The established /usage path does that at current cli.py:9691; deriving it from a model string drops provider identity whenever the model lacks a provider/model prefix.

Comment thread tui_gateway/server.py
"""Cached quota fetch for TUI gateway — same TTL pattern as CLI."""
global _tui_quota_cache, _tui_quota_cache_ts
now = time.monotonic()
if _tui_quota_cache and (now - _tui_quota_cache_ts) < TUI_QUOTA_CACHE_TTL:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An unavailable provider produces {} at line 3107, but {} is falsy, so this guard refetches on every _get_usage invocation instead of observing the TTL. Test cache initialization separately from cache contents.

Comment thread tui_gateway/server.py
pass
# --- quota / cost (cached) ---
try:
usage.update(_get_tui_quota_snapshot(agent))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only adds fields to the backend payload. The PR changes no ui-tui/ files; current ui-tui/src/types.ts has no quota fields and ui-tui/src/components/appChrome.tsx does not render them, so the advertised TUI status-bar segment remains invisible.

@teknium1 teknium1 added sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard comp/tui Terminal UI (ui-tui/ + tui_gateway/) P3 Low — cosmetic, nice to have provider/zai ZAI provider sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:risk-security-boundary Sweeper risk: may affect sandboxing, auth, credentials, or sensitive data type/feature New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(cli): show provider quota and rate-limit in status bar for all providers

4 participants