Skip to content

fix(cli): guard expensive startup model overrides - #70324

Closed
lkz-de wants to merge 1 commit into
NousResearch:mainfrom
lkz-de:fix/cli-startup-model-cost-guard
Closed

fix(cli): guard expensive startup model overrides#70324
lkz-de wants to merge 1 commit into
NousResearch:mainfrom
lkz-de:fix/cli-startup-model-cost-guard

Conversation

@lkz-de

@lkz-de lkz-de commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • run the expensive-model guard for startup -m / --provider overrides before the chat loop starts
  • fail closed for non-interactive startup overrides that select an expensive or known-confusing route
  • make the openai/gpt-5.5-pro warning fire even when no pricing entry is available, so the Codex gpt-5.5 suggestion still appears
  • classify Nous “requires available credits” 404 responses as billing exhaustion so they fail fast with billing guidance instead of generic retry noise

Why

Hermes already warns on interactive model switches before moving to very expensive models, including the easy-to-miss openai/gpt-5.5-pro vs Codex gpt-5.5 distinction. Direct startup flags did not pass through that interactive picker path, so a command-line invocation could choose the aggregator-style openai/gpt-5.5-pro model without seeing the same confirmation.

That distinction matters because these are different provider routes, not just two spellings of the same model:

  • provider=openai-codex with bare model gpt-5.5 uses the Codex OAuth route configured through hermes auth add openai-codex.
  • API-key / aggregator routes such as Nous or OpenRouter use provider billing and provider-specific model ids such as openai/gpt-5.5-pro.

A user with Codex OAuth configured can therefore still accidentally send a startup invocation to a paid API/aggregator route if the CLI accepts a provider/model override without applying the same warning used by the interactive picker.

This PR closes that gap by checking explicit startup overrides before any provider call is constructed. Non-interactive invocations fail closed; interactive invocations must confirm before continuing. The guard also covers provider-only startup overrides by evaluating the effective configured default model.

The warning fallback and billing classifier keep the same failure path legible if an override does reach a provider: the known openai/gpt-5.5-pro suggestion should appear even if pricing lookup has no entry, and a Nous “requires available credits” response for the misrouted paid model should fail fast as billing exhaustion rather than spend retries as an unknown 404.

Testing

  • scripts/run_tests.sh tests/hermes_cli/test_cli_startup_model_cost_guard.py tests/hermes_cli/test_model_cost_guard.py tests/agent/test_error_classifier.py -- --tb=short -q

Coverage added includes non-interactive startup rejection, interactive confirm/cancel/EOF handling, provider-only startup override evaluation, openai/gpt-5.5-pro warning without pricing data, and Nous “requires available credits” billing classification.

Overlap check

Related open PRs touch adjacent GPT-5.5 or model-picker behaviour, but none cover this startup-override path:

No open PR found for guarding startup -m / --provider expensive-model overrides or for classifying the Nous “requires available credits” 404 as billing exhaustion.

@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard provider/nous Nous Research API (OAuth) area/billing Account usage, credit usage, billing (cross-cutting) labels Jul 23, 2026

@teknium1 teknium1 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for covering the direct cmd_chat startup path and the missing Nous billing phrase. The underlying gap is still present on current main: cmd_chat forwards the raw overrides into _launch_tui/cli_main without invoking expensive_model_warning (hermes_cli/main.py:2656-2701).

Problems

  • The fail-closed behavior does not cover -z/--oneshot. The parser explicitly accepts -m/--model and --provider for oneshot (hermes_cli/_parser.py:127-146), but main sends that path directly to run_oneshot (hermes_cli/main.py:12362-12371) rather than cmd_chat. A non-interactive -z call can therefore still select the route this PR intends to prevent.

Suggested changes

  • Share the startup guard with run_oneshot (stderr only, so its stdout contract remains intact) and add a -z regression test for rejection plus a safe-model control case.

Automated hermes-sweeper review.

Comment thread hermes_cli/main.py
@@ -2592,6 +2653,7 @@ def cmd_chat(args):
os.environ["HERMES_SESSION_SOURCE"] = args.source

_pin_kanban_board_env()
_confirm_startup_expensive_model_override(args)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This only protects the cmd_chat route. -z/--oneshot accepts the same model/provider flags but dispatches directly to run_oneshot, so a non-interactive oneshot invocation can still select the expensive route without this guard. Please share the check with the oneshot path and add coverage there.

@teknium1 teknium1 added sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform labels Jul 30, 2026
Run the expensive-model warning for explicit startup `-m` / `--provider`
overrides before the chat loop starts, and fail closed for non-interactive
invocations that select an expensive or known-confusing model.

Also classify Nous paid-model 404s that say credits are required as billing
exhaustion so they fail fast with billing guidance.

Tested:
- scripts/run_tests.sh tests/hermes_cli/test_cli_startup_model_cost_guard.py tests/hermes_cli/test_model_cost_guard.py tests/agent/test_error_classifier.py -- --tb=short -q
@lkz-de
lkz-de force-pushed the fix/cli-startup-model-cost-guard branch from 5a9dd18 to 31b1568 Compare July 31, 2026 21:40
@lkz-de

lkz-de commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

For the startup-override path: I rechecked the current diff against the earlier oneshot concern, and the guard is already on the top-level --oneshot / -z path.

The current branch calls _confirm_startup_expensive_model_override(args) before _run_and_exit_oneshot(...), and the focused regression is test_top_level_oneshot_rejects_noninteractive_gpt55_pro_startup_override. So I do not think there is a remaining oneshot bypass in this PR as it stands.

teknium1 added a commit that referenced this pull request Aug 14, 2026
…over the light oneshot fast-path

Follow-ups on top of the salvaged #70324:
- _confirm_startup_expensive_model_override evaluates the unified
  registry (combined_selection_warning) so id-keyed guards like the
  data-training-tier warning fire at startup too, not just the cost guard.
- The Termux-adjacent light oneshot fast-path (added after the PR
  branched) ran _run_and_exit_oneshot without the guard — same bug
  class, third sibling site now covered.
@teknium1

Copy link
Copy Markdown
Contributor

Merged via PR #85953 (#85953) — thanks @lkz-de!

Your commit was cherry-picked onto current main with your authorship preserved in git log. On top of it we (a) routed the startup guard through the new unified selection-guard registry (#85917), so the data-training-tier warning also fires at startup, and (b) covered a third oneshot fast-path that was added to main.py after your branch point and had no guard at all — your fail-closed non-interactive design carried over unchanged.

The "requires available credits" billing classification and the gpt-5.5-pro no-pricing suggestion both landed as-is. Nice, complete work.

@teknium1 teknium1 closed this Aug 14, 2026
teknium1 added a commit that referenced this pull request Aug 14, 2026
…rust and gpt-5.5-pro confusion nudge (#85970)

54cc39a (distrust foreign pricing for custom providers) tested with
openai/gpt-5.5-pro fixtures; 83d373a (salvaged #70324) made that exact
id warn unconditionally as a known-confusion model. Each was green alone;
together the distrust tests fail on every main run (slice 6).

Use a neutral fixture id for the distrust tests and add a regression test
pinning the composed behavior: the id-keyed nudge survives custom-provider
pricing distrust.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/billing Account usage, credit usage, billing (cross-cutting) comp/agent Core agent runtime: loop, agent_init, prompt builder, context-compression, responses endpoint comp/cli CLI entry point, hermes_cli/, setup wizard P2 Medium — degraded but workaround exists provider/nous Nous Research API (OAuth) sweeper:blast-moderate Sweeper blast radius: moderate — a subsystem or single platform sweeper:risk-compatibility Sweeper risk: may break existing users, config, migrations, defaults, or upgrades type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants