Skip to content

fix(gateway): multiplexed profiles use their own prefill_messages_file and provider_routing - #124630

Open
AhmetArif0 wants to merge 1 commit into
NousResearch:mainfrom
AhmetArif0:fix/multiplex-prefill-provider-routing
Open

AhmetArif0 wants to merge 1 commit into
NousResearch:mainfrom
AhmetArif0:fix/multiplex-prefill-provider-routing

Conversation

@AhmetArif0

Copy link
Copy Markdown

What does this PR do?

On a multiplexed gateway, a routed profile's turns run with the launch (default) profile's prefill_messages_file and provider_routing instead of their own.

GatewayRunner._init_runtime_settings reads both once at boot, outside any profile scope, into self._prefill_messages / self._provider_routing, and every agent the gateway builds takes that copy:

  • gateway/run_turn_runner.py::_build_fresh_agent → prefill_messages=runner._prefill_messages
  • gateway/run_turn_runner.py::run_sync → pr = runner._provider_routing (→ providers_allowed/ignored/order, provider_sort, provider_require_parameters, provider_data_collection)
  • gateway/run_turn.py::_run_background_task_inner (/background) → pr = self._provider_routing

So a secondary profile's own only / ignore / order never apply, and a provider_routing.data_collection: deny set on that profile is silently dropped: its requests go out with the default profile's data-collection preference.

This contradicts multi-profile-gateways.md ("Per-turn runtime settings follow the routed profile … never from the profile the gateway was launched under"). It is also a gap in #108453: that PR listed _load_prefill_messages among the per-turn reads it moved to the active home (819517f), but the loader is only ever called at boot, so the change never took effect for a routed turn.

The fix reads both where the turn runs, inside the routed profile's _profile_runtime_scope. That matches how the same turn already resolves reasoning and service tier, and how the ephemeral system prompt was fixed for the same boot-snapshot problem in #89161. Cron already reads both per job, and the TUI/Desktop backend per agent build; the gateway was the only surface that snapshotted them.

Related Issue

No issue filed. Found while checking the per-turn settings #108453 moved to the routed profile.

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)

Changes Made

  • gateway/run_turn_runner.py: _build_fresh_agent calls runner._load_prefill_messages(); the turn computes pr = runner._load_provider_routing() next to the per-turn reasoning / service-tier resolution.
  • gateway/run_turn.py: /background computes pr = self._load_provider_routing() (it already runs inside _profile_scope_for_source).
  • gateway/run.py: _init_runtime_settings no longer snapshots _prefill_messages / _provider_routing. No production code reads them any more, and keeping a launch-profile copy around would invite the same bug.
  • gateway/run_config_loaders.py: _load_prefill_messages docstring now says it resolves from the active gateway home and must be called per agent build (it said ~/.hermes/).
  • website/docs/user-guide/multi-profile-gateways.md: adds provider_routing (including data_collection) and prefill_messages_file to the list of per-turn settings that follow the routed profile.
  • tests/gateway/test_routed_profile_prefill_routing.py (new).

How to Test

Two profiles, both prefill_messages_file: prefill.json, with different provider_routing (the routed one adds data_collection: deny). The test drives the real GatewayRunner._run_agent → TurnRunner.run_sync → _build_fresh_agent path, and the real _run_background_task, inside the routed profile's _profile_runtime_scope (the scope _profile_scope_for_source enters under multiplexing). The runner also carries the launch profile's boot copy, so the test proves a turn no longer uses it.

Routed profile's agent kwargs main this branch
prefill_messages default-PREFILL ❌ beta-PREFILL ✅
providers_allowed ["default-only"] ❌ ["beta-only"] ✅
provider_data_collection None (dropped) ❌ "deny" ✅
/background providers_allowed / data_collection ["default-only"] / None ❌ ["beta-only"] / "deny" ✅
Launch profile's own turn default values default values (unchanged)

Mutation check. Each change was reverted on its own with the test file kept:

  • prefill read → the main-turn test fails on prefill_messages
  • turn pr → the main-turn test fails on providers_allowed
  • /background pr → the background test fails

With all three restored, both tests pass.

pytest tests/gateway/test_routed_profile_prefill_routing.py -q

Notes / scope

  • Standalone gateway: the values are now read when an agent is built, not only at boot. So an edit to provider_routing / prefill_messages_file reaches newly built agents without a restart, the same as service_tier and fallback_providers today. A cached agent keeps what it was built with. Rebuilding cached agents when routing changes is what fix(gateway): reload provider routing and bust cached agents on routing changes #32088 proposes; this PR does not touch the cache signature.
  • Torn writes: a per-turn read does not drop routing while config.yaml is being written. load_user_config_effective serves the last good parse for a broken file (in-process, then the .good backup).
  • Env override unchanged: HERMES_PREFILL_MESSAGES_FILE stays a process-wide override, like HERMES_EPHEMERAL_SYSTEM_PROMPT.
  • Heads-up for feat(gateway): show configured OpenRouter routing after model switch #110814 (open, P3): it reads getattr(self, "_provider_routing", {}). After this change that attribute no longer exists, so the display would show {}. Calling self._load_provider_routing() fixes that, and also shows the routed profile's routing under multiplexing.

Checklist

Code

  • I've read the Contributing Guide
  • My commit messages follow Conventional Commits
  • I searched for existing PRs to make sure this isn't a duplicate. fix(gateway): reload provider routing and bust cached agents on routing changes #32088 reloads routing per turn for cache-busting on the pre-split run.py; it does not cover prefill or multiplexing.
  • My PR contains only changes related to this fix
  • I've added tests for my changes
  • I've run the relevant tests.
    • Full tests/gateway + related files at base 9bb2ea1b5c: 16 failures on this branch, all of which also fail on main at the same base. main has one extra flaky failure (test_control_socket::test_collect_fleet_versions_falls_back_to_state_file). No new failures.
    • After rebasing onto current main: every test file that builds a runner with these attributes (38 files, plus the new one and test_personality_routed_profile.py) passes, 258 tests.
    • ruff check is clean on the changed files, and so is scripts/check-windows-footguns.py.
  • I've tested on my platform: macOS 26.5 (Apple Silicon), Python 3.14

Documentation & Housekeeping

  • I've updated relevant documentation (multi-profile-gateways.md, loader docstring)
  • cli-config.yaml.example: N/A (no config keys added or changed)
  • CONTRIBUTING.md / AGENTS.md: N/A
  • Cross-platform impact considered: no platform-specific code. The test writes its files with an explicit encoding="utf-8".

…e and provider_routing

GatewayRunner read prefill_messages_file and provider_routing once, in
_init_runtime_settings, from the launch profile's config, and every agent
the gateway built took that copy. Under multiplexing a routed profile's
turns therefore ran with the default profile's prefill messages and
OpenRouter routing: its own only/ignore/order never applied, and a
data_collection: deny set only on that profile was dropped. NousResearch#108453 made
_load_prefill_messages resolve against the active home, but the loader
only ever ran at boot, outside any profile scope.

Read both where the turn runs, inside the routed profile's
_profile_runtime_scope, the way reasoning and service tier are already
resolved per turn and the ephemeral prompt was fixed in NousResearch#89161:
_build_fresh_agent loads the prefill messages, and the main turn and
/background load provider_routing. The boot copies are removed so nothing
can hand the launch profile's values to another profile again. Cron
already reads both per job, and the TUI/Desktop backend per agent build.

A torn config.yaml write does not drop routing on a per-turn read:
load_user_config_effective serves the last good parse.
@alt-glitch alt-glitch added type/bug Something isn't working P2 Medium — degraded but workaround exists comp/gateway Gateway runner, session dispatch, delivery area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages labels Sep 27, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/config Config system, migrations, profiles area/profiles Multi-profile isolation, HERMES_HOME scoping comp/gateway Gateway runner, session dispatch, delivery P2 Medium — degraded but workaround exists sweeper:risk-message-delivery Sweeper risk: may drop, duplicate, misroute, or suppress messages type/bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants