Skip to content

fix(model_metadata): respect context_length from custom_providers config #7998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
fancydirty wants to merge 7 commits into from
Closed

fix(model_metadata): respect context_length from custom_providers config #7998

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
aa87231
fix(gateway): trigger memory provider shutdown on /new and /reset
fancydirty Apr 11, 2026
950c0a8
feat(gateway): native vision passthrough for supported models
fancydirty Apr 11, 2026
ba15b5e
fix(gateway): increase Telegram media group wait to prevent album spl…
fancydirty Apr 11, 2026
20a109a
fix(gateway): increase media batch delay defaults for Telegram and Fe…
fancydirty Apr 11, 2026
3462e6d
Merge branch 'main' into feature/gateway-native-vision-passthrough
fancydirty Apr 11, 2026
e1817ef
fix(delegation): merge disk+runtime config in _load_config; add tests
fancydirty Apr 11, 2026
8a0015a
fix(model_metadata): read context_length from custom_providers config
fancydirty Apr 11, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 16 additions & 0 deletions agent/model_metadata.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -934,6 +934,22 @@ def get_model_context_length(
if config_context_length is not None and isinstance(config_context_length, int) and config_context_length > 0:
return config_context_length

# 0a. Config-driven overrides from custom_providers[].models[].context_length
try:
from hermes_cli.config import load_config
_cfg = load_config()
_custom_providers = _cfg.get("custom_providers", []) if isinstance(_cfg, dict) else []
for cp in _custom_providers:
if not isinstance(cp, dict):
continue
_models = cp.get("models", {})
if isinstance(_models, dict):
_ctx = _models.get(model, {}).get("context_length")
if isinstance(_ctx, int) and _ctx > 0:
return _ctx
Comment on lines +940 to +949

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The custom_providers per-model context_length override is applied without checking which custom provider is active. Because this ignores the provider and base_url arguments, a matching model name in any custom_providers entry can override context length even for unrelated providers/endpoints (and ordering would decide which wins). Consider scoping the lookup to the relevant custom_providers entry (match on provider name/slug and/or base_url) before returning the override.

Suggested change
_cfg = load_config()
_custom_providers = _cfg.get("custom_providers", []) if isinstance(_cfg, dict) else []
for cp in _custom_providers:
if not isinstance(cp, dict):
continue
_models = cp.get("models", {})
if isinstance(_models, dict):
_ctx = _models.get(model, {}).get("context_length")
if isinstance(_ctx, int) and _ctx > 0:
return _ctx
def _normalize_provider_value(value: Any) -> Optional[str]:
if not isinstance(value, str):
return None
value = value.strip().lower()
return value or None
def _normalize_base_url_value(value: Any) -> Optional[str]:
if not isinstance(value, str):
return None
value = value.strip()
if not value:
return None
parsed = urlparse(value)
scheme = parsed.scheme.lower()
netloc = parsed.netloc.lower()
path = parsed.path.rstrip("/")
if scheme or netloc:
return f"{scheme}://{netloc}{path}"
return value.rstrip("/").lower()
_cfg = load_config()
_custom_providers = _cfg.get("custom_providers", []) if isinstance(_cfg, dict) else []
_active_provider = _normalize_provider_value(provider)
_active_base_url = _normalize_base_url_value(base_url)
_valid_custom_provider_count = sum(1 for cp in _custom_providers if isinstance(cp, dict))
for cp in _custom_providers:
if not isinstance(cp, dict):
continue
_provider_keys = ("provider", "slug", "name", "id")
_base_url_keys = ("base_url", "api_base", "endpoint", "url")
_cp_provider_values = {
v for v in (_normalize_provider_value(cp.get(key)) for key in _provider_keys) if v
}
_cp_base_urls = {
v for v in (_normalize_base_url_value(cp.get(key)) for key in _base_url_keys) if v
}
_has_scope_metadata = bool(_cp_provider_values or _cp_base_urls)
if _active_provider or _active_base_url:
_provider_matches = bool(
_active_provider and _cp_provider_values and _active_provider in _cp_provider_values
)
_base_url_matches = bool(
_active_base_url and _cp_base_urls and _active_base_url in _cp_base_urls
)
if _has_scope_metadata:
_matched_available_scope = False
if _active_provider and _cp_provider_values:
_matched_available_scope = _matched_available_scope or _provider_matches
if _active_base_url and _cp_base_urls:
_matched_available_scope = _matched_available_scope or _base_url_matches
if not _matched_available_scope:
continue
elif _valid_custom_provider_count != 1:
# Unscoped entries are ambiguous when multiple custom providers exist.
continue
_models = cp.get("models", {})
if isinstance(_models, dict):
_model_entry = _models.get(model, {})
if isinstance(_model_entry, dict):
_ctx = _model_entry.get("context_length")
if isinstance(_ctx, int) and _ctx > 0:
return _ctx

Copilot uses AI. Check for mistakes.
Comment on lines +937 to +949

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This new config-driven custom_providers[].models[].context_length path isn’t covered by tests in tests/agent/test_model_metadata.py. Adding a unit test that sets load_config() to return a custom_providers entry with a per-model context_length (and verifies correct scoping to the selected provider/base_url) would prevent regressions.

Suggested change
# 0a. Config-driven overrides from custom_providers[].models[].context_length
try:
from hermes_cli.config import load_config
_cfg = load_config()
_custom_providers = _cfg.get("custom_providers", []) if isinstance(_cfg, dict) else []
for cp in _custom_providers:
if not isinstance(cp, dict):
continue
_models = cp.get("models", {})
if isinstance(_models, dict):
_ctx = _models.get(model, {}).get("context_length")
if isinstance(_ctx, int) and _ctx > 0:
return _ctx
# 0a. Config-driven overrides from custom_providers[].models[].context_length.
# Scope these overrides to the selected custom provider when base_url is
# known, so identical model IDs on different endpoints do not leak across
# providers. If no base_url is provided, preserve the historical behavior
# of allowing any matching custom provider model override.
try:
from hermes_cli.config import load_config
_cfg = load_config()
_custom_providers = _cfg.get("custom_providers", []) if isinstance(_cfg, dict) else []
_normalized_base_url = base_url.rstrip("/") if isinstance(base_url, str) and base_url else None
for cp in _custom_providers:
if not isinstance(cp, dict):
continue
if _normalized_base_url is not None:
_provider_url = cp.get("base_url") or cp.get("url") or cp.get("endpoint")
if not isinstance(_provider_url, str):
continue
if _provider_url.rstrip("/") != _normalized_base_url:
continue
_models = cp.get("models", {})
if isinstance(_models, dict):
_model_cfg = _models.get(model, {})
if isinstance(_model_cfg, dict):
_ctx = _model_cfg.get("context_length")
if isinstance(_ctx, int) and _ctx > 0:
return _ctx

Copilot uses AI. Check for mistakes.
except Exception:
pass

# Normalise provider-prefixed model names (e.g. "local:model-name" →
# "model-name") so cache lookups and server queries use the bare ID that
# local servers actually know about. Ollama "model:tag" colons are preserved.
Expand Down
2 changes: 1 addition & 1 deletion gateway/platforms/feishu.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@
_DEFAULT_TEXT_BATCH_DELAY_SECONDS = 0.6
_DEFAULT_TEXT_BATCH_MAX_MESSAGES = 8
_DEFAULT_TEXT_BATCH_MAX_CHARS = 4000
_DEFAULT_MEDIA_BATCH_DELAY_SECONDS = 0.8
_DEFAULT_MEDIA_BATCH_DELAY_SECONDS = 3.0
_DEFAULT_DEDUP_CACHE_SIZE = 2048
_DEFAULT_WEBHOOK_HOST = "127.0.0.1"
_DEFAULT_WEBHOOK_PORT = 8765
Expand Down
10 changes: 8 additions & 2 deletions gateway/platforms/telegram.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -125,7 +125,11 @@ class TelegramAdapter(BasePlatformAdapter):
# Threshold for detecting Telegram client-side message splits.
# When a chunk is near this limit, a continuation is almost certain.
_SPLIT_THRESHOLD = 4000
MEDIA_GROUP_WAIT_SECONDS = 0.8
# Albums can be split across getUpdates calls; 0.8s is too short and causes
# media groups to be flushed as separate messages (hallucination).
MEDIA_GROUP_WAIT_SECONDS = float(
os.getenv("HERMES_TELEGRAM_MEDIA_GROUP_WAIT_SECONDS", "3.0")
)

def __init__(self, config: PlatformConfig):
super().__init__(config, Platform.TELEGRAM)
Expand All @@ -136,7 +140,9 @@ def __init__(self, config: PlatformConfig):
self._reply_to_mode: str = getattr(config, 'reply_to_mode', 'first') or 'first'
# Buffer rapid/album photo updates so Telegram image bursts are handled
# as a single MessageEvent instead of self-interrupting multiple turns.
self._media_batch_delay_seconds = float(os.getenv("HERMES_TELEGRAM_MEDIA_BATCH_DELAY_SECONDS", "0.8"))
# 3.0s default prevents albums split across getUpdates calls from flushing
# as separate hallucinated messages.
self._media_batch_delay_seconds = float(os.getenv("HERMES_TELEGRAM_MEDIA_BATCH_DELAY_SECONDS", "3.0"))
self._pending_photo_batches: Dict[str, MessageEvent] = {}
self._pending_photo_batch_tasks: Dict[str, asyncio.Task] = {}
self._media_group_events: Dict[str, MessageEvent] = {}
Expand Down
85 changes: 75 additions & 10 deletions gateway/run.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@
import time
from pathlib import Path
from datetime import datetime
from typing import Dict, Optional, Any, List
from typing import Dict, Optional, Any, List, Tuple

# ---------------------------------------------------------------------------
# SSL certificate auto-detection for NixOS and other non-standard systems.
Expand Down Expand Up @@ -2790,15 +2790,21 @@ async def _prepare_inbound_message_text(
event: MessageEvent,
source: SessionSource,
history: List[Dict[str, Any]],
) -> Optional[str]:
) -> Tuple[Optional[str], Optional[List[Dict[str, Any]]]]:
"""Prepare inbound event text for the agent.

Keep the normal inbound path and the queued follow-up path on the same
preprocessing pipeline so sender attribution, image enrichment, STT,
document notes, reply context, and @ references all behave the same.

Returns a tuple of (message_text, message_content). message_content is
a list of content parts for native vision passthrough when the active
model supports it; otherwise it is None and message_text contains any
vision enrichments.
"""
history = history or []
message_text = event.text or ""
message_content: Optional[List[Dict[str, Any]]] = None

_is_shared_thread = (
source.chat_type != "dm"
Expand All @@ -2819,10 +2825,55 @@ async def _prepare_inbound_message_text(
audio_paths.append(path)

if image_paths:
message_text = await self._enrich_message_with_vision(
message_text,
image_paths,
)
# Decide whether to passthrough images natively or pre-describe them
try:
from run_agent import AIAgent
_gw_cfg = _load_gateway_config()
_model, _runtime = self._resolve_session_agent_runtime(
source=source, user_config=_gw_cfg
Comment on lines +2829 to +2833

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This async function calls AIAgent._check_native_vision_support() inline to decide passthrough. That helper can do blocking I/O (OpenRouter /api/v1/models via requests, models.dev fetch, and config.yaml disk reads), which would block the event loop when images arrive. Consider ensuring this detection path is non-blocking (e.g., only use cached/local heuristics here, or run the blocking capability lookup in an executor and cache the result).

Copilot uses AI. Check for mistakes.
)
_provider = _runtime.get("provider") or (""
if "/" not in _model else _model.split("/", 1)[0])
_api_mode = _runtime.get("api_mode") or "chat_completions"
_supports_native = AIAgent._check_native_vision_support(
_model, _provider, _api_mode
)
except Exception:
_supports_native = False

if _supports_native:
_parts = []
if message_text:
_parts.append({"type": "text", "text": message_text})
for _img_path in image_paths:
try:
import base64
from pathlib import Path
_img_data = Path(_img_path).read_bytes()
_b64 = base64.b64encode(_img_data).decode("ascii")
_suffix = Path(_img_path).suffix.lower()
_mime = {
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
".bmp": "image/bmp",
}.get(_suffix, "image/jpeg")
_parts.append({
"type": "image_url",
"image_url": {"url": f"data:{_mime};base64,{_b64}"},
})
Comment on lines +2845 to +2864

Copilot AI Apr 11, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When native passthrough is enabled, images are read and base64-encoded synchronously (Path.read_bytes + base64.b64encode) inside the async event loop. For large images or bursts this can noticeably block the loop. Consider offloading the file read/encode to a thread (asyncio.to_thread) or using async I/O, and enforcing/validating a max size to avoid excessive memory/latency.

Suggested change
_parts = []
if message_text:
_parts.append({"type": "text", "text": message_text})
for _img_path in image_paths:
try:
import base64
from pathlib import Path
_img_data = Path(_img_path).read_bytes()
_b64 = base64.b64encode(_img_data).decode("ascii")
_suffix = Path(_img_path).suffix.lower()
_mime = {
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
".bmp": "image/bmp",
}.get(_suffix, "image/jpeg")
_parts.append({
"type": "image_url",
"image_url": {"url": f"data:{_mime};base64,{_b64}"},
})
_max_native_image_bytes = 10 * 1024 * 1024
async def _encode_image_for_native_vision(_img_path: str) -> Dict[str, Any]:
def _read_and_encode_image() -> Dict[str, Any]:
import base64
_path = Path(_img_path)
_size = _path.stat().st_size
if _size > _max_native_image_bytes:
raise ValueError(
f"image size {_size} exceeds native passthrough limit "
f"of {_max_native_image_bytes} bytes"
)
_img_data = _path.read_bytes()
_b64 = base64.b64encode(_img_data).decode("ascii")
_suffix = _path.suffix.lower()
_mime = {
".png": "image/png",
".gif": "image/gif",
".webp": "image/webp",
".bmp": "image/bmp",
}.get(_suffix, "image/jpeg")
return {
"type": "image_url",
"image_url": {"url": f"data:{_mime};base64,{_b64}"},
}
return await asyncio.to_thread(_read_and_encode_image)
_parts = []
if message_text:
_parts.append({"type": "text", "text": message_text})
for _img_path in image_paths:
try:
_parts.append(await _encode_image_for_native_vision(_img_path))

Copilot uses AI. Check for mistakes.
except Exception as _img_err:
logger.warning("Failed to encode image for native vision: %s", _img_err)
if len(_parts) > 1 or (_parts and _parts[0].get("type") == "image_url"):
message_content = _parts
else:
message_text = await self._enrich_message_with_vision(
message_text, image_paths
)
else:
message_text = await self._enrich_message_with_vision(
message_text, image_paths
)

if audio_paths:
message_text = await self._enrich_message_with_transcription(
Expand Down Expand Up @@ -2938,7 +2989,7 @@ async def _prepare_inbound_message_text(
except Exception as exc:
logger.debug("@ context reference expansion failed: %s", exc)

return message_text
return message_text, message_content

async def _handle_message_with_agent(self, event, source, _quick_key: str):
"""Inner handler that runs under the _running_agents sentinel guard."""
Expand Down Expand Up @@ -3380,7 +3431,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str):
# attachments (documents, audio, etc.) are not sent to the vision
# tool even when they appear in the same message.
# -----------------------------------------------------------------
message_text = await self._prepare_inbound_message_text(
message_text, message_content = await self._prepare_inbound_message_text(
event=event,
source=source,
history=history,
Expand All @@ -3407,6 +3458,7 @@ async def _handle_message_with_agent(self, event, source, _quick_key: str):
session_id=session_entry.session_id,
session_key=session_key,
event_message_id=event.message_id,
message_content=message_content,
)

# Stop persistent typing indicator now that the agent is done
Expand Down Expand Up @@ -3796,6 +3848,11 @@ async def _handle_reset_command(self, event: MessageEvent) -> str:
_cached = self._agent_cache.get(session_key)
_old_agent = _cached[0] if isinstance(_cached, tuple) else _cached if _cached else None
if _old_agent is not None:
try:
if hasattr(_old_agent, "shutdown_memory_provider"):
_old_agent.shutdown_memory_provider()
except Exception:
pass
try:
if hasattr(_old_agent, "close"):
_old_agent.close()
Expand Down Expand Up @@ -7056,6 +7113,7 @@ async def _run_agent(
session_key: str = None,
_interrupt_depth: int = 0,
event_message_id: Optional[str] = None,
message_content: Optional[List[Dict[str, Any]]] = None,
) -> Dict[str, Any]:
"""
Run the agent with the given message and context.
Expand Down Expand Up @@ -7685,7 +7743,12 @@ def _approval_notify_sync(approval_data: dict) -> None:
_approval_session_token = set_current_session_key(_approval_session_key)
register_gateway_notify(_approval_session_key, _approval_notify_sync)
try:
result = agent.run_conversation(message, conversation_history=agent_history, task_id=session_id)
result = agent.run_conversation(
message,
conversation_history=agent_history,
task_id=session_id,
user_message_content=message_content,
)
finally:
unregister_gateway_notify(_approval_session_key)
reset_current_session_key(_approval_session_token)
Expand Down Expand Up @@ -8148,10 +8211,11 @@ async def _notify_long_running():
updated_history = result.get("messages", history)
next_source = source
next_message = pending
next_message_content = None
next_message_id = None
if pending_event is not None:
next_source = getattr(pending_event, "source", None) or source
next_message = await self._prepare_inbound_message_text(
next_message, next_message_content = await self._prepare_inbound_message_text(
event=pending_event,
source=next_source,
history=updated_history,
Expand All @@ -8169,6 +8233,7 @@ async def _notify_long_running():
session_key=session_key,
_interrupt_depth=_interrupt_depth + 1,
event_message_id=next_message_id,
message_content=next_message_content,
)
finally:
# Stop progress sender, interrupt monitor, and notification task
Expand Down
7 changes: 6 additions & 1 deletion hermes_cli/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -287,6 +287,11 @@ def _ensure_hermes_home_managed(home: Path):
# threshold before escalating to a full timeout. The warning fires
# once per run and does not interrupt the agent. 0 = disable warning.
"gateway_timeout_warning": 900,
# User overrides for native vision passthrough. Each entry is a model
# ID substring matched case-insensitively. These take precedence over
# automatic detection (OpenRouter API, models.dev) but not the
# VISION_NATIVE_PASSTHROUGH env variable.
"vision_native_models": [],
},

"terminal": {
Expand Down Expand Up @@ -638,7 +643,7 @@ def _ensure_hermes_home_managed(home: Path):
},

# Config schema version - bump this when adding new required fields
"_config_version": 14,
"_config_version": 15,
}

# =============================================================================
Expand Down
Loading
Loading