Skip to content

fix(litellm): use context limit from /model/info for custom models #9303

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jamadeo merged 1 commit into from
May 22, 2026
Merged

fix(litellm): use context limit from /model/info for custom models #9303

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 30 additions & 7 deletions crates/goose/src/providers/litellm.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ pub struct LiteLLMProvider {
model: ModelConfig,
#[serde(skip)]
name: String,
#[serde(skip)]
cached_model_info: tokio::sync::OnceCell<Vec<ModelInfo>>,
}

impl LiteLLMProvider {
Expand Down Expand Up @@ -77,10 +79,18 @@ impl LiteLLMProvider {
base_path,
model,
name: LITELLM_PROVIDER_NAME.to_string(),
cached_model_info: tokio::sync::OnceCell::new(),
})
}

async fn fetch_models(&self) -> Result<Vec<ModelInfo>, ProviderError> {
async fn get_or_fetch_models(&self) -> Result<&[ModelInfo], ProviderError> {
self.cached_model_info
.get_or_try_init(|| self.fetch_models_from_api())
.await
.map(|v| v.as_slice())
}

async fn fetch_models_from_api(&self) -> Result<Vec<ModelInfo>, ProviderError> {
let response = self
.api_client
.request(None, "model/info")
Expand Down Expand Up @@ -184,7 +194,22 @@ impl Provider for LiteLLMProvider {
}

fn get_model_config(&self) -> ModelConfig {
self.model.clone()
let mut config = self.model.clone();
// The cache is populated lazily by the first stream() call (via
// supports_cache_control). On turn 1 this will be None and we fall
// back to DEFAULT_CONTEXT_LIMIT, which is fine — the conversation is
// too small to trigger compaction. From turn 2 onward the real limit
// from /model/info is used.
if config.context_limit.is_none() {
if let Some(models) = self.cached_model_info.get() {
if let Some(info) = models.iter().find(|m| m.name == config.model_name) {
Comment on lines +203 to +205

@chatgpt-codex-connector chatgpt-codex-connector Bot May 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Trigger /model/info lookup when context_limit is missing

get_model_config() now only reads cached_model_info and never initiates a fetch when context_limit is None, so on a fresh provider instance the first request still uses the 128k default until some other path warms the cache. In the normal chat flow, provider.get_model_config() is consumed before supports_cache_control() performs any fetch, so custom LiteLLM models with smaller windows can still hit context-overflow errors on initial turns instead of compacting to the real limit.

Useful? React with 👍 / 👎.

if info.context_limit > 0 {
config.context_limit = Some(info.context_limit);
}
}
Comment on lines +205 to +209

@chatgpt-codex-connector chatgpt-codex-connector Bot May 18, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Skip non-positive context limits from model metadata

This assignment accepts info.context_limit verbatim, including 0, which can happen when upstream metadata is unset/sentinel-valued; once stored, downstream context_limit() calls treat it as authoritative and can break compaction math or force pathological behavior. The provider should ignore non-positive limits here (consistent with other model-limit normalization paths) and keep the fallback/default instead.

Useful? React with 👍 / 👎.

}
}
config
}

async fn stream(
Expand Down Expand Up @@ -237,7 +262,7 @@ impl Provider for LiteLLMProvider {
}

async fn supports_cache_control(&self) -> bool {
if let Ok(models) = self.fetch_models().await {
if let Ok(models) = self.get_or_fetch_models().await {
if let Some(model_info) = models.iter().find(|m| m.name == self.model.model_name) {
return model_info.supports_cache_control.unwrap_or(false);
}
Expand All @@ -247,10 +272,8 @@ impl Provider for LiteLLMProvider {
}

async fn fetch_supported_models(&self) -> Result<Vec<String>, ProviderError> {
let models = self.fetch_models().await.map_err(|e| {
ProviderError::RequestFailed(format!("Failed to fetch models from LiteLLM: {}", e))
})?;
Ok(models.into_iter().map(|m| m.name).collect())
let models = self.get_or_fetch_models().await?;
Ok(models.iter().map(|m| m.name.clone()).collect())
}
}

Expand Down
Loading