Fix: Make litellm import lazy in resolve_model_config.py

[juanmichelini]

Summary

This PR fixes the Integration Test Action that has been failing continuously by making the litellm import lazy in resolve_model_config.py.

Problem:
The setup-matrix job in the integration-runner.yml workflow was failing with ModuleNotFoundError: No module named 'litellm'. The workflow imports the MODELS dictionary from resolve_model_config.py without installing litellm, but the script had import litellm at the module level.

Solution:

• Moved the import litellm statement from the module level into the test_model() function where it's actually used
• Added proper type checking imports (from __future__ import annotations and TYPE_CHECKING)
• Updated all tests to patch litellm.completion directly instead of resolve_model_config.litellm.completion
• Added a regression test (test_models_importable_without_litellm) that verifies MODELS can be imported without litellm installed

This fix allows the workflow to import the MODELS dictionary during the setup-matrix job without requiring litellm to be installed, while still preserving the preflight check functionality that uses litellm.

Fixes #2124

Checklist

• If the PR is changing/adding functionality, are there tests to reflect this? - Added test_models_importable_without_litellm regression test
• If there is an example, have you run the example to make sure that it works? - N/A (no examples changed)
• If there are instructions on how to run the code, have you followed the instructions and made sure that it works? - Verified the inline script works correctly
• If the feature is significant enough to require documentation, is there a PR open on the OpenHands/docs repository with the same branch name? - N/A (this is a bug fix, not a new feature)
• Is the github CI passing? - Will be verified after PR creation

@juanmichelini can click here to continue refining the PR

---

Agent Server images for this PR

• GHCR package: https://github.com/OpenHands/agent-sdk/pkgs/container/agent-server

Variants & Base Images

Variant	Architectures	Base Image	Docs / Tags
java	amd64, arm64	eclipse-temurin:17-jdk	Link
python	amd64, arm64	nikolaik/python-nodejs:python3.12-nodejs22	Link
golang	amd64, arm64	golang:1.21-bookworm	Link

Pull (multi-arch manifest)

# Each variant is a multi-arch manifest supporting both amd64 and arm64
docker pull ghcr.io/openhands/agent-server:df088e2-python

Run

docker run -it --rm \
  -p 8000:8000 \
  --name agent-server-df088e2-python \
  ghcr.io/openhands/agent-server:df088e2-python

All tags pushed for this build

ghcr.io/openhands/agent-server:df088e2-golang-amd64
ghcr.io/openhands/agent-server:df088e2-golang_tag_1.21-bookworm-amd64
ghcr.io/openhands/agent-server:df088e2-golang-arm64
ghcr.io/openhands/agent-server:df088e2-golang_tag_1.21-bookworm-arm64
ghcr.io/openhands/agent-server:df088e2-java-amd64
ghcr.io/openhands/agent-server:df088e2-eclipse-temurin_tag_17-jdk-amd64
ghcr.io/openhands/agent-server:df088e2-java-arm64
ghcr.io/openhands/agent-server:df088e2-eclipse-temurin_tag_17-jdk-arm64
ghcr.io/openhands/agent-server:df088e2-python-amd64
ghcr.io/openhands/agent-server:df088e2-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-amd64
ghcr.io/openhands/agent-server:df088e2-python-arm64
ghcr.io/openhands/agent-server:df088e2-nikolaik_s_python-nodejs_tag_python3.12-nodejs22-arm64
ghcr.io/openhands/agent-server:df088e2-golang
ghcr.io/openhands/agent-server:df088e2-java
ghcr.io/openhands/agent-server:df088e2-python

About Multi-Architecture Support

• Each variant tag (e.g., df088e2-python) is a multi-arch manifest supporting both amd64 and arm64
• Docker automatically pulls the correct architecture for your platform
• Individual architecture tags (e.g., df088e2-python-amd64) are also available if needed

[juanmichelini]

@OpenHands we only want to do absolutely minimal changes in this PR to pass the test so:

1. remove any change in this PR that does directly help fix the issue
2. run the action using this branch
3. wait for the action to run and see if it actually succeeded
4. update the PR if needed

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Final Summary

I successfully stripped down PR #2125 to the absolute minimum changes needed to fix the litellm import issue:

✅ Changes Made

Removed non-essential changes:

• Removed from __future__ import annotations
• Removed TYPE_CHECKING import and empty if TYPE_CHECKING: block

Kept only essential changes:

1. Lazy import of litellm inside test_model() function (the core fix)
2. Updated test patches to "litellm.completion" (necessary due to lazy import)
3. Regression test test_models_importable_without_litellm() (verifies the fix)

✅ Verification Completed

1. Committed and pushed the minimal changes to openhands/fix-integration-test-litellm-import
2. Manually triggered the Integration Test workflow on this branch
3. Verified success: The critical setup-matrix job completed successfully ✓ (this was the job failing with ModuleNotFoundError: No module named 'litellm')

The PR is now ready for review with only the minimal essential changes. The fix allows the workflow to import the MODELS dictionary without requiring litellm to be installed, while preserving all functionality.

Workflow run: https://github.com/OpenHands/software-agent-sdk/actions/runs/22159883787

[juanmichelini]

@OpenHands does putting an import inside a dictionary make sense? I know you tested it and it works but it looks hacky. and might postpone the error to unpredictable moments. can you check if there is another way? in particular what are we fixing actually? what broke in the last 48 hours that made the action failed continously

[openhands-ai]

I'm on it! juanmichelini can track my progress at all-hands.dev

[openhands-ai]

Summary: Question Answered - No Changes Made

I investigated your concerns about the PR and can confirm that the current fix is actually correct, not hacky.

What I Found

What broke 7 days ago:

• PR Add preflight LLM check before dispatching evaluations #2109 (commit 662db6e) added import litellm at the top of resolve_model_config.py to support preflight checks
• This broke integration-runner.yml which imports the MODELS dictionary without having litellm installed (lines 91-118 of the workflow)

Why the current fix is correct:

• The import is NOT inside a dictionary - it's inside the test_model() function
• This is a standard Python pattern called "lazy importing" for optional dependencies
• Used extensively by Django, Flask, and other major projects
• Allows importing MODELS without litellm, while still requiring litellm when test_model() is actually called

Verification:

• ✅ Successfully imports MODELS without litellm (what integration-runner needs)
• ✅ Correctly fails with clear error when calling test_model() without litellm
• ✅ Regression test passes

Conclusion

No changes needed. The current PR uses the minimal, standard Python approach for this exact scenario. The lazy import pattern is the recommended solution for optional dependencies that aren't always needed at import time.

The PR is ready as-is and should fix the failing integration tests.

[all-hands-bot]

🟢 Good Taste - LGTM

Perfect example of eliminating special cases through lazy loading. The solution is elegant and minimal:

• Defers the litellm dependency to exactly where it's needed
• Solves a real production problem (failing CI) with minimal change
• Proper regression test using subprocess to verify clean import environment
• Test patches correctly updated from resolve_model_config.litellm to litellm module

Verdict: ✅ Worth merging

Key Insight: This is how dependencies should be managed - import only when used, not speculatively at module level.