Skip to content

feat: litellm config yaml #369

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
yxtay merged 18 commits into from
Jul 28, 2026
Merged

feat: litellm config yaml #369

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
c36eb1d
feat: litellm-config-yaml
yxtay Jul 27, 2026
2c9da88
Update compose.yaml
yxtay Jul 27, 2026
6d1cc3f
simplify supported
yxtay Jul 27, 2026
fb9379f
Update config.yaml
yxtay Jul 27, 2026
67c22bb
Update config.yaml
yxtay Jul 28, 2026
919bcab
Update config.yaml
yxtay Jul 28, 2026
cd7081a
Update config.yaml
yxtay Jul 28, 2026
9d055b1
Update config.yaml
yxtay Jul 28, 2026
f6b4fa4
Update config.yaml
yxtay Jul 28, 2026
6968903
Update config.yaml
yxtay Jul 28, 2026
bee6f8b
Update config.yaml
yxtay Jul 28, 2026
6fa9a98
Update config.yaml
yxtay Jul 28, 2026
47f7479
Update config.yaml
yxtay Jul 28, 2026
faf1cf4
Update config.yaml
yxtay Jul 28, 2026
9146c60
Update config.yaml
yxtay Jul 28, 2026
dba1cb6
Update config.yaml
yxtay Jul 28, 2026
d251cd0
Update config.yaml
yxtay Jul 28, 2026
ace752c
Update config.yaml
yxtay Jul 28, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion agent/compose.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -113,7 +113,7 @@ services:
- no-new-privileges:true
read_only: true
volumes:
- /apps/litellm/config.yaml:/app/config.yaml:ro
- ./litellm/config.yaml:/app/config.yaml:ro

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Keep the bind-mount source under /apps/litellm/.

./litellm/config.yaml violates the required host bind-mount layout. Provision the repository config to /apps/litellm/config.yaml and retain that absolute source path.

Proposed fix 
-      - ./litellm/config.yaml:/app/config.yaml:ro
+      - /apps/litellm/config.yaml:/app/config.yaml:ro

As per coding guidelines, Docker Compose files must “Use host bind mounts under /apps/<service-name>/ instead of named volumes.”

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
- ./litellm/config.yaml:/app/config.yaml:ro
- /apps/litellm/config.yaml:/app/config.yaml:ro
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agent/compose.yaml` at line 116, Update the bind-mount entry in the Compose
service to use the absolute host source path /apps/litellm/config.yaml while
retaining /app/config.yaml:ro as the container destination and read-only mode.

Source: Coding guidelines

tmpfs:
- /app/cache
- /app/migrations
Expand Down
83 changes: 83 additions & 0 deletions agent/litellm/config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
general_settings:
store_model_in_db: true

litellm_settings:
drop_params: true
request_timeout: 60
json_logs: true

model_list:
- model_name: cerebras/gemma-4-31b
litellm_params:
model: cerebras/gemma-4-31b

- model_name: google/gemini-flash-lite-latest
litellm_params:
model: gemini/gemini-flash-lite-latest

- model_name: google/gemma-4-31b-it
litellm_params:
model: gemini/gemma-4-31b-it

- model_name: groq/gpt-oss-120b
litellm_params:
model: groq/openai/gpt-oss-120b

- model_name: nous/hy3:free
litellm_params:
model: openai/tencent/hy3:free
litellm_credential_name: nous

- model_name: opencode-zen/deepseek-v4-flash-free
litellm_params:
model: openai/deepseek-v4-flash-free
litellm_credential_name: opencode_zen

- model_name: opencode-zen/mimo-v2.5-free
litellm_params:
model: openai/mimo-v2.5-free
litellm_credential_name: opencode_zen

- model_name: openrouter/free
litellm_params:
model: openrouter/openrouter/free

- model_name: smart-router
litellm_params:
model: auto_router/complexity_router
complexity_router_config:
tiers:
SIMPLE: opencode-zen/deepseek-v4-flash-free
MEDIUM: opencode-zen/deepseek-v4-flash-free
COMPLEX: nous/hy3:free
REASONING: nous/hy3:free
adaptive: true
session_affinity: true
complexity_router_default_model: nous/hy3:free
Comment on lines +46 to +57

@coderabbitai coderabbitai Bot Jul 28, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

rg -n -C2 'NOUS_API_KEY|OPENCODE_ZEN_API_KEY' \
  agent/.env.example agent/compose.yaml agent/litellm/config.yaml

Repository: yxtay/docker-stacks

Length of output: 993

🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== files =="
git ls-files agent | sed -n '1,200p'

echo "== relevant env/compose/config excerpts =="
for f in agent/.env.example agent/compose.yaml agent/litellm/config.yaml agent/Dockerfile* Dockerfile* compose.yaml; do
  if [ -f "$f" ]; then
    echo "--- $f"
    wc -l "$f"
    sed -n '1,140p' "$f"
  fi
done

Repository: yxtay/docker-stacks

Length of output: 6835

🌐 Web query:

LiteLLM config.yaml model custom_base_url os.environ API_KEY credential_values api_key os.environ/NOUS_API_KEY

💡 Result:

In LiteLLM, you can use the os.environ/ prefix within your config.yaml to dynamically load values from environment variables [1][2][3]. This syntax is supported for various configuration fields, ensuring sensitive credentials like API keys are not hardcoded in your configuration files [4][5]. Key mechanisms for using environment variables and custom configurations include: 1. Dynamic Loading with os.environ/ You can reference any environment variable by using the prefix os.environ/ followed by the variable name [3]. When LiteLLM parses the config.yaml, it will call os.getenv() to retrieve the value [2]. Example usage in model_list: model_list: - model_name: my-model litellm_params: model: openai/gpt-4 api_key: os.environ/NOUS_API_KEY # Loads from environment variable NOUS_API_KEY api_base: os.environ/CUSTOM_API_BASE_URL # Example of using it for custom base URL 2. Centralized Credential Management To avoid duplication, you can define shared credentials using the credential_list section and reference them in your model_list using litellm_credential_name [6][7]. Example: model_list: - model_name: my-azure-model litellm_params: model: azure/gpt-4 litellm_credential_name: my_shared_creds credential_list: - credential_name: my_shared_creds credential_values: api_key: os.environ/AZURE_API_KEY api_base: os.environ/AZURE_API_BASE credential_info: description: "Shared credentials" 3. Custom API Base URL The api_base parameter within litellm_params allows you to point to non-default endpoints, such as local gateways (e.g., vLLM) or specific provider regions [8][2]. This field also supports the os.environ/ syntax, enabling flexible configuration across different environments [5]. 4. Important Note on Server Root Path If your goal is to change the base URL path of the LiteLLM Proxy service itself (e.g., serving at /api/v1 instead of the root), you should set the SERVER_ROOT_PATH environment variable in your environment, rather than modifying config.yaml [9]. Never commit your config.yaml to version control if it contains plain-text secrets [4][5]. Always prefer the os.environ/ method to keep credentials secure [4].

Citations:

Declare and fail fast on the smart-router credentials.

smart-router routes to nous/hy3:free for COMPLEX, REASONING, and fallback requests, and litellm_credential_name: nous resolves its key from NOUS_API_KEY. Add NOUS_API_KEY= to agent/.env.example and require both NOUS_API_KEY and OPENCODE_ZEN_API_KEY in the LiteLLM litellm service with ${VAR:?} so an unset credential fails startup rather than causing request failure.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@agent/litellm/config.yaml` around lines 46 - 57, Declare NOUS_API_KEY= in
agent/.env.example, then update the LiteLLM service environment configuration to
require both NOUS_API_KEY and OPENCODE_ZEN_API_KEY using the ${VAR:?} fail-fast
syntax. Ensure the smart-router credentials resolve through the existing litellm
credential configuration without changing routing behavior.

Source: Coding guidelines


credential_list:
- credential_name: nous
credential_values:
api_key: os.environ/NOUS_API_KEY
api_base: https://inference-api.nousresearch.com/v1
credential_info: {}

- credential_name: opencode_zen
credential_values:
api_key: os.environ/OPENCODE_ZEN_API_KEY
api_base: https://opencode.ai/zen/v1
credential_info: {}

- credential_name: opencode_go
credential_values:
api_key: os.environ/OPENCODE_GO_API_KEY
api_base: https://opencode.ai/zen/go/v1
credential_info: {}

# guardrails:
# - guardrail_name: headroom
# litellm_params:
# guardrail: headroom
# mode: pre_call
# api_base: http://headroom:8787
# default_on: true
Loading