Support custom OpenAI-compatible HTTP endpoints in LlamaStack config templates

[jangel97]

Hi LlamaStack team,

I'd like to request native support in the stack configuration for using custom OpenAI-compatible HTTP endpoints for both inference and embeddings.

Use Case

I'm integrating with internal APIs that expose OpenAI-style /v1/embeddings and /v1/completions endpoints (e.g., IBM Granite, MiniLM via APIcast). These endpoints require custom model IDs, auth tokens, and base URLs.

I'd like to declare this setup in YAML, something like:

runtime:
  template: remote-httpx
  params:
    model_url: ${LLM_MODEL_URL}
    model_id: ${LLM_MODEL_ID}
    api_key: ${LLM_ACCESS_KEY}
    api_key_header: Authorization
    api_key_prefix: Bearer

embedding:
  template: remote-httpx
  params:
    model_url: ${MINILLM_MODEL_URL}
    model_id: ${MINILLM_MODEL_ID}
    api_key: ${MINILLM_ACCESS_KEY}
    api_key_header: Authorization
    api_key_prefix: Bearer

Why it matters

This would allow declarative use of enterprise models behind auth layers, without needing to override logic in Python. It would also unlock easier use of hosted or private models (Granite, DeepSeek, Ollama, etc.).

Question

Is there a recommended workaround today for supporting these remote providers via config? Or is this currently only possible by overriding via code?

[hardikjshah]

Hey @jangel97 ,

We recently added OpenAI compat embedding APIs to Llama Stack as part of #2314. Could you just add a provider for this implementation and should that fix your issue ?

[jangel97]

hey @hardikjshah that's great!

What about inference api, is there a way to use openai compatible api v1/completions out of the box?

Thank you for your help!

[reluctantfuturist]

hey @hardikjshah that's great!

What about inference api, is there a way to use openai compatible api v1/completions out of the box?

Thank you for your help!

Should be supported since #1894 -- can you please check if this works for your use case, or whether anything else is needed on top of that?

[github-actions]

This issue has been automatically marked as stale because it has not had activity within 60 days. It will be automatically closed if no further activity occurs within 30 days.

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant!