Image prompt template does not respect template_format and assumes f-string

[yubink]

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate

prompt = {
    'type': 'image_url',
    'image_url': {
        'url': 'data:{{ image_type }};base64,{{ image_data }}',
        'detail': 'low',
    },
}

data =  {
    'image_type': 'image/png',
    'image_data': 'dummyimageabcd12345'
}

template = ChatPromptTemplate.from_messages([('user', [prompt])], template_format='jinja2')
print(template.invoke(data))

Error Message and Stack Trace (if applicable)

No response

Description

The above code outputs:

messages=[HumanMessage(content=[{'type': 'image_url', 'image_url': {'url': 'data:{ image_type };base64, { image_data }', 'detail': 'low'}}])]

Rather than the expected output:

messages=[HumanMessage(content=[{'type': 'image_url', 'image_url': {'url': 'data:image/png;base64, dummyimageabcd12345', 'detail': 'low'}}])]

It looks like the template_format parameter is not being respected here ("f-string" is hard-coded):

langchain/libs/core/langchain_core/prompts/chat.py

Line 556 in 0ebddab

img_template[key], "f-string"

Also, ImagePromptTemplate seems to be using string.format():

langchain/libs/core/langchain_core/prompts/image.py

Line 88 in 0ebddab

formatted[k] = v.format(**kwargs)

System Info

System Information

OS: Linux
OS Version: #17~22.04.1-Ubuntu SMP Tue Sep 3 16:11:52 UTC 2024
Python Version: 3.10.12 (main, Sep 11 2024, 15:47:36) [GCC 11.4.0]

Package Information

langchain_core: 0.3.11
langchain: 0.3.3
langchain_community: 0.3.2
langsmith: 0.1.135
langchain_anthropic: 0.2.3
langchain_google_genai: 2.0.1
langchain_google_vertexai: 2.0.5
langchain_openai: 0.2.2
langchain_text_splitters: 0.3.0

Optional packages not installed

langgraph
langserve

Other Dependencies

aiohttp: 3.10.10
anthropic: 0.32.0
anthropic[vertexai]: Installed. No version info available.
async-timeout: 4.0.3
dataclasses-json: 0.6.7
defusedxml: 0.7.1
google-cloud-aiplatform: 1.70.0
google-cloud-storage: 2.18.0
google-generativeai: 0.8.3
httpx: 0.27.2
httpx-sse: 0.4.0
jsonpatch: 1.33
langchain-mistralai: Installed. No version info available.
numpy: 1.26.4
openai: 1.50.1
orjson: 3.10.7
packaging: 24.1
pillow: 10.3.0
pydantic: 2.9.2
pydantic-settings: 2.5.2
PyYAML: 6.0.2
requests: 2.32.3
requests-toolbelt: 1.0.0
SQLAlchemy: 2.0.36
tenacity: 8.5.0
tiktoken: 0.7.0
typing-extensions: 4.12.2

[chunkanglu]

@yubink Simply removing a set of quotes and the spaces around the variables in the prompt seems to give your desired results:

from langchain_core.prompts import ChatPromptTemplate, HumanMessagePromptTemplate

prompt = {
    'type': 'image_url',
    'image_url': {
        'url': 'data:{image_type};base64, {image_data}',
        'detail': 'low',
    },
}

data =  {
    'image_type': 'image/png',
    'image_data': 'dummyimageabcd12345'
}

template = ChatPromptTemplate.from_messages([('user', [prompt])], template_format='jinja2')
print(template.invoke(data))

[yubink]

That is another way of demonstrating the problem. By removing the braces and spaces, you turned the prompt template into f-string format (rather than jinja2), but despite specifying template_format='jinja2', the template is parsed as f-strings. This behavior is incorrect.

[chunkanglu]

I see, I might look into introducing a fix.

[yubink]

That'd be awesome!

[yubink]

Thank you for the fix!