multi modal input

[pgrayy]

Description

Support users passing in multi-modal input to the agent. As an example:

content = [
    {"text": "what is in this image"},
    {
        "image": {
            "format": "png",
            "source": {
                "bytes": "<IMG_BYTES>",
            },
        },
    },
]
agent(content)

As a follow up, will look into supporting multi-modal inputs for the structured_output call.

Related Issues

#212

Documentation PR

Will follow up on documentation

Type of Change

• Bug fix
• New feature
• Breaking change
• Documentation update
• Other (please describe):

Testing

How have you tested the change? Verify that the changes do not break functionality or introduce warnings in consuming repositories: agents-docs, agents-tools, agents-cli

• I ran hatch run prepare
• Wrote new unit and integ tests

Checklist

• I have read the CONTRIBUTING document
• I have added any necessary tests that prove my fix is effective or my feature works
• I have updated the documentation accordingly
• I have added an appropriate example to the documentation to outline the feature, or no new docs are needed
• My changes generate no new warnings
• Any dependent changes have been merged and published

---

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.