Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes image type, adds document type for ChatBedrockConverse #143

Merged
merged 4 commits into from
Aug 20, 2024

Conversation

3coins
Copy link
Collaborator

@3coins 3coins commented Aug 6, 2024

Description

This PR fixes the image type supported by Bedrock Anthropic and other models. It also adds a new type for supporting documents.

Image sample code

from langchain_core.messages import HumanMessage
from langchain_aws import ChatBedrock

llm = ChatBedrock(
    model_id="anthropic.claude-3-sonnet-20240229-v1:0",
    beta_use_converse_api=True
)

# Convert image to bytes
image_path = "images/random-image.jpg"
image_bytes = None
with open(image_path, "rb") as f:
    image_bytes = f.read()

messages = [
    HumanMessage(
        content=[
            {
                "type": "image",
                "image": {
                    "format": "jpeg",
                    "source": {
                        "bytes": image_bytes
                    }
                }
            },
            "What is this image about?"
        ]
    )
]

llm.invoke(messages)

Document sample code

from langchain_core.messages import HumanMessage
from langchain_aws import ChatBedrock

llm = ChatBedrock(
    model_id="anthropic.claude-3-sonnet-20240229-v1:0",
    beta_use_converse_api=True
)

# Convert document to bytes
doc_path = "documents/random-doc.pdf"
doc_bytes = None
with open(doc_path, "rb") as f:
    doc_bytes = f.read()

messages = [
    HumanMessage(
        content=[
            {
                "type": "document",
                "document": {
                    "format": "pdf",
                    "name": "random-doc",
                    "source": {
                        "bytes": doc_bytes
                    }
                }
            },
            "What is this document about?"
        ]
    )
]

llm.invoke(messages)

Related to #75
Fixes #132

@3coins 3coins requested a review from baskaryan August 8, 2024 17:33
@@ -659,7 +659,7 @@ def _anthropic_to_bedrock(
elif block["type"] == "image":
# Assume block is already in bedrock format.
if "image" in block:
bedrock_content.append(block)
bedrock_content.append({"image": block["image"]})
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could we add/update unit test for this?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for calling this out. Looks like, both image and document are not supported in the output at the moment, so we should remove both.
https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_Message.html

Let me know if I am missing anything.

Copy link
Collaborator Author

@3coins 3coins Aug 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@baskaryan
Added unit tests.

@3coins 3coins force-pushed the add-document-type branch from 7fa963b to e49995a Compare August 20, 2024 02:47
@3coins 3coins merged commit 052818e into langchain-ai:main Aug 20, 2024
12 checks passed
@3coins 3coins deleted the add-document-type branch December 4, 2024 03:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add ChatBedrockConverse support for document content blocks
2 participants