[Question]: Do you plan to integrate AWS Bedrock at some point?

[roberttodor]

It would be great to have AWS LLM integrated into your platform

[JinHai-CN]

Yes, I added this requirement to the roadmap.

[Umpire2018]

In AWS Bedrock, The different model providers have individual request and response formats. Reference: here

• Anthropic Claude

enclosed_prompt = "Human: " + prompt + "\n\nAssistant:"

body = {
    "prompt": enclosed_prompt,
    "max_tokens_to_sample": 200,
    "temperature": 0.5,
    "stop_sequences": ["\n\nHuman:"],
}

response = self.bedrock_runtime_client.invoke_model(
    modelId="anthropic.claude-v2", body=json.dumps(body)
)

response_body = json.loads(response["body"].read())
completion = response_body["completion"]

return completion

• Mistral 7B

# Mistral instruct models provide optimal results when
# embedding the prompt into the following template:
instruction = f"<s>[INST] {prompt} [/INST]"

model_id = "mistral.mistral-7b-instruct-v0:2"

body = {
    "prompt": instruction,
    "max_tokens": 200,
    "temperature": 0.5,
}

response = self.bedrock_runtime_client.invoke_model(
    modelId=model_id, body=json.dumps(body)
)

response_body = json.loads(response["body"].read())
outputs = response_body.get("outputs")

completions = [output["text"] for output in outputs]

return completions

In my opinion, it is beneficial to leverage other widely-used frameworks to achieve multi-model support, which can free up resources for related work.

• litellm

import os
from litellm import completion

os.environ["AWS_ACCESS_KEY_ID"] = ""
os.environ["AWS_SECRET_ACCESS_KEY"] = ""
os.environ["AWS_REGION_NAME"] = ""

response = completion(
  model="bedrock/anthropic.claude-3-sonnet-20240229-v1:0",
  messages=[{ "content": "Hello, how are you?","role": "user"}]
)

• llama-index-llms-xxx Source-code

from llama_index.llms.bedrock import Bedrock

llm = Bedrock(
    model="amazon.titan-text-express-v1",
    aws_access_key_id="AWS Access Key ID to use",
    aws_secret_access_key="AWS Secret Access Key to use",
    aws_session_token="AWS Session Token to use",
    aws_region_name="AWS Region to use, eg. us-east-1",
)

resp = llm.complete("Paul Graham is ")