New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[python] Claude Model Parser #1039

Merged

Ankush-lastmile merged 1 commit into main from pr1039

Jan 29, 2024

Member

Ankush-lastmile commented Jan 26, 2024 •

edited

Loading

[python] Claude Model Parser

Model Parser for Claude on AWS Bedrock

Claude on Bedrock can be called with the

anthropic_bedrock python library
AWS' Boto3 Library
Chose to use the anthropic_bedrock library because it has good abstractions and type hints

Claude on Bedrock only supports Text Completions, turn style prompts are not supported

Added Claude model parser as a core parser.
added dependency on anthropic_bedrock

Testplan

using prompt schema from diff ontop

claude.testplan.mov

Stack created with Sapling. Best reviewed with ReviewStack.

Ankush-lastmile force-pushed the pr1039 branch from 757f437 to c0e248d Compare

January 26, 2024 18:04

Ankush-lastmile changed the title ~~{wip} Claude Model Parser~~ [python] Claude Model Parser

Ankush-lastmile force-pushed the pr1039 branch 5 times, most recently from efab7cb to 933409a Compare

January 26, 2024 23:15

Ankush-lastmile mentioned this pull request

[editor] Claude Bedrock Prompt Schema #1050

Merged

Ankush-lastmile commented

View reviewed changes

python/src/aiconfig/Config.py Outdated

@@ @@ -53,6 +54,7 @@ @@
                   DefaultAnyscaleEndpointParser("AnyscaleEndpoint")
               )
               ModelParserRegistry.register_model_parser(GeminiModelParser("gemini-pro"), ["gemini-pro"])
+              ModelParserRegistry.register_model_parser(ClaudeBedrockModelParser(), ["Claude"])

Member Author

Ankush-lastmile Jan 27, 2024

Need to figure out model vs model provider but for now just registering under "Claude" as well

Ankush-lastmile commented

View reviewed changes

python/src/aiconfig/default_parsers/claude.py

+                      # See https://docs.anthropic.com/claude/docs/introduction-to-prompt-design#human--assistant-formatting
+                      completion_data[
+                          "prompt"
+                      ] = f"{HUMAN_PROMPT} {resolved_prompt}{AI_PROMPT}"

Member Author

Ankush-lastmile Jan 27, 2024

Unsure if its the best idea to manipulate the prompt inside the model parser

Ankush-lastmile commented

View reviewed changes

python/src/aiconfig/default_parsers/claude.py


		completion_data["stream"] = stream

		response = self.client.completions.create(**completion_data) # type: ignore (pyright doesn't understand response object)

Member Author

Ankush-lastmile Jan 27, 2024

API call will throw If aws credentials are not set. I removed mine locally and tested this manually.

Ankush-lastmile marked this pull request as ready for review

January 27, 2024 05:41

Ankush-lastmile requested review from saqadri, rholinshead, suyoglastmileai, jonathanlastmileai and rossdanlm as code owners

January 27, 2024 05:41

Ankush-lastmile force-pushed the pr1039 branch from 933409a to 9a53aa4 Compare

January 27, 2024 05:43

rossdanlm approved these changes

View reviewed changes

Contributor

rossdanlm left a comment

Nice, big ups

python/src/aiconfig/default_parsers/claude.py Outdated

Comment on lines 222 to 225

+                  # completion parameters to be used for Claude's Text completion api
+                  # prompt handled separately
+                  # streaming handled separately.

Contributor

rossdanlm Jan 29, 2024

nit: Add uri for easy reference

python/src/aiconfig/default_parsers/claude.py

+                  # prompt handled separately
+                  # streaming handled separately.
+                  supported_keys = {
+                      "max_tokens_to_sample",

Contributor

rossdanlm Jan 29, 2024

Note w. Ankush offline: required completion param, we should set default so people don't have to (if not defined in config)

python/src/aiconfig/default_parsers/claude.py

+                  """
+                  return ExecuteResult(
+                      output_type="execute_result",
+                      data=response.completion,

Contributor

rossdanlm Jan 29, 2024

Nit: add link to confirm this is a text-based output only

python/src/aiconfig/default_parsers/claude.py

+                      output_type="execute_result",
+                      data=response.completion,
+                      execution_count=0,
+                      metadata=response.model_dump(),

Contributor

rossdanlm Jan 29, 2024

oh wow, pydantic nice thanks Anthropic

python/src/aiconfig/default_parsers/claude.py Outdated

+                  for iteration in response:
+                      new_text = iteration.completion
+                      # Reduce

Contributor

rossdanlm Jan 29, 2024

nit: remove because "reduce" typically has other connotation to it

python/src/aiconfig/default_parsers/claude.py

+                      options (InferenceOptions): The inference options. Used to determine the stream callback.
+                  """
+                  accumulated_message = ""
+                  output = None

Contributor

rossdanlm Jan 29, 2024

nit: this always gets overridden so delete

python/src/aiconfig/default_parsers/claude.py

Comment on lines +29 to +30

		# Client will be set in the run method. This is to avoid having to set the api key in the constructor
		self.client = None

Contributor

rossdanlm Jan 29, 2024

nit: Create an issue, mention as an issue because this MP is a little weird having 2 potential sources of API KEYS:

aiconfig/extensions/Gemini/python/src/aiconfig_extension_gemini/Gemini.py

Lines 116 to 122 in c838856

    
           # Note: we don't need to explicitly set the key as long as this is set 
        
           # as an env var genai.configure() will pick up the env var 
        
           # `GOOGLE_API_KEY`, it's just that we prefer not to call 
        
           # `get_api_key_from_environment` multiple times if we don't need to 
        
           self.api_key = get_api_key_from_environment( 
        
               "GOOGLE_API_KEY", required=False 
        
           ).unwrap()

This will def come up with the SDK rewrites for MPs

python/src/aiconfig/default_parsers/claude.py

Comment on lines +171 to +174

+                      if options is not None and options.stream is not None:
+                          stream = options.stream
+                      elif "stream" in completion_data:
+                          stream = completion_data["stream"]

Contributor

rossdanlm Jan 29, 2024

Nice this is cleaner than the other implementations we have

python/src/aiconfig/default_parsers/claude.py

+                      else:
+                          output = construct_output(response)  # type: ignore
+                      # rewrite or extend list of outputs?

Contributor

rossdanlm Jan 29, 2024

Yea because this is a chat based one right? Hmmmm, let's do demo you and I today. For now unblocking and we can fix forward based on our desired behaviour

python/src/aiconfig/default_parsers/claude.py

Comment on lines +213 to +214

		# Claude outputs should only ever be string
		# format so shouldn't get here, but just being safe

Contributor

rossdanlm Jan 29, 2024

That's what I'm talking about


          [python] Claude Model Parser

94bb0d8

Model Parser for Claude on AWS Bedrock



Claude on Bedrock can be called with the
- `anthropic_bedrock` python library
- AWS' `Boto3` Library
- Chose to use the `anthropic_bedrock` library because it has good abstractions and type hints

Claude on Bedrock only supports Text Completions, turn style [prompts are not supported](https://docs.anthropic.com/claude/reference/claude-on-amazon-bedrock#:~:text=Messages%20in%20Amazon%20Bedrock)

- Added Claude model parser as a core parser.
- added dependency on `anthropic_bedrock`



## Testplan

using prompt schema from diff ontop

https://github.com/lastmile-ai/aiconfig/assets/141073967/29c1faa7-7d13-412f-8606-9ad556eb1c52

Ankush-lastmile force-pushed the pr1039 branch from 9a53aa4 to 94bb0d8 Compare

January 29, 2024 22:32

Member Author

Ankush-lastmile commented Jan 29, 2024

Removed "Claude" from default model parser registry as a Model Id, just have parser id for now. Will need to revist.
addressed comments, most of them were nits.

Ankush-lastmile added a commit that referenced this pull request


          [editor] Claude Bedrock Prompt Schema (#1050)

13b1e76

[editor] Claude Bedrock Prompt Schema


## Testplan


https://github.com/lastmile-ai/aiconfig/assets/141073967/29c1faa7-7d13-412f-8606-9ad556eb1c52

---
[//]: # (BEGIN SAPLING FOOTER)
Stack created with [Sapling](https://sapling-scm.com). Best reviewed
with
[ReviewStack](https://reviewstack.dev/lastmile-ai/aiconfig/pull/1050).
* __->__ #1050
* #1039

Ankush-lastmile merged commit 94bb0d8 into main

2 checks passed

This pull request was closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

rossdanlm rossdanlm approved these changes

saqadri Awaiting requested review from saqadri saqadri is a code owner

rholinshead Awaiting requested review from rholinshead rholinshead is a code owner

suyoglastmileai Awaiting requested review from suyoglastmileai

jonathanlastmileai Awaiting requested review from jonathanlastmileai jonathanlastmileai is a code owner

Labels

None yet