feat: added groq provider #494

lifeizhou-ap · 2024-12-18T06:40:13Z

Added groq provider
Converted content object list in message (ToolResponse message) to string to match api schema only for Ollama and Groq.
Removed the duplications in providers that compatible wit openai api schema
Extracted open ai related utils function to a new file openai_utils
Added integration tests for google and groq providers and extract the utils to mock_server.rs

Note:
I could not get the capture screenshot working with groq provider as when the message include the toolResult with image the api returns bad request. So far I found they can categorised into 2 scenarios as below:

For most non-vision models, it does not accept the user message with image although in the doc it says it does.

Error message:
{ "request_id": "req_01jfckkap9fcdv2c4cr4tepnfw", "created_at": "2024-12-18T10:04:15.434Z", "error": { "message": "message[0].content must be a string", "type": "invalid_request_error" } }
I also tried llama-3.2-11b-vision-preview, it does not accept the payload including the user message with image and system message at the same time.
Error message:
{ "request_id": "req_01jfcmfx6yf0zbjt4028v4gkcd", "created_at": "2024-12-18T10:19:51.904Z", "error": { "message": "prompting with images is incompatible with system messages", "type": "invalid_request_error" } }

I tried with the python cli, it does not work either but with different reason. I did not look in-depth. I feel this feature does not work originally and we can enhance this in the future when we have more usages with groq

github-actions · 2024-12-18T06:40:26Z

Hey there and thank you for opening this pull request! 👋🏼

We require pull request titles to follow the Conventional Commits specification and it looks like your proposed title needs to be adjusted.

Details:

No release type found in pull request title "Lifei/groq provider". Add a prefix to indicate what kind of release this pull request corresponds to. For reference, see https://www.conventionalcommits.org/

Available types:
 - feat: A new feature
 - fix: A bug fix
 - docs: Documentation only changes
 - style: Changes that do not affect the meaning of the code (white-space, formatting, missing semi-colons, etc)
 - refactor: A code change that neither fixes a bug nor adds a feature
 - perf: A code change that improves performance
 - test: Adding missing tests or correcting existing tests
 - build: Changes that affect the build system or external dependencies (example scopes: gulp, broccoli, npm)
 - ci: Changes to our CI configuration files and scripts (example scopes: Travis, Circle, BrowserStack, SauceLabs)
 - chore: Other changes that don't modify src or test files
 - revert: Reverts a previous commit

* v1.0: requires foreign architectures more xcompile deps x compilation tools chore: Cargo build tokenizers (#491) feat: build and release binaries to GH releases (#477) fix: width of bubbles and logging errors (#487)

lifeizhou-ap · 2024-12-18T06:57:42Z

crates/goose/src/providers/openai_utils.rs

+                                    }
+                                }
+                            }
+                            let concatenated_content = tool_content


Converted content object list in message (ToolResponse message) to string to match api schema. In openai, it accepted "string" or "array", but in groq it only accepts "string" based on their api

@baxen is this related to the error you were seeing on the llama3 databricks endpoint?

I've changed to only apply the concatenate logic to ollama and groq

zakiali · 2024-12-18T23:20:42Z

crates/goose/src/providers/factory.rs

    databricks::DatabricksProvider, google::GoogleProvider, ollama::OllamaProvider,
    openai::OpenAiProvider,
 };
+use crate::providers::groq::GroqProvider;


prob move this up into the super

zakiali · 2024-12-18T23:20:48Z

crates/goose/src/providers/databricks.rs

+use crate::providers::openai_utils::{
+    check_openai_context_length_error, messages_to_openai_spec, openai_response_to_message,
+    tools_to_openai_spec,
+};


nit: we should prob use the get_openai_usage and handle_response in this one as well.

zakiali · 2024-12-18T23:24:52Z

crates/goose-cli/src/commands/configure.rs

        "ollama" => OLLAMA_MODEL,
        "anthropic" => "claude-3-5-sonnet-2",
-        "google" => "gemini-1.5-flash",
+        "google" => GOOGLE_DEFAULT_MODEL,


nit: this seems like something we should standardize in the providers for openai, databricks, and anthropic

zakiali · 2024-12-18T23:33:51Z

crates/goose/src/providers/groq.rs

+        Ok(Self { client, config })
+    }
+
+    fn get_usage(data: &Value) -> anyhow::Result<Usage> {


we should consider adding get_usage as a trait for Providers in the base object

agree. This PR has many changes. I can create a follow-up PR after this PR is merged

michaelneale · 2024-12-19T05:30:36Z

@lifeizhou-ap I wouldn't worry about image support for groq and llama models for now if they are proving a problem.

zakiali · 2024-12-19T23:26:07Z

crates/goose-server/src/configuration.rs

        #[serde(default)]
        max_tokens: Option<i32>,
    },
+    Groq {


one thing I missed in the google provider and here initially was that we also want to add context_limit and estimate_factor here as well

zakiali · 2024-12-19T23:28:20Z

crates/goose-server/src/configuration.rs

+                host,
+                api_key,
+                model: ModelConfig::new(model)
+                    .with_temperature(temperature)


context_limit and estimate_factor should go here as well for Groq and Google

zakiali · 2024-12-19T23:45:45Z

crates/goose/src/providers/groq.rs

+        tools: &[Tool],
+    ) -> anyhow::Result<(Message, ProviderUsage)> {
+        let payload =
+            create_openai_request_payload(&self.config.model, system, messages, tools, true)?;


the boolean argument here (true) is a little confusing. I don't think there is a way to specify kwargs, but maybe setting a var before calling the function (e.g. concat_tool_response_contents=true) and passing that in to the function call would be a little more clear?

zakiali

looks good! left a few comments

* v1.0: Update cli-release.yml feat: added groq provider (#494) fix: use rust tls (#500) fix: Ldelalande/fix scroll (#504) feat: MCP server sdk (simple version first) (#499) tiny change to use most recent in stack (#501) stop bubbles filling screen (#495) chore: V1.0 release automation (#493)

* v1.0: (43 commits) feat: openrouter provider (#538) [ui] chore: tidy up gui providers (#537) [ui]: Polish and system theme fix (#533) [ui]: General ui polish to more closely match designs (#532) Latency issue fix with prepare_inference (#535) chore: use cross to build binaries (#507) feat: a system for non developers to augment developer system (#524) fix: Broken open directory and new session buttons in more menu (#520) refactor: move get_usage to provider trait (#506) fix: Make stop button more obvious (#516) fix: Enhance Dark mode menu dots visibility (#517) working on automating release of .zip and binaries and having them on each PR as well (#509) conditionally load memory system in goose server (#508) Adds 'instructions' field to InitializeResult (#511) feat: MCP client sdk (#505) Update cli-release.yml feat: added groq provider (#494) fix: use rust tls (#500) fix: Ldelalande/fix scroll (#504) feat: MCP server sdk (simple version first) (#499) ...

lifeizhou-ap added 7 commits December 18, 2024 14:26

added groq provider

3b8ce1d

extract handle_response

eef6f3c

extract more functions

4715cb9

fixed more format

07147eb

convert content to string to match api schema

0a91706

moved open ai specific utils to a separate file

ab705fd

fixed tests

8ce3bdd

lifeizhou-ap changed the base branch from main to v1.0 December 18, 2024 06:40

lifeizhou-ap changed the title ~~Lifei/groq provider~~ feat: added groq provider Dec 18, 2024

lifeizhou-ap marked this pull request as ready for review December 18, 2024 06:47

lifeizhou-ap added 2 commits December 18, 2024 17:49

Merge branch 'v1.0' into lifei/groq-provider

4a51019

* v1.0: requires foreign architectures more xcompile deps x compilation tools chore: Cargo build tokenizers (#491) feat: build and release binaries to GH releases (#477) fix: width of bubbles and logging errors (#487)

fix format

726471d

lifeizhou-ap commented Dec 18, 2024

View reviewed changes

lifeizhou-ap added 2 commits December 18, 2024 20:16

added llama tokenizer

38ecbdc

added missing qwen tokenizer

d9d184d

lifeizhou-ap requested a review from zakiali December 18, 2024 22:15

zakiali reviewed Dec 18, 2024

View reviewed changes

lifeizhou-ap added 7 commits December 19, 2024 11:07

clean up

c025802

refactored openai integration tests

e3485bf

refactored databricks integration tests

152ea97

refactored ollma integration tests

aa6b678

added groq integration test

d66e48a

added google provider integration tests

13d8967

used util functions

5cd0c06

only concat tool response content for ollama and groq

ab5279e

lifeizhou-ap added 4 commits December 20, 2024 07:49

only concat tool response content for ollama and groq

7f4a90f

fixed the test

529f5d1

fixed the format

2b2dad7

clean up

0674f30

zakiali reviewed Dec 19, 2024

View reviewed changes

zakiali approved these changes Dec 20, 2024

View reviewed changes

lifeizhou-ap merged commit a30094b into v1.0 Dec 20, 2024
3 checks passed

lifeizhou-ap mentioned this pull request Dec 20, 2024

refactor: move get_usage to provider trait #506

Merged

yingjiehe-xyz deleted the lifei/groq-provider branch February 5, 2025 21:05

feat: added groq provider #494

feat: added groq provider #494

Uh oh!

Conversation

lifeizhou-ap commented Dec 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Dec 18, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelneale commented Dec 19, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zakiali left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lifeizhou-ap commented Dec 18, 2024 •

edited

Loading