Allow costs and window size limits to be specified in the config_list #1682

afourney · 2024-02-14T23:15:14Z

Why are these changes needed?

See issue #1680.

TL;DR:
Token limits and costs are hardcoded in various files. Adding new models requires editing numerous files. Costs are keyed off model names, but those name may not be reliable. For example, "gpt-4" on Azure might point to an 8k, 32k, or 128k model. There is no requirement that these match with OpenAI names. Likewise, model names may not match at all (e.g., "my-gpt-4")
Adding new models or providers requires hand-editing core library files.

This PR allows costs and limits to be specified in the config. E.g.,

[
        "model": "gpt-4-turbo",
        "api_key": "blahblahblah",
        "base_url": "https://mymodel.openai.azure.com/",
        "api_type": "azure",
        "api_version": "2023-12-01-preview",
        "window_size":  { "input": 128000, "output":  4096 },
        "token_cost_1k":  { "input": 0.01, "output": 0.03 ) 
]

Limitations: Vision models are not fully-supported. We can compute the cost only after we get the tokens counts back after making the call.

Related issue number

Closes #1680, #1270

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

…LIST

codecov-commenter · 2024-02-14T23:17:50Z

Codecov Report

Attention: 12 lines in your changes are missing coverage. Please review.

Comparison is base (f68c09b) 37.85% compared to head (7148a4d) 49.60%.

Files	Patch %	Lines
autogen/oai/client.py	25.00%	12 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1682       +/-   ##
===========================================
+ Coverage   37.85%   49.60%   +11.75%     
===========================================
  Files          50       50               
  Lines        5733     5737        +4     
  Branches     1297     1408      +111     
===========================================
+ Hits         2170     2846      +676     
+ Misses       3391     2662      -729     
- Partials      172      229       +57

Flag	Coverage Δ
unittests	`49.57% <25.00%> (+11.72%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

olgavrou · 2024-02-15T14:44:14Z

autogen/oai/client.py

@@ -314,6 +327,8 @@ class OpenAIWrapper:
        "api_version",
        "api_type",
        "tags",
+        "window_size",


are we doing anything with window size? or planning to do anything with it?

Yes. Both :)

I have yet to modify WebSurfer (this is still a raft PR), but it will use WindowSize to understand how to break up pages.

I also use it in the complex_tasks branch with SocietyOfMind and will port those changes to main once this is ready and merged.

olgavrou

wondering if this needs to be passed into cost or if we can just pass all the args into the client constructor (as I believe is already done?) and re-use in different methods, that way we don't have to change the API every time we want to use a new config value

afourney · 2024-02-15T15:01:38Z

wondering if this needs to be passed into cost or if we can just pass all the args into the client constructor (as I believe is already done?) and re-use in different methods, that way we don't have to change the API every time we want to use a new config value

I'd love to figure this out. I think the issue I was working around is that the main wrapper client takes the config list, which may create many OAI or Azure clients, and we don't know which is used until we make the call. The cost method I had initially targeted for modification is a member of the wrapper client class.

I'm not sure I fully understand the solution to this you are proposing, but I would be happy to get rid of the new parameter if possible.

olgavrou · 2024-02-15T15:26:14Z

wondering if this needs to be passed into cost or if we can just pass all the args into the client constructor (as I believe is already done?) and re-use in different methods, that way we don't have to change the API every time we want to use a new config value

I'd love to figure this out. I think the issue I was working around is that the main wrapper client takes the config list, which may create many OAI or Azure clients, and we don't know which is used until we make the call. The cost method I had initially targeted for modification is a member of the wrapper client class.

I'm not sure I fully understand the solution to this you are proposing, but I would be happy to get rid of the new parameter if possible.

If we pass any extra kwargs that we need to the client constructors so for example OpenAIClient, then when OpenAIClient's cost or create or other function is called it will have access to things like token cost, window size, etc

maxim-saplin

Looks good. Though I think docs is an important piece that is missing from this PR:

https://microsoft.github.io/autogen/docs/llm_configuration
Docstrings at openai_utils.py

maxim-saplin · 2024-02-16T17:19:31Z

autogen/oai/client.py

+        self,
+        response: Union[ChatCompletion, Completion],
+        token_cost_1k: Optional[Union[float, Dict[str, float]]] = None,
+    ) -> float:


Not critical, yet it might be reasonable to switch to Decimal type (float smells in currency/money values :) - openai_utils.py included

yiranwu0 · 2024-02-28T22:52:13Z

Should we force or at least warn user to pass in token_cost if they are using azure?

From my previous experience azure always return 'gpt-4' in response['model'], this will very likely cause a wrong calculation of cost.
See #1270

afourney · 2024-02-28T23:24:35Z

Should we force or at least warn user to pass in token_cost if they are using azure?

From my previous experience azure always return 'gpt-4' in response['model'], this will very likely cause a wrong calculation of cost. See #1270

That is a good idea. This has been my experience too. Perhaps it should warn or force for any model that isn't OpenAI.

gitguardian · 2024-07-20T21:22:00Z

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

afourney · 2024-09-23T19:11:53Z

Closing this PR, since it is quite dated. Will open a new issue.

Allow costs and window size limits to be specified in the OAI_CONFIG_…

25f2bfd

…LIST

afourney had a problem deploying to openai1 February 14, 2024 23:15 — with GitHub Actions Failure

gagb requested review from olgavrou and yiranwu0 February 14, 2024 23:40

gagb added the enhancement label Feb 14, 2024

Attempt to fix CustomClient test failures.

7148a4d

afourney had a problem deploying to openai1 February 15, 2024 00:47 — with GitHub Actions Failure

sonichi requested review from maxim-saplin and davorrunje February 15, 2024 02:18

olgavrou reviewed Feb 15, 2024

View reviewed changes

maxim-saplin reviewed Feb 16, 2024

View reviewed changes

afourney mentioned this pull request Mar 6, 2024

[Roadmap]: Complex Tasks Work Items (GAIA) #1369

Closed

This was referenced Mar 14, 2024

[Roadmap] Web Browsing #2017

Closed

[Outdated] Support multimodal (GPT-4V) directly in OpenAIClient #2013

Closed

afourney mentioned this pull request Apr 18, 2024

[Feature Request]: llm_config with input_price and output_price #2435

Open

afourney closed this Sep 23, 2024

afourney deleted the i1680 branch September 25, 2024 17:37

afourney mentioned this pull request Sep 25, 2024

[Feature Request]: Generalize token accounting and budgeting #1680

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow costs and window size limits to be specified in the config_list #1682

Allow costs and window size limits to be specified in the config_list #1682

afourney commented Feb 14, 2024 •

edited by yiranwu0

Loading

codecov-commenter commented Feb 14, 2024 •

edited

Loading

olgavrou Feb 15, 2024 •

edited

Loading

afourney Feb 15, 2024

olgavrou left a comment

afourney commented Feb 15, 2024

olgavrou commented Feb 15, 2024

maxim-saplin left a comment

maxim-saplin Feb 16, 2024 •

edited

Loading

yiranwu0 commented Feb 28, 2024

afourney commented Feb 28, 2024

gitguardian bot commented Jul 20, 2024 •

edited

Loading

afourney commented Sep 23, 2024

Allow costs and window size limits to be specified in the config_list #1682

Allow costs and window size limits to be specified in the config_list #1682

Conversation

afourney commented Feb 14, 2024 • edited by yiranwu0 Loading

Why are these changes needed?

Related issue number

Checks

codecov-commenter commented Feb 14, 2024 • edited Loading

Codecov Report

olgavrou Feb 15, 2024 • edited Loading

Choose a reason for hiding this comment

afourney Feb 15, 2024

Choose a reason for hiding this comment

olgavrou left a comment

Choose a reason for hiding this comment

afourney commented Feb 15, 2024

olgavrou commented Feb 15, 2024

maxim-saplin left a comment

Choose a reason for hiding this comment

maxim-saplin Feb 16, 2024 • edited Loading

Choose a reason for hiding this comment

yiranwu0 commented Feb 28, 2024

afourney commented Feb 28, 2024

gitguardian bot commented Jul 20, 2024 • edited Loading

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

afourney commented Sep 23, 2024

afourney commented Feb 14, 2024 •

edited by yiranwu0

Loading

codecov-commenter commented Feb 14, 2024 •

edited

Loading

olgavrou Feb 15, 2024 •

edited

Loading

maxim-saplin Feb 16, 2024 •

edited

Loading

gitguardian bot commented Jul 20, 2024 •

edited

Loading