Correct token count for GPT-4v images #2420

BeibinLi · 2024-04-17T18:19:06Z

Why are these changes needed?

Related issue number

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

sonichi · 2024-04-17T20:13:40Z

test/test_token_count.py

@@ -83,7 +90,38 @@ def test_model_aliases():
    assert get_max_token_limit("gpt4-32k") == get_max_token_limit("gpt-4-32k")


+@pytest.mark.skipif(not img_util_imported, reason="img_utils not imported")
+def test_num_tokens_from_gpt_image():


Is this test covered by CI?

sonichi · 2024-04-17T20:13:56Z

test/agentchat/contrib/test_img_utils.py

@@ -290,5 +291,24 @@ def test_formatting(self):
        self.assertEqual(result, expected_output)


+@pytest.mark.skipif(skip, reason="dependency is not installed")


Is this test covered by CI?

afourney · 2024-04-17T20:19:07Z

Looks good, but have you tested these numbers with the OpenAI token calculator they publish? I'm imagining there are some subtleties to confirm (e.g., any tile overlap?)

WaelKarkoub

Changes make sense to me. I just had a question regarding supporting other models (gemini, claude, etc...) Are we saying gpt4v is a good estimate for all models or will there be a plan to support other models as well?

WaelKarkoub · 2024-04-17T21:04:50Z

autogen/token_count_utils.py

+                            warnings.warn(
+                                "img_utils or PIL not imported. Skipping image token count."
+                                "Please install autogen with [lmm] option.",
+                                ImportWarning,
+                            )


If the context history has 100 images, will this be printed 100 times?

Good news! Only 10 images are supported by OpenAI :) But yes... good question!

gitguardian · 2024-05-01T17:40:24Z

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

codecov-commenter · 2024-07-07T20:24:19Z

Codecov Report

Attention: Patch coverage is 17.64706% with 28 lines in your changes missing coverage. Please review.

Project coverage is 18.29%. Comparing base (6279247) to head (4a8a423).
Report is 10 commits behind head on main.

Files	Patch %	Lines
autogen/token_count_utils.py	20.00%	16 Missing ⚠️
autogen/agentchat/contrib/img_utils.py	14.28%	12 Missing ⚠️

❗ There is a different number of reports uploaded between BASE (6279247) and HEAD (4a8a423). Click for more details.

HEAD has 1 upload less than BASE

Flag BASE (6279247) HEAD (4a8a423)

unittests 2 1

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #2420       +/-   ##
===========================================
- Coverage   32.90%   18.29%   -14.61%     
===========================================
  Files          94       95        +1     
  Lines       10235    10444      +209     
  Branches     2193     2197        +4     
===========================================
- Hits         3368     1911     -1457     
- Misses       6580     8415     +1835     
+ Partials      287      118      -169

Flag	Coverage Δ
unittests	`18.29% <17.64%> (-14.61%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

julie2025 · 2024-07-08T09:43:20Z

Hi everyone,
I'm having an issue with the token count in MultimodalConversableAgent. It seems to be including the base64 encoded image URL when estimating the number of tokens and cost, which is incorrect.
Does anyone have any suggestions on how to fix this?

Thanks in advance.

Correct token count for GPT-4v images

4f34f5b

BeibinLi marked this pull request as ready for review April 17, 2024 18:19

BeibinLi requested review from ekzhu and sonichi April 17, 2024 18:20

sonichi reviewed Apr 17, 2024

View reviewed changes

sonichi requested review from thinkall, afourney, WaelKarkoub and yiranwu0 April 17, 2024 20:14

WaelKarkoub reviewed Apr 17, 2024

View reviewed changes

Merge branch 'main' into vision_cost

06a0a5d

BeibinLi had a problem deploying to openai1 May 31, 2024 17:39 — with GitHub Actions Failure

Merge branch 'main' into vision_cost

4a8a423

BeibinLi had a problem deploying to openai1 August 6, 2024 20:17 — with GitHub Actions Failure

BeibinLi closed this by deleting the head repository Aug 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Correct token count for GPT-4v images #2420

Correct token count for GPT-4v images #2420

BeibinLi commented Apr 17, 2024

sonichi Apr 17, 2024

sonichi Apr 17, 2024

afourney commented Apr 17, 2024 •

edited

Loading

WaelKarkoub left a comment

WaelKarkoub Apr 17, 2024

afourney Apr 17, 2024

gitguardian bot commented May 1, 2024 •

edited

Loading

codecov-commenter commented Jul 7, 2024

julie2025 commented Jul 8, 2024

		@@ -290,5 +291,24 @@ def test_formatting(self):
		self.assertEqual(result, expected_output)


		@pytest.mark.skipif(skip, reason="dependency is not installed")

Correct token count for GPT-4v images #2420

Correct token count for GPT-4v images #2420

Conversation

BeibinLi commented Apr 17, 2024

Why are these changes needed?

Related issue number

Checks

sonichi Apr 17, 2024

Choose a reason for hiding this comment

sonichi Apr 17, 2024

Choose a reason for hiding this comment

afourney commented Apr 17, 2024 • edited Loading

WaelKarkoub left a comment

Choose a reason for hiding this comment

WaelKarkoub Apr 17, 2024

Choose a reason for hiding this comment

afourney Apr 17, 2024

Choose a reason for hiding this comment

gitguardian bot commented May 1, 2024 • edited Loading

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

codecov-commenter commented Jul 7, 2024

Codecov Report

julie2025 commented Jul 8, 2024

afourney commented Apr 17, 2024 •

edited

Loading

gitguardian bot commented May 1, 2024 •

edited

Loading