Creates a submodule for contributed functions #1572

gagb · 2024-02-07T01:13:16Z

Why are these changes needed?

Currently AutoGen supports contributed agents and capabilities.
Why not support a set of contributed functions?

This PR creates a new submodule in agentchat/contrib called functions.

Function store further contains many (stateless) functions that can be used with function calling API.

See the added notebook to see those function in use.

Specifying dependencies via a `FunctionWithRequirements` decorator

Right now, I specify dependencies for each function via a decorator.
The decorator wraps the function and check availability of dependencies.
If they aren't it install it. For example,

@FunctionWithRequirements(python_packages=["foo", "bar==0.1.0"], secrets=["BETA"])
def gamma():
    # logic that requires foo, bar, and BETA

Pending

Automate specification of description from doc string [done]
Registering bulk [done
Specifying secrets [done]
Add warnings for other dependencies?

Limitations

Doesn't support stateful tools

Related issue number

#1563

Checks

I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.

codecov-commenter · 2024-02-07T01:14:39Z

Codecov Report

Attention: 152 lines in your changes are missing coverage. Please review.

Comparison is base (5d81ed4) 35.03% compared to head (084ab2e) 48.32%.
Report is 2 commits behind head on main.

Files	Patch %	Lines
autogen/agentchat/contrib/functions/file_utils.py	0.00%	95 Missing ⚠️
...gen/agentchat/contrib/functions/functions_utils.py	0.00%	46 Missing ⚠️
...togen/agentchat/contrib/functions/youtube_utils.py	0.00%	11 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1572       +/-   ##
===========================================
+ Coverage   35.03%   48.32%   +13.28%     
===========================================
  Files          44       53        +9     
  Lines        5383     5838      +455     
  Branches     1247     1429      +182     
===========================================
+ Hits         1886     2821      +935     
+ Misses       3342     2792      -550     
- Partials      155      225       +70

Flag	Coverage Δ
unittests	`48.28% <0.00%> (+13.25%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ekzhu

Just one early comment: functions instead of function_store? People typically associate the word store with either a data store or an app store.

ekzhu · 2024-02-07T03:45:50Z

autogen/agentchat/contrib/function_store/file_utils.py

+    import requests
+    from pdfminer.high_level import PDFResourceManager, PDFPageInterpreter
+    from pdfminer.converter import TextConverter
+    from pdfminer.pdfpage import PDFPage


Maybe use a function generator so the import statements are only executed once in the wrapper.

def wrapper(): import ... def _actual_func(): ... return _actual_func()

Yeah, that's a great idea.

To put it more clearly,

@require("...") def function_generator() -> Callable: import ... import ... def actual_function(): # implementation return actual function

The function generator returns a callable which is the actual function that user is going to use.

Maybe there is a more elegant way to do this. @davorrunje would know.

But I think just importing everytime the function is executed is fine too -- much easier.

It depends on the package. Some take a while to import. It slows things down.

How about this?

@require("...") def function_generator() -> Tuple[Callable[None, None], Callable[..., Any]]: def _import(): import ... import ... def _function(*args, **kwargs): # implementation ... return (_import, _function)

This way executor can choose importing strategy by calling _import() when needed.

Another issue is that Python dependencies are only part of dependencies. Many Python libraries depend on particular C libraries being installed. We could additionally add base docker dependency to make sure everything needed is installed.

autogen/agentchat/contrib/function_store/youtube_utils.py

jackgerrits · 2024-02-07T13:03:19Z

Bundling the requirements with the function is a super good idea. However, if these functions weren't executed as "tools" but were shipped into the code execution environment then this mode of dependency installation would not work right? If the execute_function call was able to inspect if dependencies were needed and satisfy them explicitly then I feel like it would be more flexible? Because then the same could be done when this was used like a skill in another code execution environment

autogen/agentchat/contrib/function_store/function_store_utils.py

afourney · 2024-02-07T21:17:56Z

The @ requires decorator is super interesting.

I note that we are in the same situation for many contrib agents that are now added as optional dependencies (RAG, Teachability, WebSurfer, etc.) I wonder if we should consolidate on mechanisms here?

gagb · 2024-02-07T21:25:08Z

Bundling the requirements with the function is a super good idea. However, if these functions weren't executed as "tools" but were shipped into the code execution environment then this mode of dependency installation would not work right? If the execute_function call was able to inspect if dependencies were needed and satisfy them explicitly then I feel like it would be more flexible? Because then the same could be done when this was used like a skill in another code execution environment

If you remove requires, the agent will install the package from package missing error. I have some ideas to improve this. Will report back.

afourney · 2024-02-07T22:26:17Z

autogen/agentchat/contrib/function_store/youtube_utils.py

+    from youtube_transcript_api import YouTubeTranscriptApi
+
+    # Extract video ID from the YouTube link
+    video_id = youtube_link.split("v=")[1]


What if there are other querystring parameters like timecodes?

Maybe:

from urllib.parse import urlparse, parse_qs parsed_url = urlparse(youtube_link) qs_params = parse_qs(parsed_url.query) video_id = qs_params['v'][0]

afourney · 2024-02-07T22:28:19Z

autogen/agentchat/contrib/function_store/file_utils.py

+
+    for page in PDFPage.get_pages(file):
+        interpreter.process_page(page)
+


Why the per-page approach rather than:

pdfminer.high_level.extract_text(file)

jackgerrits · 2024-02-08T00:11:43Z

Bundling the requirements with the function is a super good idea. However, if these functions weren't executed as "tools" but were shipped into the code execution environment then this mode of dependency installation would not work right? If the execute_function call was able to inspect if dependencies were needed and satisfy them explicitly then I feel like it would be more flexible? Because then the same could be done when this was used like a skill in another code execution environment

If you remove requires, the agent will install the package from package missing error. I have some ideas to improve this. Will report back.

I'm totally happy to move forward with this. Just wanted to make sure resolving that part of that was raised and we can circle back to it

jackgerrits · 2024-02-29T14:56:41Z

autogen/agentchat/contrib/functions/functions_utils.py

+
+
+@runtime_checkable
+class UserDefinedFunction(Protocol):


The protocol should not actually provide an implementation

jackgerrits · 2024-02-29T14:56:52Z

autogen/agentchat/contrib/functions/functions_utils.py

+    name: str
+    docstring: str
+    code: str
+    python_packages: List[str]
+    env_vars: List[str]


The protocol should not mandate these attributes

jackgerrits · 2024-02-29T14:57:09Z

autogen/agentchat/contrib/functions/functions_utils.py

+    python_packages: List[str]
+    env_vars: List[str]
+
+    def name(self) -> str:


These should be properties instead of methods

gagb · 2024-03-25T23:09:28Z

@jackgerrits check if you need this for UDFs and take appropriate actions -- add to roadmap/close.

gitguardian · 2024-07-20T21:22:01Z

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

gagb added 2 commits February 7, 2024 01:10

Add initial example

0c1f7ab

Add decorator for yt utils

3af365e

gagb had a problem deploying to openai1 February 7, 2024 01:13 — with GitHub Actions Failure

gagb had a problem deploying to openai1 February 7, 2024 01:13 — with GitHub Actions Error

gagb had a problem deploying to openai1 February 7, 2024 01:13 — with GitHub Actions Failure

gagb changed the title ~~Contrib functions~~ Creates contrib.function_store Feb 7, 2024

gagb changed the title ~~Creates contrib.function_store~~ Creates a submodules for contributed functions Feb 7, 2024

gagb requested a review from afourney February 7, 2024 01:25

gagb changed the title ~~Creates a submodules for contributed functions~~ Creates a submodule for contributed functions Feb 7, 2024

gagb requested a review from ekzhu February 7, 2024 01:27

ekzhu requested a review from davorrunje February 7, 2024 03:38

ekzhu reviewed Feb 7, 2024

View reviewed changes

autogen/agentchat/contrib/function_store/youtube_utils.py Outdated Show resolved Hide resolved

afourney reviewed Feb 7, 2024

View reviewed changes

autogen/agentchat/contrib/function_store/function_store_utils.py Outdated Show resolved Hide resolved

afourney reviewed Feb 7, 2024

View reviewed changes

Merge branch 'main' into contrib-functions

8f5cc8b

gagb had a problem deploying to openai1 February 10, 2024 10:27 — with GitHub Actions Failure

gagb had a problem deploying to openai1 February 10, 2024 10:28 — with GitHub Actions Failure

gagb added tool-usage suggestion and execution of function/tool call enhancement labels Feb 13, 2024

gagb added 2 commits February 26, 2024 18:47

Merge branch 'main' into contrib-functions

abe4c41

Simplify to use a protocol

51823ad

gagb had a problem deploying to openai1 February 29, 2024 02:10 — with GitHub Actions Failure

jackgerrits reviewed Feb 29, 2024

View reviewed changes

afourney mentioned this pull request Mar 14, 2024

[Roadmap] Web Browsing #2017

Closed

gagb closed this Aug 26, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Creates a submodule for contributed functions #1572

Creates a submodule for contributed functions #1572

gagb commented Feb 7, 2024 •

edited

Loading

codecov-commenter commented Feb 7, 2024 •

edited

Loading

ekzhu left a comment

ekzhu Feb 7, 2024

afourney Feb 7, 2024

ekzhu Feb 9, 2024

ekzhu Feb 9, 2024

afourney Feb 9, 2024

davorrunje Feb 13, 2024

jackgerrits commented Feb 7, 2024

afourney commented Feb 7, 2024 •

edited

Loading

gagb commented Feb 7, 2024

afourney Feb 7, 2024

afourney Feb 7, 2024

jackgerrits commented Feb 8, 2024

jackgerrits Feb 29, 2024

jackgerrits Feb 29, 2024 •

edited

Loading

jackgerrits Feb 29, 2024

gagb commented Mar 25, 2024

gitguardian bot commented Jul 20, 2024 •

edited

Loading


		for page in PDFPage.get_pages(file):
		interpreter.process_page(page)



		@runtime_checkable
		class UserDefinedFunction(Protocol):

Creates a submodule for contributed functions #1572

Creates a submodule for contributed functions #1572

Conversation

gagb commented Feb 7, 2024 • edited Loading

Why are these changes needed?

Specifying dependencies via a FunctionWithRequirements decorator

Pending

Limitations

Related issue number

Checks

codecov-commenter commented Feb 7, 2024 • edited Loading

Codecov Report

ekzhu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackgerrits commented Feb 7, 2024

afourney commented Feb 7, 2024 • edited Loading

gagb commented Feb 7, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jackgerrits commented Feb 8, 2024

Choose a reason for hiding this comment

jackgerrits Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gagb commented Mar 25, 2024

gitguardian bot commented Jul 20, 2024 • edited Loading

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

gagb commented Feb 7, 2024 •

edited

Loading

Specifying dependencies via a `FunctionWithRequirements` decorator

codecov-commenter commented Feb 7, 2024 •

edited

Loading

afourney commented Feb 7, 2024 •

edited

Loading

jackgerrits Feb 29, 2024 •

edited

Loading

gitguardian bot commented Jul 20, 2024 •

edited

Loading