New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable Local Function Calling using NexusRaven V2 #1937

Closed

gregnwosu wants to merge 62 commits into microsoft:main from gregnwosu:check

gregnwosu commented Mar 9, 2024

add local function calling assistant for nexusravenv2 local function calling

Why are these changes needed?

This change enables the user to create a reliable local function calling using the NexusRaven-V2-13 model

Related issue number

N/A

Checks

[ x I've included any doc changes needed for https://microsoft.github.io/autogen/. See https://microsoft.github.io/autogen/docs/Contribute#documentation to build and test documentation locally.
I've added tests (if relevant) corresponding to the changes introduced in this PR.
I've made sure all auto checks have passed.


          Create nexusravenv2_local_function_calling.py

047a52c

add local function calling assistant for nexusravenv2 local function calling

Author

gregnwosu commented Mar 9, 2024

@gregnwosu please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.
@microsoft-github-policy-service agree [company="{your company}"]
Options:

(default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.
@microsoft-github-policy-service agree
(when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.
@microsoft-github-policy-service agree company="Microsoft"
Contributor License Agreement

@microsoft-github-policy-service agree company="Heeboe Ltd"

ekzhu requested changes

View reviewed changes

Collaborator

ekzhu left a comment

Thanks for the PR. It opens door for us to consider various open source models. Would you be willing to help maintain this contrib module if this PR is merged?

autogen/agentchat/contrib/nexusravenv2_local_function_calling.py Outdated Show resolved Hide resolved

autogen/agentchat/contrib/nexusravenv2_local_function_calling.py Outdated

+                      return function_name, args_map, thought_part.strip()
+                  @override
+                  def _generate_oai_reply_from_client(self, llm_client, messages, cache) -> Union[str, Dict, None]:

Collaborator

ekzhu Mar 10, 2024

An abstraction leakage here. We might change the private method here in the future. Perhaps a complete reply function instead of overriding?

Collaborator

ekzhu Mar 10, 2024

Also type hints for the arguments.

autogen/agentchat/contrib/nexusravenv2_local_function_calling.py Outdated

		return f'''OPTION:\n<func_start>def {name}({args})<func_end>\n<docstring_start>\n"""\n{description.strip()}\n\nArgs:\n{arg_types}\n"""\n<docstring_end>'''


		def add_nexus_raven_prompts(func):

Collaborator

ekzhu Mar 10, 2024

Would you consider using a custom model client to handle the request and response transformation? Should be a better encapuslation of these transformation logics.

Author

gregnwosu Mar 11, 2024

I would need to understand how to create it. Would it be enough just to extend ConversableAgent and copy the _ generate, functions or would that be too much duplication is there are more elegant way?

autogen/agentchat/contrib/nexusravenv2_local_function_calling.py Outdated Show resolved Hide resolved

autogen/agentchat/contrib/nexusravenv2_local_function_calling.py Outdated



		class NexusFunctionCallingAssistant(autogen.AssistantAgent):
		def __init__(self, **kwargs):

Collaborator

ekzhu Mar 10, 2024

Since this is a custom agent class for a specific type of model and purpose (i.e., no code execution, etc.), I think we can have a very specialized constructor to expose only model and tool-call specific settings. You might want to customize the system message as well.

Also, if using custom model client (https://microsoft.github.io/autogen/blog/2024/01/26/Custom-Models), you also want to register the client in the constructor.

Author

gregnwosu commented Mar 11, 2024

Thanks for the PR. It opens door for us to consider various open source models. Would you be willing to help maintain this contrib module if this PR is merged?

yes , happy to maintain


          feat: ✨ resolving some comments from autogen on pull request for nexu…

622fd71

…sfunction calling agent

codecov-commenter commented Mar 12, 2024 •

edited

Loading

Codecov Report

Attention: Patch coverage is 0% with 67 lines in your changes are missing coverage. Please review.

Project coverage is 49.29%. Comparing base (f795265) to head (ea11cbc).
Report is 10 commits behind head on main.

Files	Patch %	Lines
...hat/contrib/nexusravenv2_local_function_calling.py	0.00%	67 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1937       +/-   ##
===========================================
+ Coverage   37.83%   49.29%   +11.46%     
===========================================
  Files          77       78        +1     
  Lines        7766     7847       +81     
  Branches     1663     1816      +153     
===========================================
+ Hits         2938     3868      +930     
+ Misses       4579     3655      -924     
- Partials      249      324       +75

Flag	Coverage Δ
unittest	`14.24% <0.00%> (?)`
unittests	`48.24% <0.00%> (+10.42%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gregnwosu added 5 commits

March 12, 2024 08:24


          feat: ✨ make the end-bot part of the regex for nexus optional

e703e47


          docs: 📝 added a TODO note

94ba2c5


          feat: ✨ make the end-bot part of the regex for nexus optional

b1b8749


          feat: ✨ make the end-bot part of the regex for nexus optional

e33c91b


          updated formatting

819db70

Member

jackgerrits commented Mar 12, 2024

@gregnwosu I'd love to collaborate with you on this. I reached out on discord to discuss more

gregnwosu and others added 6 commits

March 12, 2024 18:41


          feat: 🐛 fix issue with numbers being quoted

d65c007


          Merge branch 'main' into main

2673a2b


          Update random_word_generator function call parameters

3a2d5ee


          Update random_word_generator function call parameters

1fc846c


          Update to remove custom create, updated prompt formatting, support op…

88c2f68

…tional thought


          pre-commit

df44ed8

Member

jackgerrits commented Mar 12, 2024

@ekzhu I think we can can avoid a custom model client for now by having a custom reply function. Could you take a look at the new version of the branch? We changed things quite a lot

ekzhu requested changes

View reviewed changes

Collaborator

ekzhu left a comment •

edited

Loading

I really like the changes that has been made here. Though one issue we have is this is hard to test in CI and may subject to frequent changes for different NextRaven model versions.

Can we turn this into a notebook? This seems like a perfect example of a notebook we can put under /docs/topics/ see the roadmap issue #1994. If you agree we can put this under that roadmap issue. Not only this gives an example of how to use nextraven, but also showcases an example of register_reply.

Also, having this inside a notebook allow use to put in useful context like nextraven model version this is tested on and explanation of the code you wrote.

--- update

Can you add unit tests like other contrib agents?

Can you create a notebook example to show case the new agent? For example we can use the currency exchange example.

Author

gregnwosu commented Mar 16, 2024

I really like the changes that has been made here. Though one issue we have is this is hard to test in CI and may subject to frequent changes for different NextRaven model versions.

Can we turn this into a notebook? This seems like a perfect example of a notebook we can put under /docs/topics/ see the roadmap issue #1994. If you agree we can put this under that roadmap issue. Not only this gives an example of how to use nextraven, but also showcases an example of register_reply.

Also, having this inside a notebook allow use to put in useful context like nextraven model version this is tested on and explanation of the code you wrote.

i dont think this will change much over different versions of nextraven its really more dependant on the output format of nexus raven and not much on the internals. the output format should not change between versions. if it does it simple regex parsing that needs to be updated, as ive said im happy to maintain it.

im also happy to get a working example running in a notebook if that helps. nexus raven is available in ollama so version management is possible ( https://ollama.com/library/nexusraven) we can pin specific versions in notebooks

Collaborator

ekzhu commented Mar 16, 2024 •

edited

Loading

Okay. Just two more things before we can merge this:

Can you add unit tests like other contrib agents?

Can you create a notebook example to show case the new agent? For example we can use the currency exchange example.

gregnwosu added 8 commits

March 26, 2024 10:21


          Merge branch 'microsoft:main' into main

e07b06b


          first attempt at mocking nexusraven local function calling.

88b01fc

TODO: make it work, fix mocks


          fixed errors but no results

f7af102


          fixed errors and results but i would like user proxy to execute the f…

2bc340d

…unction call!


          fixed errors and results but i would like user proxy to execute the f…

11f1be1

…unction call!


          fixed type errors for early python

347184f


          added pytest-mock dependency

acafed0


          added pytest-mock dependency

4b5f1ef

gregnwosu added 13 commits

March 31, 2024 00:39


          remove need to override receive

22b8fec


          Merge branch 'main' of https://github.com/gregnwosu/autogen into check

6a312f5


          Merge branch 'check' of https://github.com/gregnwosu/autogen into check

0f956c5


          Merge branch 'main' into check

0e1acf5


          formatting

f5681fa


          formatting

7bf751d


          Merge branch 'check' of https://github.com/gregnwosu/autogen into check

be79732


          formatting adding test assertions

35523ac


          removing overrider for receive

c59802e


          reformatting

dcc0be0


          reformatting

19ed13b


          refactored and improve tests

bdecfbf


          refactored and improve tests

ea11cbc

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gregnwosu had a problem deploying to openai1

May 1, 2024 23:41

— with

GitHub Actions Failure

gitguardian bot commented Jul 20, 2024 •

edited

Loading

⚠️ GitGuardian has uncovered 96 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

Since your pull request originates from a forked repository, GitGuardian is not able to associate the secrets uncovered with secret incidents on your GitGuardian dashboard.
Skipping this check run and merging your pull request will create secret incidents on your GitGuardian dashboard.

🔎 Detected hardcoded secrets in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
12853598	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`eff19ac`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`06a0a5d`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`0524c77`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`d7ea410`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e43a86c`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`841ed31`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`802f099`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`9a484d8`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`e973ac3`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`89650e7`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`e07b06b`	.github/workflows/dotnet-release.yml	View secret
10404662	Triggered	Generic CLI Secret	`abe4c41`	.github/workflows/dotnet-build.yml	View secret
10404662	Triggered	Generic CLI Secret	`7362fb9`	.github/workflows/dotnet-release.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`abad9ff`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`c7bb588`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`b97b99d`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`49e8053`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`d422c63`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`97fa339`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`501610b`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`954ca45`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`e43a86c`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`bdb40d7`	test/oai/test_utils.py	View secret
12853602	Triggered	Generic High Entropy Secret	`79dbb7b`	test/oai/test_utils.py	View secret
11616921	Triggered	Generic High Entropy Secret	`a86d0fd`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`394561b`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`3eac646`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`f45b553`	notebook/agentchat_agentops.ipynb	View secret
11616921	Triggered	Generic High Entropy Secret	`6563248`	notebook/agentchat_agentops.ipynb	View secret
12853598	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853598	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404693	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404662	Triggered	Generic CLI Secret	`954ca45`	.github/workflows/dotnet-build.yml	View secret
12853599	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853599	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404694	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`3b79cc6`	test/oai/test_utils.py	View secret
10404695	Triggered	Generic High Entropy Secret	`11baa52`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853600	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
12853601	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`11baa52`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10493810	Triggered	Generic Password	`3b79cc6`	notebook/agentchat_pgvector_RetrieveChat.ipynb	View secret
10404696	Triggered	Generic High Entropy Secret	`0a3c6c4`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`76f5f5a`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret
10404696	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`2b3a9ae`	test/oai/test_utils.py	View secret
10422482	Triggered	Generic High Entropy Secret	`c03558f`	test/oai/test_utils.py	View secret

and 16 others.

🛠 Guidelines to remediate hardcoded secrets

Understand the implications of revoking this secret by investigating where it is used in your code.
Replace and store your secrets safely. Learn here the best practices.
Revoke and rotate these secrets.
If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

following these best practices for managing and storing secrets including API keys and other credentials
install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

Member

jackgerrits commented Sep 23, 2024

Hey @gregnwosu are you still interested in this change?

Author

gregnwosu commented Sep 24, 2024

Hey @gregnwosu are you still interested in this change?

no i have given up on it

Member

jackgerrits commented Sep 24, 2024

No worries at all, going to close this then. Let me know if you’re interested in future :)

jackgerrits closed this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet