Add option to install plugin dependencies #3068

dychen · 2023-04-24T00:27:00Z

Background

Currently, it is impossible to run plugins that require third party dependencies. For instance, the Twitter plugin requires tweepy and pandas. Running Auto-GPT with this plugin errors out.

Reference thread: #757 (comment)

@ntindle

Changes

Updated the base CLI to optionally accept an install-plugin-deps parameter which installs plugin dependencies during setup. Temporarily extracts the top level requirements.txt file and runs pip install on it. Does not recursively search for requirements.txt files as there might be unintended behavior. Does this through a system call, which is probably not the best implementation in the long run.

Includes a new install-plugin-deps parameter which defaults to false to minimize interaction with existing code.

Documentation

There is some documentation in the code.

Test Plan

N/A

PR Quality Checklist

My pull request is atomic and focuses on a single change.
I have thoroughly tested my changes with multiple different prompts.
I have considered potential risks and mitigations for my changes.
I have documented my changes clearly and comprehensively.
I have not snuck in any "extra" small tweaks changes

dychen · 2023-04-24T00:28:54Z

@ntindle let me know if this is what you had in mind

ntindle · 2023-04-24T00:37:58Z

autogpt/plugins.py

+def install_plugin_dependencies(zip_path: str, debug: bool = False) -> None:
+    """
+    Inspect a zipfile for a requirements file and install if exists. Only
+    searches at top level for now.


Remove “for now”

ntindle · 2023-04-24T00:39:14Z

autogpt/plugins.py

+            basedir = zfile.namelist()[0]
+            basereqs = os.path.join(basedir, 'requirements.txt')
+            extracted = zfile.extract(basereqs)
+            os.system(f"pip install -r {extracted}")


This is a potential pain point. Can you get the running python exec and use it with -m pip …

Would you rather just have this all in check_requirements.py?

576f9a1

If this is better, I can revert the previous changes ^

codecov · 2023-04-24T00:39:26Z

Codecov Report

Patch coverage has no change and project coverage change: -8.53 ⚠️

Comparison is base (dfcbf6e) 49.52% compared to head (976b0f1) 41.00%.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3068      +/-   ##
==========================================
- Coverage   49.52%   41.00%   -8.53%     
==========================================
  Files          64       64              
  Lines        2946     2946              
  Branches      497      497              
==========================================
- Hits         1459     1208     -251     
- Misses       1365     1679     +314     
+ Partials      122       59      -63

see 15 files with indirect coverage changes

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report in Codecov by Sentry.
📢 Do you have feedback about the report comment? Let us know in this issue.

…te docstring

dychen · 2023-04-24T03:24:44Z

@ntindle

I have two options here:

Update plugins.py and the CLI interface. The dependencies are installed when the user runs: python -m autogpt --install-plugin-deps
Update check_requirements.py. The dependencies are installed alongside base project requirements automatically when the user runs: run.sh or run.bat

Let me know your preference. I think 2 might be better as it is minimally invasive.

ntindle · 2023-04-24T03:25:16Z

2 is preferred

ntindle · 2023-04-24T21:43:24Z

Did a few changes get dropped?

ntindle · 2023-04-24T22:11:20Z

I'd like this to exist within the CLI still but I'm going to test and merge this if it passes

dychen · 2023-04-24T22:54:36Z

I'd like this to exist within the CLI still but I'm going to test and merge this if it passes

My bad. Wasn't sure if you wanted it standalone, as part of run.sh/check_requirements.py, or as part of the CLI, so I opted for the least invasive path. I added it back in this commit, so the following should both work:

$ ./run.sh --install-plugin-deps
$ python -m autogpt --install-plugin-deps

Will open up a PR.

ntindle · 2023-04-24T22:55:57Z

Ping me when it’s up :)

* Clean up GitHub Workflows (Significant-Gravitas#3059) * initial cleanup of github workflows * only run pr-label workflow on push to master * move docker ci/release summaries to scripts * add XS label for PR's under 2 lines * draft test job for Docker CI * fix & activate Docker CI test job * add debug step to docker CI * fix Docker CI test container env * Docker CI build matrix * fixup build summaries * fix pipes in summary * optimize Dockerfile for layer caching * more markdown escaping * add gha cache scopes * add Docker CI cache clean workflow * More robust null byte checking * Adjust test_json_parser file (Significant-Gravitas#1935) Co-authored-by: Reinier van der Leer <[email protected]> * Add indirection layer between cli and application start * Agent and agent manager tests (Significant-Gravitas#3116) * Update Python version and benchmark file in benchmark.yml * Refactor main function and imports in cli.py * Update import statement in ai_config.py * Add set_temperature and set_memory_backend methods in config.py * Remove unused import in prompt.py * Add goal oriented tasks workflow * Added agent_utils to create agent * added pytest and vcrpy * added write file cassette * created goal oriented task write file with cassettes to not pay openai tokens * solve conflicts * add ability set azure because github workflow needs it off * solve conflicts in cli.py * black because linter fails * solve conflict * setup github action to v3 Signed-off-by: Merwane Hamadi <[email protected]> * fix conflicts Signed-off-by: Merwane Hamadi <[email protected]> * Plugins: debug line always printed in plugin load * add decorator to tests Signed-off-by: Merwane Hamadi <[email protected]> * move decorator higher up Signed-off-by: Merwane Hamadi <[email protected]> * init * more tests * passing tests * skip gitbranch decorator on ci * decorator skiponci * black * Update tests/utils.py decorator of skipping ci Co-authored-by: Nicholas Tindle <[email protected]> * black * I oopsed the name * black * finally * simple tests for agent and manager * ísort --------- Signed-off-by: Merwane Hamadi <[email protected]> Co-authored-by: Merwane Hamadi <[email protected]> Co-authored-by: Merwane Hamadi <[email protected]> Co-authored-by: Richard Beales <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]> Co-authored-by: BillSchumacher <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]> * anontation fix (Significant-Gravitas#3018) * anontation fix * fix param name and type --------- Co-authored-by: Richard Beales <[email protected]> * Hotfix/global agent manager workaround (Significant-Gravitas#3157) * Add indirection layer to entry point * Get around singleton pattern for AgentManager to fix tests * Remove dead permanent memory module (Significant-Gravitas#3145) * Remove dead permanent memory module * Delete sqlite db that snuck in * Refactor/move singleton out of config module (Significant-Gravitas#3161) * Add option to install plugin dependencies (Significant-Gravitas#3068) Co-authored-by: Nicholas Tindle <[email protected]> * Re-add install-plugin-deps to CLI (Significant-Gravitas#3170) * Update ci.yml (Significant-Gravitas#3179) * Update bulletin to warn about deprication (Significant-Gravitas#3181) * Ensure Fore.RED is followed by Fore.RESET (Significant-Gravitas#3182) This properly resets the terminal, ensuring that the red text is red and the normal text remains unaffected. Co-authored-by: Richard Beales <[email protected]> * Add a little more descriptive installation instruction (Significant-Gravitas#3180) * add Dev Container installation instruction to installation.md * add Dev Container installation instruction to installation.md * Update installation.md --------- Co-authored-by: Richard Beales <[email protected]> * Update installation.md (Significant-Gravitas#3166) * Refactor/decouple logger from global configuration (Significant-Gravitas#3171) * Decouple logging from the global configuration * Configure logging first * Clean up global voice engine creation * Remove class vars from logger * Remove duplicate implementation of --------- Co-authored-by: Richard Beales <[email protected]> * docs fix to image generation (Significant-Gravitas#3186) * Update README.md * Feature: Added Self Feedback (Significant-Gravitas#3013) * Feature: Added Self Feedback * minor fix: complied to flake8 * Add: Self Feedback To Usage.md * Add: role/goal allignment * Added: warning to usage.md * fix: Formatted with black --------- Co-authored-by: Richard Beales <[email protected]> * Update README.md * this changes it so the file from config is used, rather than a hardcoded name that might not exist (Significant-Gravitas#3189) * Test `Agent.create_agent_feedback` (Significant-Gravitas#3209) * remove plugin notice from CONTRIBUTING.md (Significant-Gravitas#3227) * Extract OpenAI API retry handler and unify ADA embeddings calls. (Significant-Gravitas#3191) * Extract retry logic, unify embedding functions * Add some docstrings * Remove embedding creation from API manager * Add test suite for retry handler * Make api manager fixture * Fix typing * Streamline tests * Use explicit API keys when querying openai rather than import time manipulation of the package attributes (Significant-Gravitas#3241) * Fix readme centering (Significant-Gravitas#3243) * Remove import time loading of config from llm_utils (Significant-Gravitas#3245) * adding probably erroneously removed return value from execut_shell, giving 'None' in return always otherise - not ideal (Significant-Gravitas#3212) Co-authored-by: James Collins <[email protected]> * Load .env in package init (Significant-Gravitas#3251) * Fix PR size autolabeler message (Significant-Gravitas#3194) * Fix `get_ada_embedding` return type (Significant-Gravitas#3263) --------- Signed-off-by: Merwane Hamadi <[email protected]> Co-authored-by: Reinier van der Leer <[email protected]> Co-authored-by: James Collins <[email protected]> Co-authored-by: YOUNESS ZEMZGUI <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]> Co-authored-by: Media <[email protected]> Co-authored-by: Merwane Hamadi <[email protected]> Co-authored-by: Merwane Hamadi <[email protected]> Co-authored-by: Richard Beales <[email protected]> Co-authored-by: BillSchumacher <[email protected]> Co-authored-by: Nicholas Tindle <[email protected]> Co-authored-by: Lei Zhang <[email protected]> Co-authored-by: Daniel Chen <[email protected]> Co-authored-by: Deso <[email protected]> Co-authored-by: Lawrence Neal <[email protected]> Co-authored-by: Duong HD <[email protected]> Co-authored-by: ✔️ITtechtor <[email protected]> Co-authored-by: Toran Bruce Richards <[email protected]> Co-authored-by: AbTrax <[email protected]> Co-authored-by: Peter Petermann <[email protected]> Co-authored-by: Peter Svensson <[email protected]> Co-authored-by: Robin Richtsfeld <[email protected]>

Add option to install plugin dependencies

61ebea1

github-actions bot added the size/m label Apr 24, 2023

ntindle reviewed Apr 24, 2023

View reviewed changes

Install plugin dependencies from check_requirements.py

576f9a1

github-actions bot added size/l and removed size/m labels Apr 24, 2023

Address PR comments: install in current python exec instance and upda…

cba8826

…te docstring

ntindle self-assigned this Apr 24, 2023

Revert previous changes that affect plugins.py etc.

d7c62a7

github-actions bot added size/m and removed size/l labels Apr 24, 2023

dychen added 4 commits April 23, 2023 20:46

Merge branch 'master' into master

5f1c90e

Revert all changes and just add a separate script

da8dd62

Clean up extracted files

1b62f5b

Lint

f6fae5a

Merge branch 'master' into master

0cf821a

Merge branch 'master' into master

976b0f1

ntindle approved these changes Apr 24, 2023

View reviewed changes

ntindle merged commit 29284a5 into Significant-Gravitas:master Apr 24, 2023

Boostrix mentioned this pull request May 4, 2023

Separate the Plugin interface into multiple different interfaces for different types of plugins #3652

Closed

15 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add option to install plugin dependencies #3068

Add option to install plugin dependencies #3068

dychen commented Apr 24, 2023 •

edited

Loading

dychen commented Apr 24, 2023

ntindle Apr 24, 2023

ntindle Apr 24, 2023

dychen Apr 24, 2023

dychen Apr 24, 2023

codecov bot commented Apr 24, 2023 •

edited

Loading

dychen commented Apr 24, 2023

ntindle commented Apr 24, 2023

ntindle commented Apr 24, 2023

ntindle commented Apr 24, 2023

dychen commented Apr 24, 2023

ntindle commented Apr 24, 2023

Add option to install plugin dependencies #3068

Add option to install plugin dependencies #3068

Conversation

dychen commented Apr 24, 2023 • edited Loading

Background

Changes

Documentation

Test Plan

PR Quality Checklist

dychen commented Apr 24, 2023

ntindle Apr 24, 2023

Choose a reason for hiding this comment

ntindle Apr 24, 2023

Choose a reason for hiding this comment

dychen Apr 24, 2023

Choose a reason for hiding this comment

dychen Apr 24, 2023

Choose a reason for hiding this comment

codecov bot commented Apr 24, 2023 • edited Loading

Codecov Report

dychen commented Apr 24, 2023

ntindle commented Apr 24, 2023

ntindle commented Apr 24, 2023

ntindle commented Apr 24, 2023

dychen commented Apr 24, 2023

ntindle commented Apr 24, 2023

dychen commented Apr 24, 2023 •

edited

Loading

codecov bot commented Apr 24, 2023 •

edited

Loading