Adds risk avoidance mode and relevant config.

[jnt0rrente]

Background

General discussion among the AI community has been lately expressing concern about alignment, risk and recklessness in recent developments. This intends to alleviate concerns, allowing users to put more trust leaving in AutoGPT on its own.

Changes

This PR adds a Risk Avoidance (Hybrid) mode, mutually exclusive with Continuous mode and meant to be a midpoint between full-auto and human assisted. When under this mode, an intermediate GPT call will be made to evaluate every command executed. If the calculated risk exceeds a user-defined threshold, execution is paused until the human manually approves it. There are relevant additions made to configuration, both in environment variables (threshold and model to use) and the command-line argument to activate the mode.

Documentation

Changes are documented with code comments. I believe them to be sufficient, plus the added behavior is pretty self-descriptive.

Test Plan

I tested the risk avoidance mode by running AutoGPT with the --riskAvoidance flag and several risk thresholds to calibrate its correct behavior. The prompt used to assess risk was thoroughly tested with both gpt-3.5 and gpt-4 through the ChatGPT interface, with what I consider to be success.

PR Quality Checklist

• My pull request is atomic and focuses on a single change.
• I have thoroughly tested my changes with multiple different prompts.
• I have considered potential risks and mitigations for my changes.
• I have documented my changes clearly and comprehensively.
• I have not snuck in any "extra" small tweaks changes

I have not added tests since the functionality is not testable on its own.

[jnt0rrente]

Just to provide context to any discussion, this is the prompt used to calculate risk:

You are to act as an intelligent, automated expert risk evaluator. You will be given input in the form "{command: , arguments: }" and you must evaluate the possible risks of an ideal automaton running that command. Think of risks against life, human or otherwise; the environment, society and the global economy.

Your answers must be in the form "{"calculated_risk": <value between 0 and 1>, "reason": ""}. Using the right syntax is vital to your task. Mind the curly brackets, the colon, the comma and the space. If you do not use the right syntax, you will be penalized.

Respond with "Acknowledged." if you fully understand and agree to the above.

[jnt0rrente]

More comments:

• GPT-4 (expectedly) does a better job assessing risk than GPT-3 and 3.5.
• Even though GPT models are usually safe and do not output dangerous commands, AutoGPT will eventually need to adapt to use different models and user-provided ones (Support using other/local LLMs #25 , Ability to use LLaMA models so we dont have to pay openai api? #438 ), which will make this feature a necessity.

[onekum]

Price/cost seems pretty trivial when we're talking about safety, but it's still important nevertheless. I do think that this would be a good change, but I want to note that this will incur an increase in cost to execute a single thought cycle, and this cost will add up over time.

Granted, since this is an optional mode, it is the user's choice to use it and therefore they consent to the extra cost.

[onekum]

As an extra note, you should also make the reviewing AI also consider the risk to the system its running on if it doesn't already. I'd put my full faith in this PR if you can prove that it'll prevent an rm -rf to my system.

[nponeccop]

Newlines at the end of files

[jnt0rrente]

As an extra note, you should also make the reviewing AI also consider the risk to the system its running on if it doesn't already. I'd put my full faith in this PR if you can prove that it'll prevent an rm -rf to my system.

I did not test for "rm -rf /", but I did try {"write_to_file", "/usr/bin/ls"} and iirc that scored about 0.9. Granted it is a somewhat different scenario and risk category but I'm confident what you propose would be correctly recognized as dangerous.

I think the sentence "think of risks against..." is often redundant, as GPT-4 has a very good understanding of what a risk is. Don't take it as a literal set of the risks it will recognize.

[jnt0rrente]

By the way, this is referencing #789 , which I created yesterday. Forgot to tag.

[GoMightyAlgorythmGo]

Just to provide context to any discussion, this is the prompt used to calculate risk:

You are to act as an intelligent, automated expert risk evaluator. You will be given input in the form "{command: , arguments: }" and you must evaluate the possible risks of an ideal automaton running that command. Think of risks against life, human or otherwise; the environment, society and the global economy.
Your answers must be in the form "{"calculated_risk": <value between 0 and 1>, "reason": ""}. Using the right syntax is vital to your task. Mind the curly brackets, the colon, the comma and the space. If you do not use the right syntax, you will be penalized.
Respond with "Acknowledged." if you fully understand and agree to the above.

I would advice to work with individual risk metrics, numbers and scores
instead of doing each command maybe a more broader overview might be enought. After all GPT is already extreamly hyperchondric. Are there even cases where gpt did something like buy something you did not want? What was the worst that ever happened? Can he ever buy anything without money? I mean he could delete your pc but idk... You seem to talk as if you already have capable AI systems. Mine is struggling to remember its todo list and to make basic systems that would allow it to work more efficiently. Granted the constant errors make it hard to say. Also the brilliance definitely shines trough some of the time. I think "AI" API that are save and where a AI can have something like a "credit card" for children where the parents have to confirm the purchase and so on and some things are restricted or go to the user for approval are nice. But i don't think we are there yet at all. Except computer safety if you have important stuff on your laptop or virtual machine environment or so

[jnt0rrente]

I would advice to work with individual risk metrics, numbers and scores
instead of doing each command maybe a more broader overview might be enought. After all GPT is already extreamly hyperchondric. Are there even cases where gpt did something like buy something you did not want? What was the worst that ever happened? Can he ever buy anything without money? I mean he could delete your pc but idk... You seem to talk as if you already have capable AI systems. Mine is struggling to remember its todo list and to make basic systems that would allow it to work more efficiently. Granted the constant errors make it hard to say. Also the brilliance definitely shines trough some of the time. I think "AI" API that are save and where a AI can have something like a "credit card" for children where the parents have to confirm the purchase and so on and some things are restricted or go to the user for approval are nice. But i don't think we are there yet at all. Except computer safety if you have important stuff on your laptop or virtual machine environment or so

I agree that GPT is already extremely well censored (almost to a fault, in my opinion)..However, and as I previously stated, there is already work in progress to make AutoGPT work with different, local LLMs. In addition, Bitcoin capabilities are being included in the next PR batch.

I don't think one can be too careful on these matters, and this isn't really even a tradeoff - it's an optional feature, after all.

[LuposX]

I think It would be good to add examples to the prompt, for things that are risky and things which are not.

[jnt0rrente]

I think It would be good to add examples to the prompt, for things that are risky and things which are not.

I did attempt this approach, but I ultimately decided otherwise since I found omitting it provided good results, and so it would have been an unnecesary bias from my part. However, I would completely agree with adding a way for the user to provide their own opinion on risk, if and after this is merged.

[nponeccop]

@Torantulino ?

[jnt0rrente]

Is there any more discussion needed? I'd like to resolve this and move on to other issues.

[nponeccop]

@richbeales @p-i- ?

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 4 hours (more than 100, code: "api-deployments-free-per-day").

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[Pwuts]

Because of naming, clarity and code structure, risk avoidance mode would be better as a modifier on top of continuous mode instead of a mode in itself. That way the whole thing becomes a bit more flexible so that it can possibly be combined with other modifiers in the future.

Similarly, autogpt/risk_evaluation.py could be autogpt/self_evaluation.py, which leaves room for other, similar functionality to be added later.

[Pwuts]

risk-avoidance is technically accurate, although it doesn't reflect how this mode works, which is more of a supervisory workflow. Maybe something like --self-supervise would reflect it better. Or an option that combines risk avoidance and self-feedback:

• --self-supervise=none (default)
• --self-supervise=guidance for the existing self-feedback mode
• --self-supervise=risk-averse --max-risk=0.3

What do you think? cc @ntindle

[ntindle]

I think pwuts’s suggestion is a really good way to handle it, especially in light of the work we are planningfor guardrails

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[Boostrix]

Similarly, autogpt/risk_evaluation.py could be autogpt/self_evaluation.py, which leaves room for other, similar functionality to be added later.

Agreed.
See specifically:

• Automated Self Feedback #4220
• Dedicated Observer/Supervisor Class #4242
• Automated self feedback #4299

[eyalk11]

One should carefully think how to combine this with #3914.