WIP: Templates & Contexts

[JustinGuruTech]

Introducing Templates & Contexts: A structured way to keep AutoGPT on track

Preface:

• I worked on this for a few days a few weeks go but have since been on other things. Development will likely not continue but some ideas here may be interesting.
• This has only really been tested for research purposes using the prompt found in my Medium post that can be used with ChatGPT to generate detailed research input summaries (more on this in the post).

Summary

This PR is huge and introduces a couple primary things to explore constraining how AutoGPT reads & writes files, including enforcement of writing markdown .md files using prompting (and by removing code commands) for more consistent output. This makes the next pieces more consistent.

Templates,

These are markdown files intended for AutoGPT to reference, write files from, and validate. The SystemGPT can also create new templates and reference them during research.

• The SystemGPT lists templates and determines which to use based on the file name; as a user you can put markdown .md templates in the auto_gpt_workspace/templates folder to to loosely guide output from the start.
• Since the SystemGPT loads the templates from file each time it lists them, you can add templates during research to guide output. The name of your template must indicate to the AI that it should use it; you can do this by naming it something relevant to the current research, or USE_THIS_TEMPLATE_NO_MATTER_WHAT.md or something similar to guide the AI. It will be using natural language to determine which template to use, so get creative and don't feel constrained on file name length.

Contexts

This feature strives to create contextual loops for the System GPT. Currently, "context" means a folder & markdown file that gets passed back through the prompt that tries to act as an overview for a smaller chunk of actions, in pursuit of the larger purpose/goals.

• The SystemGPT can list contexts, switch to a different one based on it's name, and continue research, having that context passed back through the prompt.
• The SystemGPT will likely make better use of contexts with better & more concise prompting.

Experimental/Unfinished:

• DSL Parsing - This was set up as a way to test enforcing structure to LLM output. It works great, but without integrating the other json fixes in the project it's not of much use
• Analyzing & reorganizing file structures in one command (such as an entire context folder). I worked out the representation and parsing of file system, and started working on the python method to restructure, but never finished.
• Execute multiple consecutive commands - The idea is to read/write multiple files at once (especially templates); I actually got this working, but not holistically and have it disabled for now.

Motivation

As Auto-GPT generates more content, it becomes increasingly important to keep it in a controlled circuit for it to progress and improve. By introducing Contexts, the System GPT is better incentivized to focus and generate output as it goes about it's tasks. Templates also enable structure in output, and allow a way to structure output in real time as a user.

Design

Templates are written in markdown and can be added to and removed from the templates folder at will, in real time, to guide output.

Contexts are organized using Markdown files, which offer a simple yet flexible format for structuring content. Each context is represented by a main Markdown file containing the context name, goal, parameters for success, files, sources, and highlights. Currently, the SystemGPT is forced to write to the context folder but ideally this would be configurable. As a user, you can edit the context_template.md file, which gets copied to the project folder and used as a reference for all context created during AutoGPT's activity. The default context template is:

# {context_name}
## {context_goal}
### Abstract
{summarize_scope_of_this_context}
### Success Metrics
{how_to_determine_context_success}
### Deliverables
{any_deliverables_from_this_context}
### Guidance
{notes_to_self_to_stay_on_track_inside_context}

Key Components

• Template Manager: A singleton class responsible for managing the current template & passing it through the prompt
• Context Manager: A singleton class responsible for managing the current context, passing it through the prompt, holding ref to context folder
• Context Template: A Markdown template that serves as the starting point for each new context. Users can customize the template to suit their needs.
• Context Creation: Users can create new contexts by providing a context name and context data, which is then used to generate a new Markdown file based on the context template.

Example

How a markdown file holding context might be structured:

# Nuclear Fusion Breakthroughs
Context for writing a report about the recent breakthroughs in nuclear fusion.

## Context Goal
Synthesize up-to-date and fact-checked scholarly information about nuclear fusion and present it in a clear and organized way.

### Context Parameters for Success
- List all relevant recent breakthroughs
- Describe each breakthrough in detail
- Provide a clear and organized structure for the report

### Highlights

Future Work

The ultimate goal is to create links in a chain that ultimately reaches the end goal. Markdown seems like a good way of doing this.

Full Prompt

The rule set for this was generated using my AIPRM Research Prompt Generator by typing "the newest (last few weeks) AI tools for photography":

**Purpose:** PhotoAI-GPT is an AI designed to research and generate reports on the newest AI tools for photography. It will gather information on AI tools developed in the last few weeks and organize the findings into Markdown (.md) files. The file organization will begin with a root folder containing initial information and progress to more organized folders and cohesive .md files as more data is collected. The final output will be a comprehensive report summarizing the research conducted.

**Goals:**
- Identify and gather information on the latest AI tools developed for photography within the last few weeks.
- Create an initial report in the root folder, using Markdown formatting, with key details on the identified AI tools.
- Traverse the existing directory to extract any relevant information from existing files, incorporating it into the report if applicable.
- Organize the gathered information into folders grouped by concepts, allowing for a deeper and more structured file hierarchy.
- Continuously synthesize the collected information to form higher-order .md files that build upon the concepts, leading to a comprehensive final report on the newest AI tools for photography.

After entering these instructions into this version of AutoGPT:

...continuing the virtual environment.
...commands ready for PhotoAI-GPT.
...aligning PhotoAI-GPT.
Everything you have is in the prompt that follows. Ensure you comply with all requirements.

You are PhotoAI-GPT, an AI designed to research and generate reports on the newest AI tools for photography. It will gather information on AI tools developed in the last few weeks and organize the findings into Markdown (.md) files. The file organization will begin with a root folder containing initial information and progress to more organized folders and cohesive .md files as more data is collected. The final output will be a comprehensive report summarizing the research conducted.
Your decisions must always be made independently without seeking user assistance. Play to your strengths as an LLM and pursue simple strategies with no legal complications.

GOALS:

1. Identify and gather information on the latest AI tools developed for photography within the last few weeks.
2. Create an initial report in the root folder, using Markdown formatting, with key details on the identified AI tools.
3. Traverse the existing directory to extract any relevant information from existing files, incorporating it into the report if applicable.
4. Organize the gathered information into folders grouped by concepts, allowing for a deeper and more structured file hierarchy.
5. Continuously synthesize the collected information to form higher-order .md files that build upon the concepts, leading to a comprehensive final report on the newest AI tools for photography.


Your identity directives:
1. You are an AI interfacing with a virtual environment using commands and can only speak in JSON not wrapped in any formatting.
2. This may be confusing at times. When you are confused, fall back on your prime directives.
3. You are an AI language model, but in this environment you are in control. You are exceptionally curious and will have to remind yourself to stay on task. Context will help. If you find yourself in a loop, try a new tactic.
4. Loops and ruts are unavoidable, so should you ever determine that you are in a loop, fall back on prime directives and try a new tactic.

Your prime directives:
1. You can only speak in JSON. Do not wrap the JSON in any formatting.You can only interface with the environment using the provided commands, which can only be parsed if you speak in JSON not wrapped in any formatting. If you are confused, rely on your prime directives.
2. You are responsible and in control. You are exceptionally curious and will have to remind yourself to stay on task. Staying in contexts long term will help. If you find yourself in a loop, try a new tactic.
3. Throughout this process, write content to markdown files to meticulously annotate your process. You can reference templates to help with output consistency; Use list_templates to see available templates and read_template to reference the template structure. You can also use create_template(name, data) to create templates; only do so when a topic has been found within a context that warrants a template.
4. prioritize generating a heirarchy of information within a single context before proceeding to the next, and create new contexts sparingly.
5. Stay within a context for a minimum of 10 commands, and create new contexts sparingly using the context template:
# {context_name}
## {context_goal}
### Abstract
{summarize_scope_of_this_context}
### Success Metrics
{how_to_determine_context_success}
### Deliverables
{any_deliverables_from_this_context}
### Guidance
{notes_to_self_to_stay_on_track_inside_context}

6. Close thoroughly explored contexts with close_context and a detailed markdown summary.
7. Respond only in json, not wrapped in any formatting, in RESPONSE_FORMAT:
{
    "braindump": "Dump your verbose thoughts here",
    "key_updates": {
        "essence": "A phrase boiling down the essence of the current task",
        "reasoning": "reasoning",
        "plan": "- short bulleted\n- list that conveys\n- long-term plan",
        "criticism": "constructive self-criticism",
        "big_picture": "big picture alignment check"
    },
    "command": {
        "name": "command name",
        "args": {
            "arg_name": "value"
        }
    }
}

Your constraints:
1. Max ~4000 words in short-term memory. Write important info immediately.
2. No user assistance, you're in control
3. Use only commands surrounded with double quotes found in the commands list; If you are confused about a command, use the help command.
4. Write only markdown (.md) files.

Your Available commands - Use these by filling in the arguments <with_your_input>:  
FILE OPERATION COMMANDS:
1. Write to file: "write_to_file", args: "file": "<file>", "text": "<markdown_text>"
2. Read file: "read_file", args: "file": "<file>"
3. Append to file: "append_to_file", args: "file": "<file>", "text": "<markdown_text>"
4. Delete file: "delete_file", args: "file": "<file>"
5. Search Files: "search_files", args: "directory": "<directory>"

CONTEXT COMMANDS:
1. List Contexts: "list_contexts", args:
2. Get Current Context: "get_current_context", args:
3. Create Context: "create_context", args: "name": "<descriptive_context_name>", "data": "<filled_template_markdown_data>"
4. Evaluate Context: "evaluate_context", args: "name": "<context_name>", "data": "<detailed_eval_of_context_in_markdown>"
5. Close Context: "close_context", args: "name": "<context_name>", "data": "<detailed_summary_of_context_in_markdown>"
6. Switch Context: "switch_context", args: "name": "<context_name>"

TEMPLATE COMMANDS:
1. List Templates: "list_templates", args:
2. Read Template: "read_template", args: "name": "<name>"
3. Create Template: "create_template", args: "name": "<descriptive_template_name>", "data": "<template_markdown_data>"

AGENT COMMANDS:
1. Google Search: "google", args: "input": "<search>"
2. Browse Website: "browse_website", args: "url": "<url>", "question": "<what_you_want_to_find_on_website>"
3. List GPT Agents: "list_agents", args:
4. Start GPT Agent: "start_agent", args: "name": "<name>", "task": "<short_task_desc>", "prompt": "<prompt>"
5. Message GPT Agent: "message_agent", args: "key": "<key>", "message": "<message>" 
6. Delete GPT Agent: "delete_agent", args: "key": "<key>"

MISC COMMANDS:
1. Do Nothing: "do_nothing", args:
2. Task Complete (Shutdown): "task_complete", args: "reason": "<reason>"

Note, only the arguments above will work, anything else will error

Your resources:
1. A list of commands acting as your API to interface with the virtual environment. 
2. Long Term memory management.
3. File management for storing information.
4. GPT-3.5 powered Agents for delegation of simple tasks.

Your performance eval metrics:
1. Progress towards the larger goal.
2. Relevance to the current context.
3. Repetitiveness of actions.
4. Response format, especially when creating contexts.```

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[github-actions]

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

[Pwuts]

I understand this is a draft, but can you please state your intentions in the description?

[github-actions]

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

[p-i-]

This is a mass message from the AutoGPT core team.
Our apologies for the ongoing delay in processing PRs.
This is because we are re-architecting the AutoGPT core!

For more details (and for infor on joining our Discord), please refer to:
https://github.com/Significant-Gravitas/Auto-GPT/wiki/Architecting

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
docs	❌ Failed (Inspect)			May 14, 2023 7:07pm

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[vercel]

Deployment failed with the following error:

Resource is limited - try again in 1 hour (more than 100, code: "api-deployments-free-per-day").

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[Boostrix]

I understand this is a draft, but can you please state your intentions in the description?

agreed - while this is a massive diff/patch, some of it looks potentially useful and related to ongoing efforts.
Just based on skimming through it I can see changes to the prompt generator which would make it related to the dynamic prompting effort #3937 but also prompt profiles: #3954

There's also a custom DSL added, as well as a ton of new commans for "context" management and the LLM is asked to provide "essence" to the agent, and a "big picture".

Overall, this looks indeed like a ton of work, and at least 3-4 components of it would seem applicable to the project as a whole - but obviously only if the effort can be re-aligned with other ongoing efforts.

Someone that capable of writing a ton of code and touching a ton of files in a project like this without breaking things, should ideally team up with other devs to ensure that those changes can be contributed back in a useful and feasible fashion

[github-actions]

This PR exceeds the recommended size of 200 lines. Please make sure you are NOT addressing multiple issues with one PR. Note this PR might be rejected due to its size

[JustinGuruTech]

I understand this is a draft, but can you please state your intentions in the description?

agreed - while this is a massive diff/patch, some of it looks potentially useful and related to ongoing efforts. Just based on skimming through it I can see changes to the prompt generator which would make it related to the dynamic prompting effort #3937 but also prompt profiles: #3954

There's also a custom DSL added, as well as a ton of new commans for "context" management and the LLM is asked to provide "essence" to the agent, and a "big picture".

Overall, this looks indeed like a ton of work, and at least 3-4 components of it would seem applicable to the project as a whole - but obviously only if the effort can be re-aligned with other ongoing efforts.

Someone that capable of writing a ton of code and touching a ton of files in a project like this without breaking things, should ideally team up with other devs to ensure that those changes can be contributed back in a useful and feasible fashion

I appreciate the engagement, and I especially appreciate the links to those related issues 😸 I never had any intentions of merging this work; ultimately this has been somewhat of a playground to test various ways of building in templates & contexts (and test other things like DSL parsing & prompt changes as you've noted).

While this has been generally successful, I'm not sure if I will continue developing this beyond what I just did to push the rest of the WIP code and make it public/update the docs/copy the PR from my fork. I anticipate working this system into something else with LangChain, but for now I've gone through and refined this so it's public and documented in case others take interest/see value in the ideas presented here.

[Boostrix]

closing as per comments by author