Skip to content

ComputerTool improvements #663

@markb-trustifi

Description

@markb-trustifi

Please read this first

  • Have you read the docs? YES
  • Have you searched for related issues? YES

Describe the feature

Please add the following functionality to the computerTool:

  • Computer Tool misses needsApproval, so it can be approved/rejected only on the Agent level:
const computerAgent = new Agent({tools: [computerTool({ computer })]});
const computerTool = computerAgent.asTool({
   needsApproval: async (runContext, args) => {...}
});

See in executeComputerActions, it misses code something like:

const approvalItem = new RunToolApprovalItem(toolCall, agent, computerTool.name);
const requiresApproval = await computerTool.needsApproval(runContext, toolCall.action, toolCall.callId);
  • Computer Tool doesn't pass runContext to the ComputerBase class.
    The _runComputerActionAndScreenshot should pass it to every computer action, something like:
    computer.click(action.x, action.y, action.button, runContext)
  • The ComputerBase class instance is actually a statefull and doesn't differ between Runner.run executions. It can also lead to the security risks because different Agent loops can be run by different users. To reinitialize it before every runner loop, ComputerBase could have initRun method that will be called on "agent_start" event.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions