Web Navigation Challenge

[waynehamadi]

Duplicates

• I have searched the existing issues

Summary 💡

Auto-GPT should be able to navigate website, enter text and submit forms.

FYI @NotSkynet

Examples 🌈

No response

Motivation 🔦

No response

[anonhostpi]

I would really like to learn more about these challenges you keep coming up with. I'm assuming that these serve as guidelines for writing unit tests?

[waynehamadi]

yes exactly, but they're more end to end tests that unit tests. We test the system, we don't test the code. This way even if in 6 months we do a complete refactorisation the tests will still be valid, and they will allow us to refactor Auto-GPT.
ping me on discord https://discord.gg/autogpt
merwanehamadi

[Boostrix]

Auto-GPT should be able to navigate website, enter text and submit forms.

good thinking:

• detecting if a website is using static HTML or lots of JavaScript would be useful to be able to use a different crawler/spider strategy
• the very first milestone should probably be accepting cookies and privacy TOS :-)
• the next step might be always trying to fetch sitemap.xml to orient itself if it's available
• and to fetch the accessible version of a website to reduce parsing/processing overhead
• being able to detect/use search forms/fields would be enormously useful !
• it should be able to memorize entrypoints for searching and be able to revisit/reuse those as needed whenever information is missing

[anonhostpi]

It's probably best to use selenium awaiters for a portion of that, and prompt the human user to navigate those, as the human would be who is agreeing to them, not the bot.

[Boostrix]

those were based on observing the agent "browse" with the browser instance set to visible - it's not confirming anything, and also isn't using search facilities and also doesn't seem to be using sitemap.xml to find its way (personally, not a huge fan of using selenium for non-dynamic websites, where classic crawling should suffice)

[waynehamadi]

@anonhostpi @Boostrix I like that you think of how to solve this problem.

This issue is more about how to test whether the problem has been solved (aka create the challenge).

@NotSkynet is going to help us find good static website where we can test whether Auto-GPT is able to navigate.

[Boostrix]

just keep the browser open/visible to see for yourself that browsing could use some TLC ...

This issue is more about how to test whether the problem has been solved (aka create the challenge).

re navigation: randomly generate navbar structures using a nested loop with different labels and menus/sub menus, generate a matching sitemap.xml and use the two in combination to see if the agent is able to "visit" a certain part, each link would trigger the same python CGI script to tell the back-end which links were found/clicked.
Evaluation-wise, we will then need to specify a goal "navigate to the contacts/team/about/company page", the crawler/selenium should be able to determine what link that is (we could use random file names here to ensure that the LLM isn't guessing). The Python CGI running inside HTTPServer would then register an even if the link was found or not.
We could probably bootstrap the whole thing by copying a bunch of drupal/wordpress templates into a directory to have actual navigation bars that a python helper would then customize (randomize) with different tiles/descriptions and a href links

re the search form: probably analogous to the click form we talked about yesterday, just with the twist that it's a single SEARCH field (input text) and a simple python based back-end that we can execute via a conventional HTTPServer instance, at which point we can then hook up the whole thing to pytest, as before.

[zachary-kaelan]

We can just use the XPath //form | //input | //textarea | //button | //select | | //a[@href] with maybe a couple other things, to find all the important elements on the page it needs to know about for forms and navigation. Feed it the list of results and their paths, and it should be able to infer what it needs to do to navigate the webpage.

[Boostrix]

Yes, exactly what I suggested here: #3551 (comment)

I believe, this sort of feature could be useful in general, so we could just as well implement a more generic "browse_website" or extend it as needed (with support for xpath, like you say)

If this is augmented with sitemap.xml data, it's probably rather flexible as is:

browse_website <url> <focus> <constraints> [<xpath>, <use_sitemap.xml>]

[zachary-kaelan]

If this is augmented with sitemap.xml data, it's probably rather flexible as is:

browse_website <url> <focus> <constraints> [<xpath>, <use_sitemap.xml>]

Oh yeah, duh. I forgot sitemaps are a thing. We can integrate one of the many, many pre-existing crawling tools there are out there and automatically generate a sitemap for every site visited that doesn't already have one.

I was already thinking about overcomplicating things and indexing the pages on the fly ourselves when I remembered that search engines can do things like site:example.com and inurl:contacts.

[Boostrix]

there is one PR that maps HTTP Request to some custom commands for scraping purposes I believe: #2730
Also, this would seem like a really good idea: #2181

[anonhostpi]

For a collection of issues/PRs/discussions to base the challenge off of you could use my trackers: Gist (Alt)

You are more than welcome to use them as the "static website." recommend using the README.md file. I do update my tracker files frequently, but I imagine you could find a github xpath on it that doesn't change.

[Boostrix]

That gist is really looking good and super useful, must have been a ton of work, thanks for that !

(probably should be added to the wiki in a similar shape or form)

[anonhostpi]

That gist is really looking good and super useful, must have been a ton of work, thanks for that !

Thank you!

(probably should be added to the wiki in a similar shape or form)

I would have added it to the catalyzing and/or moderator page, but I think you have to have repo ownership privileges for that. You can't make a PR against a wiki, unfortunately.

[anonhostpi]

@Boostrix, if you are interested in contributing to my gist, you can take a look at https://github.com/anonhostpi/AUTOGPT.TRACKERS.

I use Github Copilot to autofill a lot of the data, and then use the powershell script https://github.com/anonhostpi/AUTOGPT.TRACKERS/blob/main/.SCRIPTS/CONTRIBUTE.ps1 to push updates to it.

[waynehamadi]

yeah great gist.
Anyone up for the task on this challenge creation ?
@NotSkynet did you find a website we can do that in ?

[Boostrix]

I would say, this could just as well be a static local website - i.e. via a Python HTTP Server that is locally running as part of the test suite, we talked about using that idea in the context of the "contact form".

So basically we need a pytest module that starts up a HTTP server to track "actions" (GET/POST requests).

To the agpt agent it should not be relevant if it's navigating google.com or 127.0.0.1 :-)

[BaseInfinity]

Oh I was able to get this working with ChromeGPT (https://github.com/richardyc/Chrome-GPT) using the AutoGPT agent.

I was able to login to Reddit with pretty basic sentence

Here's my demo:
https://www.youtube.com/watch?v=RkpvyGla0PA

Not only that but I got extensions working so I was able to use the ScribeAI Plugin to create an auto generated guide (though basic)

Edit:
I'm realizing this is talking about testing and not "is this possible", my bad

[Boostrix]

testing would be the level to see if any submissions solve the challenge or not - ideally, we would be able to throw different URLs at the challenge, with goals to navigate to some pre-defined page, and possibly "action" (contact form etc).

For starters/experiments, this could be based on localhost - but eventually, it will need to work using arbitrary websites.
If you are working on this, you should inform @merwanehamadi so that he can update the list of challenges accordingly - to prevent others from working on the same problem, also do state clearly if you'd like to team up with others on the same challenge

[waynehamadi]

@Noots123 thanks for the suggestion !
Anyone knows what to do to create a challenge around this ?
FYI @Boostrix @BaseInfinity @anonhostpi @zachary-kaelan

[waynehamadi]

Mind2Web is a ground breaking news for web navigation agents : https://osu-nlp-group.github.io/Mind2Web/
@xiang-deng, @ysu1989 thank you for your work, hopefully I will look into this soon

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[BaseInfinity]

I would love to get some momentum on this again. Recently I played around with HyperWriteAI and their autonomous browser atm is the best I have played with. Here's a small demo

HyperWriteAI is closed source and ChromeGPT is a bit slow so I think AutoGPT can make a splash here

I mentioned this to @Pwuts and he put it on his radar, he mentioned having a selenium wrapper that could do something similar to ChromeGPT's implementation:
https://discord.com/channels/1092243196446249134/1111659953493651547/1149034954488037469

I don't see why AutoGPT can't have as good browser navigation as HyperWriteAI, and it would be a huge win for the OS community.

The whole reason why I've been going down this rabbit hole is to experiment with generating E2E tests given a "task". I've been able to prototype with ChromeGPT but would prefer a solution inside of the AutoGPT repo.

Plus I feel like one of the best things about autonomous agents is being able to browse the web effectively and is something people want which you can tell by the hype HyperwriteAI got when it released (it was number 4 on Product Hunt that day). General public went nuts over it understandably. Seeing your browser perform a task like a human in front of your eyes is powerful stuff lol

Any ways hopefully this sparks up some convo and momentum in the browser navigation area and if there's any ways I can help push it forward =)

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[BaseInfinity]

bump, unfortunately I haven't had time to dig into this

[ballonJourn]

Beautiful show ! That's my forte @BaseInfinity

[BaseInfinity]

@ballonJourn while I haven't been able to make progress in this repo, I have been able to make progress in others that support AutoGPT.

Here's my latest prototype, PlaywrightGPT:
https://www.youtube.com/watch?v=DH9cIm1qfug

I imagine a future where tests can self repair themselves and even generate code diffs on code change in CI.

[github-actions]

This issue has automatically been marked as stale because it has not had any activity in the last 50 days. You can unstale it by commenting or removing the label. Otherwise, this issue will be closed in 10 days.

[github-actions]

This issue was closed automatically because it has been stale for 10 days with no activity.