WebHost: Add robots.txt to WebHost

[LegendaryLinux]

What is this fixing or adding?

This adds a robots.txt file to the WebHost, which will prevent any bot which respects the file from including the site in search results. The main site, https://archipelago.gg should set a config value in config.yaml to disable allow itself to be indexed.

I also added an ASSET_RIGHTS = false entry to config.yaml. If false, /robots.txt will be served. The app.config value defaults to false if the file is not present.

How was this tested?

I ran the WebHost locally and accessed the file at localhost://robots.txt

[mpql]

I would like to also add an option in a config file to exclude robots.py from being loaded

It might then be reasonable to just have robots.py check if the host is archipelago.gg and in such a case return e.g.:

return app.send_static_file('robots_archipelago.gg.txt')

[LegendaryLinux]

How do you propose testing that outside of the production environment?

[mpql]

How do you propose testing that outside of the production environment?

Not sure what your current tests look like or which file in test/webhost/ would be responsible for this (feel free to point me there and I can write something up), but assuming TDD, I'd think it'd roughly be:

• check that the contents of @app.route('/robots.txt') match the contents of robots.txt
• mock the host to archipelago.gg, then check that the contents of @app.route('/robots.txt') match the contents of robots_archipelago.gg.txt

[LegendaryLinux]

I went with the config file route. We have a preexisting config.yaml which controls WebHost behavior already, so I just used that.

[mpql]

Gotcha, makes sense!

[Berserker66]

I don't like controlling the behaviour via import, someone importing the module later and breaking this seems like a likely future bug.

[Berserker66]

within the function, nothing of this is needed, just app.config["ASSET_RIGHTS"] suffices

[Berserker66]

Suggested change

	configpath = os.path.abspath("config.yaml")
	if not os.path.exists(configpath):
	configpath = os.path.abspath(Utils.user_path('config.yaml'))
	if os.path.exists(configpath) and not app.config["TESTING"]:
	import yaml
	app.config.from_file(configpath, yaml.safe_load)

[powerlord]

While this is a great idea, Google's own developer documentation says that it will still index pages blocked by robots.txt if other pages link to it:

If your web page is blocked with a robots.txt file, its URL can still appear in search results, but the search result will not have a description.

While Google won't crawl or index the content blocked by a robots.txt file, we might still find and index a disallowed URL if it is linked from other places on the web.

They only block it completely if the page has a meta noindex tag, like <meta name="robots" content="noindex">... and they won't even look for a meta noindex tag if the page is already covered by robots.txt.

[mpql]

Good catch @powerlord; I didn't know Google did that.

Assuming y'all still want to differentiate, it might do to just have that config option show a banner, on by default, so satellite sites will say something at the top like:

Note: You're viewing an Archipelago instance. You can find the main Archipelago site at archipelago.gg.