Issue with X-Robots-Tag response header set to none [3.x]:

[asethii]

What happened?

Description

We are running craft 3.7 and all of our pages' response header include x-robots-tag: none. We are trying to find out how to remove this, or set it to all, so our page can get indexed by crawlers.

In general.php, we have disallowRobots => geten('DISALLOW_ROBOTS'). And DISALLOW_ROBOTS is set to false

Please let me know if you need more information

Steps to reproduce

1. Load any front end craft page
2. Observe response header.
3. look for X-Robots-Tag:none

Expected behavior

1. X-Tobots-Tag value should not appear, or be set to "all"

Actual behavior

Header response value is X-Robots-Tag:none

Craft CMS version

3.7

PHP version

7.4.16

Operating system and version

entOS 7

Database type and version

MySQL version 8.0.29

Image driver and version

No response

Installed plugins and versions

• SEO version 3.7.4 (Ether Creative)
  the only customization on this plugin, is under the robots tab, which is
  {# Sitemap URL #}
  Sitemap: {{ url(seo.sitemapName ~ '.xml') }}

User-agent: *
Disallow: /cpresources/

[brandonkelly]

This is almost certainly being added by your Ether SEO plugin.

Make sure you have an ENVIRONMENT environment variable defined, and set to production.

[tgupta-adc]

Our .env file is setting it to production. We have this line added to .env ENVIRONMENT="production". it gets loaded by bootstrap file.

[asethii]

To add to what @tgupta-adc has said, we do have the environment variable defined. We also tried disabling the SEO plugin in our test environment to see if that would resolve the issue, with no luck.

Is there any other information I can give to help troubleshoot this issue?

[brandonkelly]

Is the site cached with Cloudflare, etc., which could be serving old content?

[tgupta-adc]

No, we cleared cache couple of times. Also we have been using Craft for more than 2 years and it has always been set to none.

[brandonkelly]

What happens if you remove the disallowRobots config setting in config/general.php?

[brandonkelly]

DISALLOW_ROBOTS is set to false

Do you mean like this?

DISALLOW_ROBOTS=false

That would just result in it getting set to the string 'false' in Craft 3, as getenv() doesn’t have any special handling for false values.

Which means this condition:

cms/src/web/Application.php

Lines 174 to 182 in d261955

	// Tell bots not to index/follow CP and tokenized pages
	if (
	$generalConfig->disallowRobots ||
	$request->getIsCpRequest() ||
	$request->getToken() !== null ||
	($request->getIsActionRequest() && !($request->getIsLoginRequest() && $request->getIsGet()))
	) {
	$headers->set('X-Robots-Tag', 'none');
	}

would end up passing, because it would effectively be:

if (
    'false' ||
    // ...

Which would equate to true.

You should either not be setting DISALLOW_ROBOTS in .env, or setting it to 0 or an empty string:

DISALLOW_ROBOTS=0

In Craft 4, however, false is supported, but only if you pull in the environment variable via craft\helpers\App::env().

'disallowRobots' => craft\helpers\App::env('DISALLOW_ROBOTS'),

[brandonkelly]

In case that was the culprit, I’ve gone ahead and made Craft more defensive of those values for the next Craft 3 release (b4c6f95).

[tgupta-adc]

@brandonkelly When do we expect this version to be live?

[brandonkelly]

Should go out tomorrow! You can update earlier by changing your craftcms/cms requirement in composer.json to:

dev-develop as 3.8.6

then run composer update.

[brandonkelly]

Craft 3.8.7 is out now. Going to close this, assuming it was in fact the issue.