Change robots to deny by default with allowlist

[aduth]

🛠 Summary of changes

Updates robots.txt to disallow all by default, with few exceptions. Also removes redundant robots <meta> tag.

Related Slack discussion: https://gsa-tts.slack.com/archives/C0NGESUN5/p1711054927329839

Why?

• The previous robots.txt was incomplete and allowed crawling beyond what we would expect to be allowed
• Since there are very few pages we expect to be crawlable, it's easier to list them explicitly
• By preventing most all pages from being crawled, it's no longer necessary to also include a meta directive preventing indexing (related resource)

Open questions:

• Trailing slash? Technically both /en/ and /en are valid. We tend to link to the one without the final trailing slash, and it seems reasonable to only want to allowing crawling of one and not both versions of the page.

📜 Testing Plan

1. Visit http://localhost:3000/robots.txt
2. Verify entries match expected crawlable routes

[mitchellhenke]

Do we want to try to do this programmatically with https://github.com/18F/identity-idp/blob/4087aba3d76b0607fdf7f441781ff581d203ba0b/lib/idp/constants.rb#L3?

[aduth]

Yeah, you're probably right, we could do this with a custom route + controller and it'd probably be better, and it would also let us use URL route helpers as well.

[zachmargolis]

are you suggesting making that a live file served by a Rails controller, or an ERB that we write out to public/ as a build step?

[aduth]

are you suggesting making that a live file served by a Rails controller, or an ERB that we write out to public/ as a build step?

I was thinking a controller, and implemented a first pass in 3ee1ae8, though I do like the idea of a static file since it wouldn't be something we expect to change. Then again, we probably don't get much traffic to this so hopefully it's not a big deal either way?

[zachmargolis]

LGTM but where's the robots_controller_spec.rb?

[aduth]

but where's the robots_controller_spec.rb?

I found it hiding behind the "Ready for review" button 😂

Added in 2b15d49