Ban indexing in preview apps

Adds a noindex,nofollow to pages rendered using the application layout in Heroku review apps. Also disallows spiders from crawling this app using the robots.txt. The robots.txt will only be accessible from apps that are not behind router. This means that Heroku preview apps will indicate to crawlers that they should not index the site, but applications in production environments will be unaffected.
alphagov · Sep 3, 2019 · b53d145 · b53d145
1 parent 7493c6b
commit b53d145
Show file tree

Hide file tree

Showing 3 changed files with 8 additions and 2 deletions.
diff --git a/app/views/development/index.html.erb b/app/views/development/index.html.erb
@@ -4,6 +4,7 @@
   <title>
     government-frontend development page
   </title>
+  <meta name="robots" content="noindex, nofollow">
 </head>
 <body>
   <div id="wrapper">

diff --git a/app/views/layouts/application.html.erb b/app/views/layouts/application.html.erb
@@ -8,6 +8,11 @@
       <%= @content_item.page_title %> - GOV.UK
     <% end %>
   </title>
+
+  <% if ENV['HEROKU_APP_NAME'].present? %>
+    <meta name="robots" content="noindex, nofollow">
+  <% end %>
+
   <%= stylesheet_link_tag "application", integrity: true, crossorigin: 'anonymous' %>
   <% if Rails.env.test? && params[:medium] == 'print' %>
     <%= stylesheet_link_tag "print.css", :media => "screen", integrity: true, crossorigin: 'anonymous' %>

diff --git a/public/robots.txt b/public/robots.txt
@@ -1,5 +1,5 @@
 # See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
 #
 # To ban all spiders from the entire site uncomment the next two lines:
-# User-agent: *
-# Disallow: /
+User-agent: *
+Disallow: /