Skip to content

Commit

Permalink
Allow spidering
Browse files Browse the repository at this point in the history
I only put this in two days ago 🤦

I need this to be allowed in order to review structured data in checkers such
as https://search.google.com/structured-data/testing-tool/u/0/ and
https://search.google.com/test/rich-results

This only affects review apps as this robots.txt is not publicly accessible
from production-like environments

We noindex, nofollow all the content pages in Heroku review apps, so I think
we're safe to allow robots to access these pages.
  • Loading branch information
sihugh committed Sep 5, 2019
1 parent e013796 commit 4efa9de
Showing 1 changed file with 2 additions and 2 deletions.
4 changes: 2 additions & 2 deletions public/robots.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-agent: *
Disallow: /
# User-agent: *
# Disallow: /

0 comments on commit 4efa9de

Please sign in to comment.