Skip to content

Change robots to deny by default with allowlist #10292

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aduth merged 6 commits into from
Mar 25, 2024
Merged

Change robots to deny by default with allowlist #10292

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
ea0ad3f
Change robots to deny by default with allowlist
aduth Mar 22, 2024
77e7848
Remove redundant robots meta tag
aduth Mar 22, 2024
e594775
Update specs
aduth Mar 22, 2024
3ee1ae8
Dynamically generate robots.txt
aduth Mar 22, 2024
13394eb
Remove unused disallow_all_web_crawlers config
aduth Mar 25, 2024
2b15d49
Add specs for RobotsController
aduth Mar 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 0 additions & 4 deletions app/views/layouts/base.html.erb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,6 @@
<meta content="<%= yield(:meta_refresh) %>" http-equiv="refresh" />
<% end %>

<% if session_with_trust? || FeatureManagement.disallow_all_web_crawlers? %>
Comment thread
aduth marked this conversation as resolved.
<meta content="noindex,nofollow" name="robots" />
<% end %>

<title><%= title %> | <%= APP_NAME %></title>

<%= javascript_tag(nonce: true) do %>
Expand Down
14 changes: 10 additions & 4 deletions public/robots.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
User-agent: *
Disallow: /users/password/edit
Disallow: /sign_up/enter_password
Disallow: /sign_up/email/confirm
Disallow: /api/saml/
Disallow: /
Allow: /$
Allow: /es$
Allow: /fr$

@mitchellhenke mitchellhenke Mar 22, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to try to do this programmatically with https://github.com/18F/identity-idp/blob/4087aba3d76b0607fdf7f441781ff581d203ba0b/lib/idp/constants.rb#L3?

@aduth aduth Mar 22, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, you're probably right, we could do this with a custom route + controller and it'd probably be better, and it would also let us use URL route helpers as well.

@zachmargolis zachmargolis Mar 22, 2024

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you suggesting making that a live file served by a Rails controller, or an ERB that we write out to public/ as a build step?

@aduth aduth Mar 22, 2024

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are you suggesting making that a live file served by a Rails controller, or an ERB that we write out to public/ as a build step?

I was thinking a controller, and implemented a first pass in 3ee1ae8, though I do like the idea of a static file since it wouldn't be something we expect to change. Then again, we probably don't get much traffic to this so hopefully it's not a big deal either way?

Allow: /sign_up/enter_email$
Allow: /es/sign_up/enter_email$
Allow: /fr/sign_up/enter_email$
Allow: /users/password/new$
Allow: /es/users/password/new$
Allow: /fr/users/password/new$
7 changes: 0 additions & 7 deletions spec/controllers/sign_up/passwords_controller_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -169,13 +169,6 @@

describe '#new' do
render_views
it 'instructs crawlers to not index this page' do
token = 'foo token'
create(:user, :unconfirmed, confirmation_token: token)
get :new, params: { confirmation_token: token }

expect(response.body).to match('<meta content="noindex,nofollow" name="robots" />')
end

it 'rejects when confirmation_token is invalid' do
invalid_confirmation_sent_at =
Expand Down
3 changes: 1 addition & 2 deletions spec/controllers/users/reset_passwords_controller_spec.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -141,7 +141,7 @@
allow(user).to receive(:email_addresses).and_return([email_address])
end

it 'displays the form to enter a new password and disallows indexing' do
it 'displays the form to enter a new password' do
expect(email_address).to receive(:email).twice

forbidden = instance_double(ForbiddenPasswords)
Expand All @@ -156,7 +156,6 @@

expect(response).to render_template :edit
expect(flash.keys).to be_empty
expect(response.body).to match('<meta content="noindex,nofollow" name="robots" />')
end
end
end
Expand Down