Introducing the MDN HTTP Observatory! 🎉

[pransh15]

Hello! 👋

We are delighted to introduce to our community that the HTTP Observatory is now a part of MDN! 🎉

🛡️ What is MDN HTTP Observatory?

The MDN HTTP Observatory performs an in-depth assessment of your site's HTTP headers and key security configurations. It provides developers and website administrators with detailed, actionable feedback, focusing on identifying potential vulnerabilities. With over 47 million scans conducted on 6.9 million websites, our tool is trusted by millions worldwide to deliver top-notch security assessments.

🔍 What does the HTTP Observatory test?

• Secure cookie configuration: Access to cookies should be locked down as much as possible, to prevent attacks such as Cross-site scripting (XSS) and Cross-site request forgery (CSRF).
• Cross-origin Resource Sharing (CORS): Limit cross-origin resource access to no more than necessary.
• Content Security Policy (CSP): Provides fine-grained control over the locations from which resources on a site can be loaded.
• Strict-Transport-Security: Enforces site connections via HTTPS only, to prevent manipulator-in-the-middle (MiTM) attacks.
• Correct redirection behavior to HTTPS on initial loading: Prevents attacks during initial loading.
• Referrer-Policy: Prevents leaking of sensitive information.
• …and many more that you can read in the blog!

At Mozilla, we are committed to a safer and more secure internet. Integrating HTTP Observatory into MDN reflects our dedication to providing the tools and insights needed to protect your site.

🔗 Check your site’s security now: https://developer.mozilla.org/en-US/observatory/
📢 Read the full announcement: https://developer.mozilla.org/en-US/blog/mdn-http-observatory-launch/

Let's work together to make the web a safer place. We look forward to your feedback and questions!

Best,
The MDN Team ❤️

[cysp]

This is amazing, congratulations and well done! 🎊

I had been running scans periodically throughout the beta period without issue but recently I've found that Cloudflare's automated bot detection has started to classify Observatory scans as "definite bot" requests which are triggering our WAF rules that result in Cloudflare presenting a challenge rather than the expected webpage result, which now fails the scan.

Cloudflare have a "verified bots" scheme, where bot operators can register their bots and that can be used a signal for WAF evaluation, avoiding the request being challenged.
As an outsider, I don't have quite enough information to complete the enrolment request form, primarily the "IP Validation" requirements. Would you be open to enrolling in Cloudflare's Verified Bot scheme, in accordance with its policy?
https://developers.cloudflare.com/bots/reference/verified-bots-policy/

I can assist with the enrolment using my organisation's Cloudflare account if desired. 🙂
(This would not claim ownership of the bot, I'm simply offering because Cloudflare's documentation indicates that enrolment via an account will receive a quicker response.)

[argl]

Hi, thanks for reporting this. We have the issue on our list, hopefully we can have a solution soon.

[argl]

Looking further into the issue, I do believe that observatory requests do not generally qualify as bot requests for cloudflare. Scanning shopify.com (which is presumably backed by cloudflare) does succeed, for example. OTOH, Sites having "Super Bot Fight Mode" enabled show the dreaded anti-bot dialog practically every time in any random browser. Depending on your cloudflare plan, it is probably possible to configure bot management more fine-grained?

On bot enrollment, we do not currently meet the some prerequisites (minimum number of requests, ip ranges), and we currently do not have the resources to push that further. That may change in the future though. I would suggest running the observatory scans on the origin servers directly and not go through cloudflare for the time being.

[cysp]

👍 From what I can see in my account, Cloudflare is accurately detecting observatory requests as bot requests now but the observed behaviour will vary depending on the configuration of the particular Cloudflare account and domain. My organisation has specifically configured a waf rule to challenge "definite bot" requests while allowing "verified bot" requests to bypass that challenge, whereas other domains may be configured differently or use different combinations of signals to trigger challenge behaviour. (Note that our configuration is in line with Cloudflare's documentation so I expect we're far from the only organisation with this kind of configuration.)
As you say, it's well within our power to carve out an exception for observatory requests, it would just be far simpler to rely on Cloudflare's verified bot program both in terms of the configuration change and also because the verified bot program's requirements provide a level of safety around that exception (e.g. by requiring that the source ip address can not be used by other, unrelated services that we may not want to allow through our waf)

Thank you for looking into the verified bot program, I certainly understand it's not a trivial checkbox to tick and especially for a service provided gratis for the public good. If circumstances change and if there's anything I can do to help out, please feel free to reach out. 🙂

[ALF-it]

I have a site that when attempting to scan, gives me the error: The site seems to be down. However I have confirmed the site is up, and resolves for the legacy Observatory, SSL Labs etc. I tried both with and without prepending https:// to the site fqdn in the new MDN. The site otherwise has had no problems. Is there something additional I should look for?

[argl]

Hi, without knowing the host in question, I cannot really look into this specifically. It does sound strange though, given that other tools are reporting the site to be ok. Be sure the server listens on the common port numbers and responds with 2xx status code.

[ALF-it]

Apologies for the replies slipping by -- I can provide the host URL
account, but would like to ensure it is not broadly known, is there a
secure way to do this? What is odd is I have a couple more subdomains
configured the same without issue.

[argl]

Hi, you can use the feedback form if you want to provide the URL to us.