Let's upgrade how http:// external links are a flaw

[peterbe]

Here's how it currently works in Yari

Any and every external URL is always forcibly converted to https:// no matter if that external domain as a valid TLS cert.

<a href="http://hg.mozilla.org/comm-central/">comm-central</a>
<a href="http://visualiser.fr/babylon/crowd">crowd</a>

becomes:

<a class="external" rel="noopener" href="https://hg.mozilla.org/comm-central/">comm-central</a>
<a class="external" rel="noopener" href="https://visualiser.fr/babylon/crowd">crowd</a>

(note the extra s on the href value)

The problem

What happens is that, for example, http://visualiser.fr/babylon/crowd was working but https://visualiser.fr/babylon/crowd isn't.
Now our users are presented with external hyperlinks that don't work but would have worked if we just left them alone.

Our security worries have probably gone too far! And (together with my Yari PR reviewers) I apologize for that for it was my design. Sorry. Yes, we are Mozilla and we really want a secure web, but I think this went a bit too far.

The other problem is that it's hard to know if a URL is going to work if you force it to HTTPS. It's a timely and fragile calculation that we can't conceivably do every single time for every single link in tens of thousands of documents.

Proposed solution

Let's not automatically force all external http:// URLs to https:// anymore. Let's do something a bit smarter.

Last week, I ran a complete build to get a complete list of every possible http:// link and for each and every domain in that list I have used a Python script to painstakingly test if switching to https:// would work.
@hamishwillee and I have been using this script output to slowly roll out mass fixes to the mdn/content repo for the domains we know it will definitely work for. See this search for PRs

My suggestion is to check in the .json file it built into Yari and use this in Yari building and the flaw system.
So if a page has a <a href="http://...."> link, if that domain can't be forced to https:// it will stand out as a flaw.
So that only affects the flaw system (which is exposed in the PR Review Companion).
Alongside with this, we'll stop automatically and forcibly turn all http:// links to https:// at build-time.

[Ryuno-Ki]

Perhaps you could ask @EFForg folks how they came up with a strategy for https://github.com/EFForg/https-everywhere ?
After all, they have a browser extension that checks on upgrade-ability and might ran into some edge cases we don't have on our radar?

[hamishwillee]

So if a page has a <a href="http://...."> link, if that domain can't be forced to https:// it will stand out as a flaw.

Am I getting this correctly? A flaw is reported in the cases that can't be fixed by the user?

I would have thought that the flaw would appear if the domain can be forced but the link is using http. Obviously we'll miss any case we haven't got in our list of "can be forced" domains.

<a class="external" rel="noopener" href="https://hg.mozilla.org/comm-central/">comm-central</a>

Note that this would be the rendered URL but I would expect the in-source URL to be <a href="https://hg.mozilla.org/comm-central/">comm-central</a> - because you can't have those classes in markdown and you can infer them from the fact it is an external URL.

In any case, the idea behind your suggestion seems solid to me.

PS Back on deck tomorrow and will look at any unapproved PRs for me then.

[peterbe]

Let's be more clear. It's only going to be a flaw if the http:// link can with certainty be changed to https://. All other links will be left untouched.

[peterbe]

Hey, now there's a pull request to resolve this: mdn/yari#3657

[peterbe]

I don't see a way to "close" a discussion but as of mdn/yari#3657 we have a solution.
Now, only http:// links that are known to be safe to force to https:// are forced. All other http:// links are left as-is. They're not even logged as flaws. But they do stand out a bit on the PR Review Companion.

If anything, the next action is to keep hunting for http:// and trying to "improve them".