What is mod_md's idea of a fallback cert?

[icing]

From a mail to apache-users mailing list:

I had an interesting dilemma come up. I want to start using mod_md, but needed an answer 
as to what to do if lets encrypt can't auth.

Now, unlike any other certificate solution, mod_md will not block a vhost from starting if no 
cert is defined. This is good. But it places the following in the logs on first run of an MDomain.

[Sun May 09 11:16:02.989759 2021] [ssl:info] [pid 72605] AH01914: Configuring server drivingdemocrats.org:443 for SSL protocol [Sun May 09 11:16:02.989900 2021] [ssl:warn] [pid 72605] AH10085: Init: drivingdemocrats.org:443 will respond with '503 Service Unavailable' for now. There are no SSL certificates configured and no other module contributed any. [Sun May 09 11:16:02.991557 2021] [ssl:info] [pid 72605] AH02568: Certificate and private key drivingdemocrats.org:443:0 configured from /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-cert.pem and /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-privkey.pem [Sun May 09 11:16:02.991980 2021] [ssl:error] [pid 72605] AH02604: Unable to configure certificate drivingdemocrats.org:443:0 for stapling [Sun May 09 11:16:11.090952 2021] [md:notice] [pid 72625] AH10059: The Managed Domain drivingdemocrats.org has been setup and changes will be activated on next (graceful) server restart.

(apachectl graceful)

[Sun May 09 11:16:33.957317 2021] [md:info] [pid 72605] AH10068: drivingdemocrats.org: staged set activated [Sun May 09 11:16:33.958937 2021] [ssl:info] [pid 72605] AH01914: Configuring server drivingdemocrats.org:443 for SSL protocol [Sun May 09 11:16:33.960105 2021] [ssl:info] [pid 72605] AH02568: Certificate and private key drivingdemocrats.org:443:0 configured from /usr/local/etc/apache24/md/domains/drivingdemocrats.org/pubcert.pem and /usr/local/etc/apache24/md/domains/drivingdemocrats.org/privkey.pem

This file doesn't exist either.

Apache seems to have some concept of a "fallback" cert. Something that I could generate, self-signed, with straight openssl, or perhaps use this as a mechanism from moving to a different ssl signing solution (an external acme script, perhaps...or just a classic commercial cert).

This means if DNS isn't pointed right (say, the site is being staged on my server but hasn't 
been re-pointed, the user can still CONNECT, get a certificate warning, and preview their site.)

The problem?

This isn't in the docs AT ALL. The only mention of the word "fallback" in https://httpd.apache.org/docs/trunk/mod/mod_md.html (or http://httpd.apache.org/docs/current/mod/mod_md.html)

is:

"It is recommended that you have virtual hosts for all managed domains and do not rely on 
the global, [fallback] server configuration."

There seems to be no way to configure a global fallback cert, instead of per-domain. 
(That is to say, if I'm going to get a cert warning anyway, I might as well just use a single 
cert for a staging site)

[icing]

Maybe this needs to be added to the README, but let's clarify here first.

The motivation for this discussion seems to be: "What happens when Lets Encrypt stops working (for whatever reason)? How can I protect my site?"

ACME fails for a New Managed Domain (MD)

For any virtual https host (with SSLEnging on), Apache needs a certificate. For this reason, mod_md will on such occasion create a self-signed certificate with all domain names of that MD. This is called a "fallback cert" by mod_md.

While the module is negotiating with Lets Encrypt for a "real" certificate, visitors to this site will see the fallback. Their browser will not trust it, but they may "click through". Once the ACME certificate is there, and the server is reloaded, the fallback is archived and replaced with the new cert.

Should the ACME server fail to give a new certificate, errors will be logged, notifications will be triggered (see MDMessageCmd) and the fallback cert stays in place.

ACME fails for an existing MD

Let's assume the initial setup worked and you got certificates from Lets Encrypt. Then, for some reason, renewals stop working. What will mod_md do?

First, it will continue to use the certificate it has. Even if it already expired. The client browser then will not trust it, but the user may "click through". The module will keep on trying to get you a new cert, using a backoff delay in retries, but do try that at least once per day or more often. It will tell you about this via MDMessageCmd which is highly recommended.

With the default settings, a Lets Encrypt cert lives for 90 days and Apache will renew it 30 days before its lifetime runs out. So, this failure would need to go unfixed for 30 days before your users notice.

You can lower some risks of failures by adding a second ACME server to you config. See how to setup a failover CA in the documentation.

Your own "fallback" cert?

There is no way to configure mod_md with your own fallback certificate. As mentioned above, when no certificate is there at all, it will create a self-signed fallback for you. If a cert exists, it will continue to use that, even when expired.

This seems as good as providing your own self-signed fallback, as in all cases users will get a warning and need to "click through". Now, if you are able to generate a valid fallback certificate at all times, you do not need mod_md in the first place. Then just use your own cert all the time.

[thegushi]

"Click-through" does not happen in all cases, unfortunately.

A warning is not issued if a site has HSTS records present. You simply universally get a "cannot establish a secure connection" error, and not all browsers even indicate it is related to HSTS.

Thus, a cert (wildcard or otherwise), signed by our own "internal" CA which would allow us to continue to use things, until LE did its thing.

It would be useful to have the behavior of how the fallback cert is generated (perhaps just copied from somewhere else on disk?) configurable.

[thegushi]

Second, note my logs entry above, it implies the existence of a fallback entry that is per domain:

configured from /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-cert.pem and /usr/local/etc/apache24/md/domains/drivingdemocrats.org/fallback-privkey.pem

Whereas the File Storage section of the readme only references those being in the main folder, and not the per-domain folders.

In the readme, you say "What is happening? At start up, mod_md generated a self-signed certificate for the new https host to use and switch that host to 503 responses. This made sure that your server started without delay and that your other hosts could start working." but there's no mention made that this is the "Fallback" cert mentioned.

===

And, third, finally, on the main apache site -- NO mention is made of how and when these fallback certs are generated. Maybe that needs a fix with an actual apache bug, but it would be nice to point them to a corrected readme here (i.e. mention the filenames in the readme, and in the File Storage section above.

[icing]

This would not only affect the fallback on a new site, but especially the behaviour when a cert has expired, correct? This would only be possible on a server reload, however, since once a certificate has been loaded, it is fixed for the runtime of the server. Hmm...

As for the documentation, this can always be improved and you are welcome to add a PR for better explaining things. So far, the fallback cert was treated as an initial thing with no peculiar interest to users, since it usually disappears after some minutes.

However, your use case is new. To get the benefits you describe, you need clients with an updated root store, which is outside enterprises generally frowned upon. (If I understand your case correctly.)

I do not doubt that your motivations are valid. However, I'd like to spend my free time more on things benefitting a wider range of users.

[icing]

Just an idea: if you have your internal "CA" anyway, why not give it an ACME server (pebble is quite uncomplicated) and configure that server as fallback for mod_md? That should give you exactly what you want.

[thegushi]

That's a thought -- right now, my initial goal, as well as the one in the email from a couple years ago on apache users is "understand in which cases the fallback cert is used".

Ignoring HSTS (and just so I can suggest better docs), is there a way to configure mod_md to treat a site using the fallback cert as a valid site, instead of the 503?

Also, is there a global fallback cert, as mentioned in the file storage of the README, or is that a typo?

[icing]

Ignoring HSTS (and just so I can suggest better docs), is there a way to configure mod_md to treat a site using the fallback cert as a valid site, instead of the 503?

No. I think your idea of what a "fallback" certificate is and what mod_md calls its initial self-signed are different. We need to untangle that for a meaningful discussion.

Also, is there a global fallback cert, as mentioned in the file storage of the README, or is that a typo?

No, there is not for some considerable time. The documentation was not updated. I just removed the mention. Thanks for pointing out.

[thegushi]

Okay, so:

• The "fallback" (as named by the code) cert really is generated by apache on the first invocation of mod_md for a given domain, and it exists just so apache can start up, since it can't start up with a null cert.
• Even if you click through it, you get a 503, and that's going to happen until a valid cert is in place.
• There's no way to tell the code to use anything else on that first invocation
• and it will be used until an apachectl graceful happens -- that's the only way to advance the "State machine".

Is that a correct understanding?

If so, I guess my only other question is (for documenting this), what params are used to generate that cert (key strength, algorithm, lifetime, subject name?)

My assumption of "fallback" (based purely on that word -- I would have used "initial" or "preliminary" or something) could have been any of:

• A commercial cert you had bought and wanted to use prior to swinging over to full acme.
• An internal wildcard cert
• An internally-issued cert (say, something dropped in place by your provisioning system).
• A cert to use in case LE just failed past the expiry date.

But I see this isn't correct. Hopefully you can understand my confusion, and why I was mystified that it wasn't documented.

(To be clear, I'm not asking for it to be any of those, just explaining my thought process.)

[icing]

Okay, so:

• The "fallback" (as named by the code) cert really is generated by apache on the first invocation of mod_md for a given domain, and it exists just so apache can start up, since it can't start up with a null cert.
• Even if you click through it, you get a 503, and that's going to happen until a valid cert is in place.
• There's no way to tell the code to use anything else on that first invocation
• and it will be used until an apachectl graceful happens -- that's the only way to advance the "State machine".

Is that a correct understanding?

Yes.

If so, I guess my only other question is (for documenting this), what params are used to generate that cert (key strength, algorithm, lifetime, subject name?)

It's all in the code. The generation is in md_crypt.c:1955. Keys are generated according to the MDPrivateKeys specified for the domain.

My assumption of "fallback" (based purely on that word -- I would have used "initial" or "preliminary" or something) could have been any of:

• A commercial cert you had bought and wanted to use prior to swinging over to full acme.
• An internal wildcard cert
• An internally-issued cert (say, something dropped in place by your provisioning system).
• A cert to use in case LE just failed past the expiry date.

But I see this isn't correct. Hopefully you can understand my confusion, and why I was mystified that it wasn't documented.

(To be clear, I'm not asking for it to be any of those, just explaining my thought process.)

Naming is hard. All is fine.