[Bug?] mod_md loads an (older) static certificate in favour of a (newer) managed certificate [Apache 2.4.54 / mod_md 2.4.17]

[whereisaaron]

Apache 2.4.54 / mod_md 2.4.17

Is this a bug, or have I got the process wrong?

When I configure mod_md with a MDCertificateFile manual certificate and MDRenewMode always I expected mod_md to eventually renew the certificate and then, after a graceful restart, use that new managed certificate. However, in my testing mod_md prefers to load the old static certificate in favour of the new certificate it just issued. This triggers mod_md to enter an endless renewal loop.

I'm attempting to perfect the technique to migrate a domain to mod_md without outage. To achieve this I am employ the following approach.

1. Configure a manual certificate with MDCertificateFile and also set MDRenewMode manual
2. Switch the DNS entry to mod_md, there is no outage and now HTTP01 challenge renewals are possible
3. Update the configuration to set MDRenewMode always to enable automatic renewal when required
4. When the manual certificate becomes due for renewal mod_md issues a new certificate via an HTTP01 challenge
5. Following an MDMessageCmd renewed example.com apache2 performs a graceful restart
6. On restart I expected mod_md/apache2 to load the new automatic certificate instead of the old static one, however this is not what happens, instead mod_md/apache2 kept loading the static certificate, then since the static certificate was due for renewal it would loop to issue another new certificate, restarting, loading the old static certificate, etc.

Configurations and logs for the steps above below.

The MDMessageCmd handler I used is documented in #298

Configuration for (1) a manual certificate with MDCertificateFile and also set MDRenewMode manual

This configuration works fine and I can then switch the DNS record over with zero outage.

<MDomain example.com>
  MDRenewMode manual
  MDCertificateFile /foo/example-com.crt
  MDCertificateKeyFile /foo/example-com.key
</MDomain>

<VirtualHost *:443>
  ServerName example.com
  SSLEngine on
</VirtualHost>

Configuration for (3) set 'MDRenewMode always' to enable automatic renewal when required.

I use MDRenewWindow 90% here to ensure the manual certificate was due for renewal. With this setting the new 90-day certificate would then not be due for renewal for ~9 days.

<MDomain example.com>
  MDRenewMode always
  MDRenewWindow 90%
  MDCertificateFile /foo/example-com.crt
  MDCertificateKeyFile /foo/example-com.key
</MDomain>

<VirtualHost *:443>
  ServerName example.com
  SSLEngine on
</VirtualHost>

Logs for (4..6):

The following log entries after setting MDRenewMode always looks perfect, mod_md sets up a challenge, issues the certificate, signals a graceful restart, then after a restart we see MDMessageCmd installed example.com which suggests the newly issued managed certificate is installed. The mod_md events we see are exactly what you'd expect:

• MDMessageCmd renewing example.com
• MDMessageCmd challenge-setup:http-01:example.com example.com
• MDMessageCmd renewed example.com
• (graceful restart)
• MDMessageCmd installed example.com

However... when I check the endpoint with curl I observe the older manual certificate is still the active one, and when I check md-status is lists the old manual certificate and is busy renewing it again. Checking /etc/apache2/md/domains/example.com the automatically issued certificate is there in pubcert.pem and privkey.pem but it is not the one being loaded by apache2.

[Sun Oct 23 16:21:19.008636 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
Running as root: md_event.sh renewing example.com
Running as root: md_event.sh challenge-setup:http-01:example.com example.com
[Sun Oct 23 16:21:31.571078 2022] [md:notice] [pid 26:tid 139890920719936] AH10059: The Managed Domain example.com has been setup and changes will be activated on next (graceful) server restart.
Running as root: md_event.sh renewed example.com
Domain 'example.com' has been renewed, restarting apache2
[Sun Oct 23 16:21:31.703930 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00493: SIGUSR1 received.  Doing graceful restart
Successful restart of apache2 after renewal of 'example.com'
Running as root: md_event.sh installed example.com

Logs where (6) goes pear-shaped and mod_md loops constantly renewing the manual certificate

Thereafter mod_md enters a loop of the following events.

• MDMessageCmd installed example.com
• MDMessageCmd renewing example.com
• MDMessageCmd renewed example.com
• (graceful restart)
• MDMessageCmd installed example.com
• MDMessageCmd renewing example.com
• MDMessageCmd renewed example.com
• (graceful restart)
• MDMessageCmd installed example.com
• MDMessageCmd renewing example.com
• MDMessageCmd renewed example.com
• (graceful restart)
  ...

Note that I have retries set to MDRetryDelay 300s that avoid more than 5 attempts/hour hitting the Let's Encrypt rate limits. However that didn't work here as mod_md started a fresh renewal job after each restart and so triggered with rate limit within 3.5 minutes.

Note that on these repeat issuing of the certificate there is no MDMessageCmd challenge-setup:http-01:example.com example.com as there was for the first renewal. I guess there challenge is the same and already there?

Raw logs:

Running as root: md_event.sh installed example.com
[Sun Oct 23 16:21:32.215018 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun Oct 23 16:21:32.215072 2022] [core:notice] [pid 14:tid 139890941249408] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Running as root: md_event.sh renewing example.com
[Sun Oct 23 16:21:43.765635 2022] [md:notice] [pid 175:tid 139890920719936] AH10059: The Managed Domain example.com has been setup and changes will be activated on next (graceful) server restart.
Running as root: md_event.sh renewed example.com
Domain 'example.com' has been renewed, restarting apache2
[Sun Oct 23 16:21:43.903387 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00493: SIGUSR1 received.  Doing graceful restart
Successful restart of apache2 after renewal of 'example.com'
Running as root: md_event.sh installed example.com
[Sun Oct 23 16:21:44.320451 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun Oct 23 16:21:44.320506 2022] [core:notice] [pid 14:tid 139890941249408] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Running as root: md_event.sh renewing example.com
[Sun Oct 23 16:21:54.975044 2022] [md:notice] [pid 309:tid 139890920719936] AH10059: The Managed Domain example.com has been setup and changes will be activated on next (graceful) server restart.
Running as root: md_event.sh renewed example.com
Domain 'example.com' has been renewed, restarting apache2
[Sun Oct 23 16:21:55.124157 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00493: SIGUSR1 received.  Doing graceful restart
Successful restart of apache2 after renewal of 'example.com'
Running as root: md_event.sh installed example.com
[Sun Oct 23 16:21:55.631641 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun Oct 23 16:21:55.631693 2022] [core:notice] [pid 14:tid 139890941249408] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Running as root: md_event.sh renewing example.com
[Sun Oct 23 16:22:05.573364 2022] [md:notice] [pid 444:tid 139890920719936] AH10059: The Managed Domain example.com has been setup and changes will be activated on next (graceful) server restart.
Running as root: md_event.sh renewed example.com
Domain 'example.com' has been renewed, restarting apache2
[Sun Oct 23 16:22:05.722905 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00493: SIGUSR1 received.  Doing graceful restart
Successful restart of apache2 after renewal of 'example.com'
Running as root: md_event.sh installed example.com
[Sun Oct 23 16:22:06.224742 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun Oct 23 16:22:06.224998 2022] [core:notice] [pid 14:tid 139890941249408] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Running as root: md_event.sh renewing example.com
[Sun Oct 23 16:22:17.464737 2022] [md:notice] [pid 578:tid 139890920719936] AH10059: The Managed Domain example.com has been setup and changes will be activated on next (graceful) server restart.
Running as root: md_event.sh renewed example.com
Domain 'example.com' has been renewed, restarting apache2
[Sun Oct 23 16:22:17.602595 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00493: SIGUSR1 received.  Doing graceful restart
Successful restart of apache2 after renewal of 'example.com'
Running as root: md_event.sh installed example.com
[Sun Oct 23 16:22:18.046323 2022] [mpm_event:notice] [pid 14:tid 139890941249408] AH00489: Apache/2.4.54 (Ubuntu) OpenSSL/3.0.2 configured -- resuming normal operations
[Sun Oct 23 16:22:18.046380 2022] [core:notice] [pid 14:tid 139890941249408] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND'
Running as root: md_event.sh renewing example.com
[Sun Oct 23 16:22:20.798260 2022] [md:warn] [pid 711:tid 139890920719936] (70013)Missing parameter for the specified command line option: acme problem urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: example.com, retry after 2022-10-24T11:58:15Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/
[Sun Oct 23 16:22:20.826856 2022] [md:error] [pid 711:tid 139890920719936] (70013)Missing parameter for the specified command line option: AH10056: processing example.com: Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: example.com, retry after 2022-10-24T11:58:15Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/

[whereisaaron]

This documentation section seems most relevant:

https://github.com/icing/mod_md#how-to-migrate-a-https-host

It suggests using SSLCertificateFile and SSLCertificateKeyFile instead of MDCertificateFile and MDCertificateKeyFile. But it sounds like it also doesn't address the situation where apache2 will be automatically restarted after a managed certificate is issued, and may still loop in the same way?

The way the example is worded, you have to disable automatic restarts before the managed certificate is issued, then remove the SSLCertificateFile and SSLCertificateKeyFile before apache2 is restarted. I assume it has the same looping problem if the apache2 restart occurs before the SSLCertificateFile and SSLCertificateKeyFile lines are removed?

I think mod_md should realise that if it has a valid managed certificate, that isn't due for renewal, and that it just installed that very session, that it shouldn't try to renew the manual certificate over and over.

I think there should be MDRenewMode setting or other setting that either prefers a managed certificate over a manual one, or prefers the (unexpired) manual or managed certificate that has the newest valid from date/time.

Suggestion 1: Change (fix?) 'MDRenewMode always' behavior or add 'MDRenewMode force'

I expected MDRenewMode always would prefer a managed certificate once issued (whereas manual / auto would not). But if that is not the desired behaviour, I suggest an MDRenewMode force that will always issue a managed certificate, even if a manual one exists, and will prefer to get apache2 to load a valid managed certificate over the manual one (ie. 'force' the issue and use of a managed certificate as soon as possible).

Suggestion 2: Add MDCertificateLoadOrder setting

If the certificate install and loading logic is entirely divorced from renewal, then another option would be to leave MDRenewMode always unchanged but introduce a MDCertificateLoadOrder that determines which certificate to pass to apache2 on start-up if more than one option is available, e.g.

Current behavior, load the manual cert even if a managed one is available

MDCertificateLoadOrder manual,managed

Optional new behavior, load a managed cert, even is a manual one is configured

MDCertificateLoadOrder managed,manual

MDCertificateLoadOrder would skip the first listed option if that manual/managed certificate does not (yet) exist or has expired.
MDCertificateLoadOrder could have a global default an be used within MDomain/MDomainSet for individual managed domains.