-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Problems when using mod_md "clustered" using shared storage directory on NFS #292
Comments
In such a setup, with a shared fs, you are asking for trouble when you reload 2 or more cluster nodes at the same time. All reloading instances will try to activate the newly staged certificates and stumble over each other. Now, when I say "stumble" this means error message as the one you see in the log. The activation of the new cert set in I'd recommend to first reload on cluster node and then the others. It does not matter which one. Or you use Some people also live without a shared filesystem and use |
Yeah, I actually already took many measures to spread out reloads in general, just when writing this up, it occured to me that it might be the logrotate job that triggers these ones close to another. I've spread them out now. However, the directories in So maybe there is something that you could probably tweak there after all ;-) I don't really like the solutions where (ab)using |
Looking at my code in this light again, the overall strategy is on a start/reload:
this is all nice on a single host and survives aborted restarts quite well. However on a cluster with a shared file system, several nodes may be working on the same The best approach here, without some cluster-wide locking, is probably to have The only safe way I can think of would require some cluster sync, like holding some lock while processing a staged domain. Which leads us back to There is file locking in the Apache runtime, but I do not know if/how that translates to your NFSv3. Any ideas? |
Would be nice if you could try v2.4.18 with the new |
@moschlar maybe this escaped your notice. Could you test if the new version addresses the restarting in your cluster? |
Hey @icing,
Am Freitag, dem 29.07.2022 um 02:40 -0700 schrieb Stefan Eissing:
@moschlar maybe this escaped your notice. Could you test if the new
version addresses the restarting in your cluster?
Sorry, no, I did see it but there had been many other projects that
needed attention, especially with summer vacation time...
I explicitly left the GitHub notification open to come back at it when
I have the time.
|
Apache 2.4.54 only ships with v2.4.17, but would love to test Q: From the docs I take it the Q: Would/should nodes that were not restarting and thus did not activate the staged certificate, notice that the |
It sounds like it, but I cannot verify.
Yes, that is how it is intended to work.
I do not see a nice way to make that happen. The node already has read If you want a way to detect that your nodes are not all using the same certificate, you might want to asses the |
Thank you for the answers @icing! I'm hoping for a way for nodes to detect and restart themselves purely by observing the shared filesystem, without knowledge or network access to the other nodes. I was thinking that if the node installing a new certificate also touched a world readable empty file somewhere in the shared filesystem (e.g. BTW, is |
We keep seeing problems with the renewal of certificates on our four node Apache httpd "cluster" using mod_md (with ~1000 managed domains) with a shared storage directory on NFSv3.
The node
-01
is configured withso it could be called "primary".
And the other nodes are configured with
I suspect that it happens when one of the secondary nodes first reloads/restarts httpd (I think maybe due to logrotate) before the primary node got a chance to and therefore it somehow gets messed up. (Or it might be that the logrotates run too simultaneously...)
From the Apache error logs regarding the domain https://crt.sh/?q=infosys.informatik.uni-mainz.de as an example:
Node
-01
:Node
-02
:Now to come to the actual issue/request:
This does not show up in the monitoring page at
md-status
- there, all domains are listed and seem fine...Or do you have any other ideas or shall I look for additional clues?
The text was updated successfully, but these errors were encountered: