Vault Service stucks in reloading state

[s01u5]

Environment:

• Vault Version: Vault v1.10.0 (7738ec5d0d6f5bf94a809ee0f6ff0142cfa525a6)
• Operating System/Architecture: Ubuntu 20.04 LTS

Vault Config File:

storage "raft" {
  path = "/opt/vault/raft/"
  node_id = "node_1"
}

listener "tcp" {
  address = "192.168.56.103:8200"
  tls_disable = "true"
}

api_addr = "http://192.168.56.103:8200"
cluster_addr = "http://192.168.56.103:8201"
disable_mlock = true
ui=true

seal "transit" {
  address = "http://192.168.56.102:8200"
  disable_renewal = "false"
  key_name = "autounseal"
  mount_path = "transit/"
  token = "<token-here>"
  tls_skip_verify = "true"
}

log_level = "Debug"

Startup Log Output:
Vault logs of restart sudo systemctl restart vault.service

Expected Behavior:
Vault should successfully reload.

Actual Behavior:
Vault gets stuck in a reloading state.

Steps to Reproduce:

1. Vault logrotate Config

/var/log/vault/audit.log {
  daily
  rotate 7
  notifempty
  missingok
  compress
  delaycompress
  postrotate
  #     /usr/bin/systemctl reload vault 2> /dev/null || true
     /usr/bin/kill -HUP `pidof vault` >> /home/solus/stderr.log 2>&1 || true
  endscript
  create 0600 vault vault
}

2. Run the logrotate manually sudo logrotate -vf /etc/logrotate.d/vault
3. Check the status of vault sudo systemctl status vault.service

Important Factoids:

References:

https://discuss.hashicorp.com/t/vault-stucks-in-reloading-state/38762/7

[heatherezell]

The amazing @maxb thinks the issue may come from here:

vault/command/server.go

Line 1547 in 0be45d0

c.notifySystemd(systemd.SdNotifyReloading)

[maxb]

🙂

Specifically, the above-mentioned code doesn't implement this requirement:

Note that a service that sends this notification must also send a "READY=1" notification when it completed reloading its configuration.

Documented at https://www.freedesktop.org/software/systemd/man/sd_notify.html

[cipherboy]

In particular, down here on line 1629 I think we need a c.notifySystemd(systemd.SdNotifyReady), per its docs.

If someone else wants to open the PR :-)

[jdoss]

This issue is a dupe of #7028

There are two PRs that are open that address this issue. The first one seems a bit more complete.

#9802
#15041

[jdoss]

Actually, I take that back #15041 builds cleanly after a quick rebase on main. #9802 is going to need some love to get it up to speed.

After testing #15041 it does let vault reload but it doesn't seem to work when using system to do a reload-or-restart such as ExecStartPost=systemctl reload-or-restart vault.service

[ncabatoff]

@jdoss #9802 is actually about systemd for agent, not server. Thanks for testing #15041. I'm not sure what we should about the reload-or-restart case, but it seems like support for reload is progress anyway so I'm merging it.

[jdoss]

Thanks @ncabatoff !!