Vault is inaccessible if an etcd unit is lost

[gnuoy]

Describe the bug
When vault is deployed in a cluster, using etcd for HA management, losing the first etcd unit in the 'address' list results in vault returning 500s

To Reproduce

• Deploy 3 units of vault. In my test case use 1 mysql unit for storage and 3 units of etcd for ha_storage.
• Initialise and unseal vault and validate API with "vault status"
• Shutdown the etcd unit that corresponds to the first IP in the 'address' list (in the 'ha_storage "etcd" ' section)
• Run "vault status" with VAULT_ADDR pointing at any of the vault units

Error checking leader status: Error making API request.

URL: GET http://10.53.82.200:8200/v1/sys/leader
Code: 500. Errors:

* context deadline exceeded

Expected behavior
I would expect vault to continue to function and respond to API requests, without interruption, in the event of losing an etcd unit.

Environment:

• Vault Server Version: 0.10.3
• Vault Client Version: 0.10.1
• Server OS: Ubuntu 16.04.5 (Xenial) x86_64

Vault server configuration file(s):

api_addr = "http://10.53.82.113:8200"
cluster_addr = "http://10.53.82.113:8201"
disable_mlock = true
storage "mysql" {
  username = "vault"
  password = "kpzLkcjfmX9w45n542zwLyrJBppfg5rP"
  database = "vault"
  address = "10.53.82.226:3306"
}
ha_storage "etcd" {
  ha_enabled = "true"
  address = "https://10.53.82.119:2379,https://10.53.82.150:2379,https://10.53.82.157:2379"
  tls_ca_file = "/var/snap/vault/common/etcd-ca.pem"
  tls_cert_file = "/var/snap/vault/common/etcd-cert.pem"
  tls_key_file = "/var/snap/vault/common/etcd.key"
  etcd_api = "v3"
}
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 1
}

# Localhost only listener for charm access to vault.
listener "tcp" {
  address = "127.0.0.1:8220"
  tls_disable = 1
}

Additional context

On the active vault unit the following entries appear in its log when the etcd unit goes down:

2018-07-20T08:24:16.614Z [WARN ] core: leadership lost, stopping active operation
2018-07-20T08:24:16.614Z [INFO ] core: pre-seal teardown starting
2018-07-20T08:24:16.614Z [INFO ] core: stopping cluster listeners
2018-07-20T08:24:16.614Z [INFO ] core: shutting down forwarding rpc listeners
2018-07-20T08:24:16.614Z [INFO ] core: forwarding rpc listeners stopped
2018-07-20T08:24:16.907Z [INFO ] core: rpc listeners successfully shut down
2018-07-20T08:24:16.907Z [INFO ] core: cluster listeners successfully shut down
2018-07-20T08:24:16.907Z [INFO ] rollback: stopping rollback manager
2018-07-20T08:24:16.908Z [INFO ] core: pre-seal teardown complete

The standby vault units do not log anything as a result of the lost etcd unit.

[gnuoy]

I think the bug may actually reside in the etcd go client library. I have raised a bug there as well etcd-io/etcd#9949

[tdwyer]

I have the same problem.

I have 3 Etcd servers in a cluster and 3 vault v0.10.4 servers in a cluster. All three Vault servers only communicate with the first Etcd URL listed in on the address = line in the Vault server config.

If I run systemctl stop etcd on the Etcd server listed first in the Etcd address in the Vault server config, none of the Vault servers start talking to the other Etcd servers and all Vault servers return "context deadline exceeded".

Even worse, sometimes, if I restart the stopped Etcd server, all Vault servers say broken. I have to reboot all the Vault servers to get them working again.

[tdwyer]

Workaround:

I have 6 Ubuntu 16.04 servers

• 3x Vault v0.10.4
• 3x Etcd v3.3.0

I configure each of the 3 Vault servers with Etcd backend and ha_enabled = "true" but I set a different Etcd server address for each Vault server.

The Vault client I wrote tries all Vault servers in go routines, and my program just talks to the first Vault server to respond with a non-error. Effectively, my Vault client preforms the function of a load-balancer.

Running vault status on each of the Vault servers shows that HA Enabled true and elect a HA Mode active server.

If I kill the Etcd server that the Active Vault server is talking too, that Vault server stops working and returns a 500 context. The other 2 Vault servers elect a new HA Mode active server and continue to work. When I restart the stopped Etcd server, the 3'rd Vault server comes back up and is put in HA Mode standby as one would expect.

If I stop 2/3 Etcd servers, Vault cluster stops working, as one would expect. When I restart 1/3 server (so 2 started 1 stopped) the Vault cluster becomes active again.

NOTE: When each Vault server is configured like this, with only 1 of the 3 Etcd addresses, all Vault servers still try to connect to all Etcd servers and produce the same TLS errors on the other 2 Etcd servers. However, vault server successfully connects to the first Etcd server they try, so each Vault server is connected to the Etcd server in it's config.

[tdwyer]

I think the HA problem and the TLS errors are symptoms of the same issue.

When I disable TLS on the Etcd Client port and configure Vault to communicate with Etcd via plaintext HTTP. All vault servers stay up regardless of which Etcd server I take down. The Vault server in HA Mode active dose not even change.

I think the Vault servers use the Hostname of the first Etcd server in the address list for 'all' the Etcd servers TLS communications. This causes TLS to fail on all Etcd servers except the first in the list, so the Vault servers do not fail-over like they should.

I have tried Vault v0.10.4 and Vault v0.9.0 both with Etcd v3.3.0 and both versions of Vault have the same issue. I have also tried Etcd v3.3.0+git and Etcd v3.1.16, but both have the same issue.

New Workaround idea:

Configure Etcd Client port as plaintext HTTP and Vault to use plaintext HTTP. Then configure IPsec, OpenVPN, or OpenSSH port-forwarding to secure the network communication.

Best Workaround:

Setting etcd_api = "v2" in the Vault Server config solves the problem. HA works as expected and there are no TLS errors.

[raoofm]

cc @xiang90 @philips

[edganiukov]

We experienced the same issue. Seems like vault uses only first etcd node in the list of endpoints.
So when two vault instances have different first etcd endpoint - then they both become a master. This is a huge issue.

[gnuoy]

We experienced the same issue. Seems like vault uses only first etcd node in the list of endpoints.
So when two vault instances have different first etcd endpoint - then they both become a master. This is a huge issue.

Hi @edganiukov I didn't see any evidence of multiple vault instances thinking they were master. As far as I could see the issue was limited to the vault cluster having a single etcd unit as a single point of failure. The bug is still a very high priority imo but I don't think its a split-brain issue.

[edganiukov]

Hi @gnuoy ,
Just checked one more time - now everything works correctly (dunno why I observed this strange behavior before). And seems like the issue was fixed here - etcd-io/etcd#9860

[jsok]

Upgrading the etcd clientv3 library vendored in vault should allow us to close this issue now.

[jefferai]

Updating was done well after that point so going to close this!

[jsok]

@jefferai sorry I wasn't clearer - etcd-io/etcd#10911 is the PR that actually fixed this underlying issue (etcd-io/etcd#9949) and was only merged days ago.

[jsok]

Can this be re-opened please? Vault is still affected by this issue until the etcd client is upgraded.