Cassandra database plugin intermittent timeouts connection to Cassandra

[nmylesnbx]

Describe the bug
I'm getting intermittent timeout errors from the Cassandra database plugin connecting to Cassandra. This happens when calling database/creds/<role>. Most of the time it works but I'm hitting this every 10 or so calls to get new database credentials.

hvac python stacktrace:

File "/usr/local/lib/python3.7/site-packages/hvac/v1/__init__.py", line 195, in read
   return self._adapter.get('/v1/{0}'.format(path), wrap_ttl=wrap_ttl).json()
File "/usr/local/lib/python3.7/site-packages/hvac/adapters.py", line 93, in get
   return self.request('get', url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/hvac/adapters.py", line 276, in request
  utils.raise_for_error(response.status_code, text, errors=errors)
File "/usr/local/lib/python3.7/site-packages/hvac/utils.py", line 39, in raise_for_error
   raise exceptions.InternalServerError(message, errors=errors)
hvac.exceptions.InternalServerError: 1 error occurred:
  * read tcp 192.168.144.4:37610->192.168.144.2:9042: i/o timeout

To Reproduce
Steps to reproduce the behavior:

1. Run vault secrets enable database
2. Run:

vault write database/config/cassandra \
    plugin_name="cassandra-database-plugin" \
    hosts=cassandra \
    protocol_version=4 \
    username=username \
    password=password \
    allowed_roles='*'

3. Run:

vault write database/roles/role \
    db_name=cassandra \
    creation_statements="CREATE USER '{{username}}' WITH PASSWORD '{{password}}' NOSUPERUSER; \
          GRANT SELECT ON ALL KEYSPACES TO {{username}};" \
    default_ttl="1h" \
    max_ttl="24h"

4. Call API multiple times to retrieve creds: GET http://vault:8200/v1/database/creds/role
5. See intermittent 500 errors

Expected behavior
I would expect timeouts to be retried to avoid intermittent 500's

Environment:
This happens both locally (docker-compose) and in a deployed environment

• Vault Server Version (retrieve with vault status): 1.2.2
• Vault CLI Version (retrieve with vault version): 1.2.2
• Server Operating System/Architecture: official vault docker image

Vault cassandra plugin configuration was something like this:

vault write database/config/cassandra \
    plugin_name="cassandra-database-plugin" \
    hosts=cassandra \
    protocol_version=4 \
    username=username \
    password=password \
    allowed_roles='*'

[tyrannosaurus-becks]

Marking as an enhancement because the request appears to be to add retrying when Cassandra returns 500's.

[captain-kark]

If work is going to be done under the hood to fix this, it might help to stem the number of total roles that this plugin generates and maintains.

If some kind of ALTER ROLE strategy can be used in place of DROP ROLE when revoking a role, it would allow a role to be "removed" when a TTL expires without introducing a NULL in the column. In addition, this would also allow for re-instating a role should the exact same access configuration be requested again later (limit one in-flight per TTL, else generate a second copy of the same account), which in my use case is rather common.

Using this plugin with DROP ROLE as a revocation statement can tombstone your system_auth.role_permissions table, which in my experience is one way to get these timeouts to appear.

[munjalpatel]

Any updates here? I am also running into this exact issue. @nmylesnbx were you able to resolve / work around this?

[denglertai]

@munjalpatel it is possible to increase the connect_timeout within the database secret engine configuration settings.
Unfortunately this does not seem to be documented at all.

In my case that solved the problem.
But it was probably related to a slower Cassandra database rather than a problem with Vault itself.

[aphorise]

Hey folks - I feel that this issue is no longer relevant and may be closed. @nmylesnbx any chance you can confirm if / how this may still be applicable for you?

On a related note there's #10467 expected in the next major release of 1.12 which adds initial_connection_timeout, simple_retry_policy_retries.

PS - this request seems somewhat related to #15899

[aphorise]

Closing due to no response. Please re-open if still relevant.