token store expiration failed to revoke non SSCT (missing cubbyhole ID)

[shaj13]

Describe the bug
Vault Server unable to revoke old tokens (27 bytes) non SSCT.
2022-12-05T20:27:51.825Z [ERROR] expiration: failed to revoke lease: lease_id=auth/cert/login/h6d0772972fd1601be0b77048fb70dd7bb217982fefd4f0eb41c2fba8b49bc188 error="failed to revoke token: failed to revoke entry: missing cubbyhole ID while destroying"

To Reproduce

1. Run Vault server and set VAULT_DISABLE_SERVER_SIDE_CONSISTENT_TOKENS to true
2. vault login
3. vault auth enable cert
4. vault write auth/cert/certs/web
   display_name=web
   policies=web,prod
   [email protected]
   ttl=60
5. Run multiple login curl -k https://localhost:8200/v1/auth/cert/login -X POST --cert ./web-cert.pem --key ./web-key.pem | jq
6. wait a 1-2 min
7. check the logs
8. [ERROR] expiration: failed to revoke lease: lease_id=auth/cert/login *****

Expected behavior
token revoked, without any error
For example
2022-12-11T20:22:27.714Z [INFO] expiration: revoked lease: lease_id=auth/cert/login/h2e1b24fb284004483ea5110449916305107dd93387a5edeec7b5e141d10802cf

Environment:

• Vault Server Version (retrieve with vault status):
  Key Value

---

Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.12.2
Storage Type postgresql

• Vault CLI Version (retrieve with vault version):
  Vault v1.12.2
• Server Operating System/Architecture:
  docker ubuntu image

/etc/os-release
NAME="Ubuntu"
VERSION="20.04.5 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.5 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Vault server configuration file(s):

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/tmp/vault.pem"
  tls_key_file  = "/tmp/vault-key.pem"
  tls_disable   = 0
}

api_addr     = "https://0.0.0.0:8200"
cluster_addr = "https://0.0.0.0:8201"

storage "postgresql" {
  connection_url = "*******"
  ha_enabled     = false
}

[Kardi5]

I have been hitting the exact same bug. I also have SSCT disabled and initially noticed the bug due to an increase in memory consumption for the Consul storage backend. consul snapshot inspect showed about 500k old Vault leases. The number of leases was confirmed via the Vault web UI.

@hsimon-hashicorp Neither revoke nor force revoke work as both complain about the missing cubbyhole ID. Any pointers how to delete the old leases and free up memory/storage despite the errors?