Error making API request. Code: 400

[asmorodskyi]

Some calls to generate SA end up like this -

vault read "qac/gcp/key/openqa-role"                                                                                                 
Error reading qac/gcp/key/openqa-role: Error making API request.

URL: GET https://<Vault server>/vault/v1/qac/gcp/key/openqa-role
Code: 400. Errors:

* googleapi: Error 400: Precondition check failed., failedPrecondition

Some info about setup :

1. running Vault in container

2. Vault version

vault    | ==> Vault server configuration:
vault    | 
vault    |                      Cgo: disabled
vault    |               Go Version: go1.15.13
vault    |               Listener 1: tcp (addr: "0.0.0.0:1234", cluster address: "0.0.0.0:1235", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault    |                Log Level: debug
vault    |                    Mlock: supported: true, enabled: true
vault    |            Recovery Mode: false
vault    |                  Storage: inmem
vault    |                  Version: Vault v1.7.3
vault    |              Version Sha: 5d517c864c8f10385bf65627891bc7ef55f5e827

3. Vault config :

vault    | 2021-07-12T09:45:09.752Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault    | 2021-07-12T09:45:09.753Z [WARN]  no `api_addr` value specified in config or in VAULT_API_ADDR; falling back to detection if possible, but this value should be manually set
vault    | 2021-07-12T09:45:09.753Z [DEBUG] core: set config: sanitized config={"api_addr":"","cache_size":0,"cluster_addr":"","cluster_cipher_suites":"","cluster_name":"","default_lease_ttl"
:14400000000000,"default_max_request_duration":0,"disable_cache":false,"disable_clustering":true,"disable_indexing":false,"disable_mlock":false,"disable_performance_standby":false,"disable_pr
intable_check":false,"disable_sealwrap":false,"disable_sentinel_trace":false,"enable_response_header_hostname":false,"enable_response_header_raft_node_id":false,"enable_ui":false,"listeners":
[{"config":{"address":"0.0.0.0:1234","tls_disable":1},"type":"tcp"}],"log_format":"unspecified","log_level":"Debug","max_lease_ttl":86400000000000,"pid_file":"","plugin_directory":"","raw_sto
rage_endpoint":false,"seals":[{"disabled":false,"type":"shamir"}],"storage":{"cluster_addr":"","disable_clustering":true,"redirect_addr":"","type":"inmem"}}
vault    | 2021-07-12T09:45:09.753Z [DEBUG] storage.cache: creating LRU cache: size=0

4. gcp config :

vault    | 2021-07-12T09:45:38.342Z [DEBUG] secrets.gcp.gcp_d7312728: updating roleset with new account
vault    | 2021-07-12T09:45:38.342Z [DEBUG] secrets.gcp.gcp_d7312728: adding WALs for old roleset resources
vault    | 2021-07-12T09:45:38.342Z [DEBUG] secrets.gcp.gcp_d7312728: skip WALs for nil roleset resources
vault    | 2021-07-12T09:45:38.342Z [DEBUG] secrets.gcp.gcp_d7312728: adding WALs for new roleset resources
vault    | 2021-07-12T09:45:38.342Z [DEBUG] secrets.gcp.gcp_d7312728: creating service account: project=..... request="&{vaultopenqa-role-..... [] []}"
vault    | 2021-07-12T09:45:38.343Z [DEBUG] secrets.gcp.gcp_d7312728: loading credentials
vault    | 2021-07-12T09:45:38.343Z [DEBUG] secrets.gcp.gcp_d7312728: creating oauth2 http client
vault    | 2021-07-12T09:45:39.089Z [DEBUG] secrets.gcp.gcp_d7312728: creating IAM bindings: account_email=......
vault    | 2021-07-12T09:45:39.090Z [DEBUG] secrets.gcp.gcp_d7312728: setting IAM binding: resource=//cloudresourcemanager.googleapis.com/projects/.... roles="map[roles/compute.admin:{
} roles/iam.serviceAccountUser:{} roles/storage.objectAdmin:{} roles/viewer:{}]"
vault    | 2021-07-12T09:45:39.090Z [DEBUG] secrets.gcp.gcp_d7312728: getting IAM policy for resource name: name=//cloudresourcemanager.googleapis.com/projects/.....
vault    | 2021-07-12T09:45:39.365Z [DEBUG] secrets.gcp.gcp_d7312728: got IAM policy for resource name: name=//cloudresourcemanager.googleapis.com/projects/......
vault    | 2021-07-12T09:45:39.366Z [DEBUG] secrets.gcp.gcp_d7312728: setting IAM policy for resource name: name=//cloudresourcemanager.googleapis.com/projects/......
vault    | 2021-07-12T09:45:39.953Z [DEBUG] secrets.gcp.gcp_d7312728: skip deletion for nil roleset resources

Note : I am aware about https://www.vaultproject.io/docs/secrets/gcp#service-account-keys-quota-limits but decided to create this anyway because I haven't found any prove that this is exactly about Quota limit :

1. Error message from API not saying anything about breaking some limit
2. I checked Quotas in the project they all green and Google not saying that any of them should be enlarged

So even if this really about Quotas I see room for error message improvement