Nomad Consul Legacy ACL Workflow Failing to Deregister Catalog Services on Job Updates

[natemollica-nm]

Consul Nomad Legacy ACL Workflow Service Deregistrations Failing

Creating separate Issue for tracking and resolution from original issue #16616 for Nomad + Consul ACL legacy (non-WI) workflow.

Product Versioning

Consul: v1.19.0
Nomad: v1.8.1

Summary: Consul Catalog Service deregistrations fail upon Nomad Job updates despite the Nomad Agent's Consul ACL token being specified due to Consul defaulting to the anonymous token.

Workaround: Configure Consul client agent ACL stanza to include a token attached to a Consul ACL Token Policy that contains the rules below.

agent_prefix "" {
  policy = "read"
}
node_prefix "" {
  policy = "read"
}
service_prefix "" {
  policy = "write"
}
key_prefix "" {
  policy = "read"
}

Consul Client Agent Configuration Example:

      acl {
        enabled = true
        default_policy = "deny"
        down_policy = "extend-cache"
        tokens {
          initial_management = "<my_nomad_client_token_secretID>"
        }
      }

[tgross]

Internal ref https://hashicorp.atlassian.net/browse/NET-10360

[tgross]

As noted in #23800

Since Consul 1.15 (hashicorp/consul#16097) Consul client agent only uses the agent token to perform anti-entropy after you deregister a service via the Agent API. so the Consul client agent must have its own agent token.