Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Workload Identity forces disabling of mutual TLS client verification, contradicting Nomad’s own best practices #29435

Open
lattwood opened this issue Jan 28, 2025 · 0 comments
Labels

Comments

@lattwood
Copy link

lattwood commented Jan 28, 2025

Describe the bug
When using Nomad Workload Identity (JWT-based) to integrate with Vault, Nomad’s documentation states:

“It is highly recommended to use mutual TLS in production deployments of Nomad. With mTLS enabled, the tls.verify_https_client configuration must be set to false since it is not possible to provide client certificates to the Vault auth method.”

This effectively disables Nomad’s mutual TLS client verification for Vault calls. Requiring tls.verify_https_client = false directly conflicts with the general recommendation to use mTLS in production. This omission forces operators to choose between best-practice security (mTLS everywhere) and enabling JWT-based workload identity.

In other words, we are asked to run our cluster in a way that is not aligned with the very security best practices Nomad prescribes.

To Reproduce

  1. Configure Nomad for JWT-based Workload Identity.
  2. Enable mutual TLS across the Nomad cluster.
  3. Attempt to use tls.verify_https_client = true for Vault calls.
  4. Observe that JWT authentication to Vault fails because you cannot provide client certificates on the Vault JWT auth method.

Expected behavior
We expect full mTLS support during JWT authentication to Vault. Ideally, Vault’s JWT auth method (or Nomad’s usage of it) should be able to present a client certificate to Nomad, such that we can keep tls.verify_https_client = true while still using workload identities.

Environment:

  • Vault Server Version (retrieve with vault status): a version supporting jwt/oidc
  • Vault CLI Version (retrieve with vault version): a version supporting jwt/oidc
  • Server Operating System/Architecture: Linux of some kind

Vault server configuration file(s):

jwt_validation_pubkeys = [...]
jwks_ca_pem = "..."

Nomad service configuration file(s):

tls {
  http = true
  rpc  = true

  verify_server_hostname = true
  verify_https_client    = true  # Fails with JWT-based workload identity
}

Additional context

The docs explicitly state to disable tls.verify_https_client, despite also saying mTLS is recommended.

Security Impact

Disabling client verification undermines the otherwise strong security model that mTLS provides. With client certificate authentication disabled, we rely solely on token-based authentication without the additional layer of cryptographic identity verification at the TLS layer.

Request

Please add official support for presenting client certificates through the Vault JWT auth method, or provide an alternative approach that does not require disabling tls.verify_https_client. This feature would align Nomad Workload Identity with the recommended production security posture.

Or, don't deprecate Vault Token authentication until Workload Identity is at feature parity while adhering to best practices. (Nomad docs say vault token authentication goes away in 1.10, and latest tag is 1.9.5)

I couldn't find an open issue after some quick searching, so I opened this.

edit: Opened hashicorp/nomad#24970 to push out the deprecation again, or until this feature exists

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants