Feat/nomad consul vault tokens #3222

paladin-devops · 2022-04-14T03:04:53Z

Add support for authenticating to Consul & Vault when registering Nomad job for both Nomad deployers. Consul token should be supplied via the CONSUL_HTTP_TOKEN env var, and Vault token via the VAULT_TOKEN env var of the runner.

Relevant Nomad API code:
https://github.com/hashicorp/nomad/blob/8f7abae89fb6aef25161a90457e42a1a3cc07fa5/api/jobs.go#L843-L844

Addresses #3206

… basic error checking.

paladin-devops · 2022-04-14T20:04:13Z

Working as expected, I used the nomad-jobspec example from the waypoint-examples repo to test this out, but added a service and vault stanza to trigger the Consul/Vault integrations. Also, my Nomad config file runs with the allow_unauthenticated configs set to false for Consul and Vault:

consul {
  address = "127.0.0.1:8500"
  allow_unauthenticated = false
  token = "my-token"
}

vault {
  enabled = true
  address = "http://127.0.0.1:8200"
  allow_unauthenticated = false
  token = "my-token"
}

Consul

Below shows before/after running deploy with CONSUL_HTTP_TOKEN specified in the runner config.

% waypoint config get -runner                                                                     
  SCOPE | NAME | VALUE | WORKSPACE | LABELS  
--------+------+-------+-----------+---------

% waypoint deploy                

» Deploying example-nodejs...
  Performing operation locally
✓ Running deploy v8
✓ Deleting job: 
! 2 errors occurred:
  	* Unexpected response code: 500 (job-submitter consul token denied: missing
  consul token)
  	* Error during rollback: Unexpected response code: 500 (missing job ID for
  deregistering)
  
  
% waypoint config set -runner -scope=global CONSUL_HTTP_TOKEN=my-token

% waypoint config get -runner                                        
  SCOPE  |       NAME        |                VALUE                 | WORKSPACE | LABELS  
---------+-------------------+--------------------------------------+-----------+---------
  global | CONSUL_HTTP_TOKEN | my-token |           |         
% waypoint deploy                       

» Deploying example-nodejs...
  Performing operation locally
✓ Running deploy v9
✓ Job registration successful
✓ Allocation "ALLOC_ID" created: node "NODE_ID", group "app"
✓ Evaluation status changed: "pending" -> "complete"
✓ Evaluation "EVAL_ID" finished with status "complete"
✓ Deployment successfully rolled out!

  Performing operation locally
✓ Finished building report for Nomad platform
✓ Getting job info...
✓ Job "web" is reporting ready!

Vault

Below shows before/after running deploy with VAULT_TOKEN specified in the runner config.

% waypoint deploy -release=false

» Deploying example-nodejs...
  Performing operation locally
✓ Running deploy v10
✓ Deleting job: 
! 2 errors occurred:
  	* Unexpected response code: 500 (Vault policies requested but missing Vault
  Token)
  	* Error during rollback: Unexpected response code: 500 (missing job ID for
  deregistering)
  
  
% waypoint config set -runner -scope=global VAULT_TOKEN=my-token              

% waypoint deploy                      

» Deploying example-nodejs...
  Performing operation locally
✓ Running deploy v11
✓ Job registration successful
✓ Allocation "ALLOC_ID" created: node "NODE_ID", group "app"
✓ Evaluation status changed: "pending" -> "complete"
✓ Evaluation "EVAL_ID" finished with status "complete"
✓ Deployment successfully rolled out!

briancain

Looks good to me, a fairly straightforward change! 🎉

I'm curious, how can we let users of the Nomad plugin know how to use this in both a local and remote runner context? 🤔 I imagine we should document that Waypoint will respect these env vars, but it might not be obvious that they'd need to configure runners with the env var for remote runs.

briancain · 2022-04-14T20:47:25Z

builtin/nomad/auth.go

+}
+
+func VaultAuth() (string, error) {
+	return os.Getenv("VAULT_TOKEN"), nil


It would be good to confirm this works with remote runners if you haven't already given that a shot with this PR. I think you'd have to do a waypoint config set -runner CONSUL_HTTP_TOKEN=abcd VAULT_TOKEN=1234 and then try an Up operation with ODR/Remote runners and see that the deployment succeeded with these values.

@briancain Looking good! I first unset the tokens to verify that it would fail without them w/remote-operations:

% waypoint config set -scope=global -runner VAULT_TOKEN= % waypoint config set -scope=global -runner CONSUL_HTTP_TOKEN= % waypoint deploy -p=example-nodejs -local=false -release=false » Deploying example-nodejs... Performing this operation on a remote runner with id "JOB_ID" » Cloning data from Git URL: https://github.com/hashicorp/waypoint-examples Ref: feat/nomad-consul-vault-integration ✓ Running deploy v14 ✓ Deleting job: ! 2 errors occurred: * Unexpected response code: 500 (Vault policies requested but missing Vault Token) * Error during rollback: Unexpected response code: 500 (missing job ID for deregistering)

After setting them again, the job deployed:

% waypoint config set -scope=global -runner CONSUL_HTTP_TOKEN=my-token % waypoint config set -scope=global -runner VAULT_TOKEN=my-token % waypoint deploy -p=example-nodejs -local=false -release=false » Deploying example-nodejs... Performing this operation on a remote runner with id "JOB_ID" » Cloning data from Git URL: https://github.com/hashicorp/waypoint-examples Ref: feat/nomad-consul-vault-integration ✓ Job registration successful ✓ Allocation "ALLOC_ID" created: node "NODE_ID", group "app" ✓ Evaluation status changed: "pending" -> "complete" ✓ Evaluation "EVAL_ID" finished with status "complete" ✓ Deployment successfully rolled out! ✓ Running deploy v15 Performing this operation on a remote runner with id "JOB_ID" » Cloning data from Git URL: https://github.com/hashicorp/waypoint-examples Ref: feat/nomad-consul-vault-integration ✓ Finished building report for Nomad platform ✓ Getting job info... ✓ Job "web" is reporting ready!

Nice! 🎉 Looks great

briancain · 2022-04-14T20:52:49Z

Actually, in the topic of where to document this, I think each plugin has a top level Doc that can be defined. It might also have an Env var field?

waypoint/builtin/nomad/jobspec/platform.go

Line 513 in 4e84fdf

func (p *Platform) Documentation() (*docs.Documentation, error) {

xiaolin-ninja · 2022-04-14T21:50:42Z

builtin/nomad/platform.go

@@ -211,6 +211,18 @@ func (p *Platform) resourceJobCreate(
 	job.TaskGroups[0].Tasks[0].Config = config
 	job.TaskGroups[0].Tasks[0].Env = env

+	// Get Consul ACL token from environment
+	*job.ConsulToken, err = ConsulAuth()


Soooo I spent a while trying to wrap my head around this, for other reviewers: this is assigned to pointer job because ConsulToken is a pointer string.

xiaolin-ninja

looks good!

…m runner config.

briancain

Looks great to me @paladin-devops ! You might need to generate the website docs, but otherwise looking gooood 👍🏻

briancain · 2022-04-18T15:42:01Z

builtin/nomad/auth.go

+}
+
+func VaultAuth() (string, error) {
+	return os.Getenv("VAULT_TOKEN"), nil


Nice! 🎉 Looks great

briancain · 2022-04-18T15:42:38Z

builtin/nomad/platform.go

+		"vault_token",
+		"The Vault token used to deploy the Nomad job with a token having specific Vault policies attached.",
+		docs.Summary("Uses the runner config environment variable VAULT_TOKEN."),
+		docs.EnvVar("VAULT_TOKEN"),


Yes! Awesome, thank you. These env var fields are exactly what I was thinking. Looks gooood 👍🏻

You might need to run a `make gen/website-mdx. I think this will generate the website docs based on these fields.

@briancain done!

briancain

🚀

paladin-devops added 2 commits April 13, 2022 22:48

Add support Consul & Vault tokens for job submission.

cca9009

Modularized retrieval of Consul/Vault tokens from environment & added…

fb1d7c4

… basic error checking.

github-actions bot added plugin plugin/nomad labels Apr 14, 2022

Changelog.

cc8048f

vercel bot temporarily deployed to Preview April 14, 2022 03:05 Inactive

briancain requested a review from a team April 14, 2022 20:39

briancain approved these changes Apr 14, 2022

View reviewed changes

xiaolin-ninja reviewed Apr 14, 2022

View reviewed changes

xiaolin-ninja approved these changes Apr 14, 2022

View reviewed changes

Update documentation for Nomad plugin.

f61b881

paladin-devops requested a review from briancain April 18, 2022 14:09

paladin-devops added 2 commits April 18, 2022 10:12

Merge branch 'main' into feat/nomad-consul-vault-tokens

b62fa63

Update documentation for Nomad plugin to indicate reading env var fro…

6720a63

…m runner config.

briancain approved these changes Apr 18, 2022

View reviewed changes

Generated Nomad deployer website docs.

d31365b

github-actions bot added the website label Apr 18, 2022

vercel bot deployed to Preview April 18, 2022 17:25 View deployment

briancain approved these changes Apr 18, 2022

View reviewed changes

paladin-devops added the backport/0.8.x label Apr 18, 2022

paladin-devops merged commit a335f2f into main Apr 18, 2022

paladin-devops deleted the feat/nomad-consul-vault-tokens branch April 18, 2022 18:20

hc-github-team-waypoint mentioned this pull request Apr 18, 2022

Backport of Feat/nomad consul vault tokens into release/0.8.x #3234

Closed

paladin-devops mentioned this pull request May 11, 2022

Add support for Vault and Consul tokens, when submitting Nomad jobs #3206

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat/nomad consul vault tokens #3222

Feat/nomad consul vault tokens #3222

paladin-devops commented Apr 14, 2022

paladin-devops commented Apr 14, 2022

briancain left a comment

briancain Apr 14, 2022

paladin-devops Apr 15, 2022

briancain Apr 18, 2022

briancain commented Apr 14, 2022

xiaolin-ninja Apr 14, 2022 •

edited

Loading

xiaolin-ninja left a comment

briancain left a comment

briancain Apr 18, 2022

briancain Apr 18, 2022

briancain Apr 18, 2022

paladin-devops Apr 18, 2022

briancain left a comment

Feat/nomad consul vault tokens #3222

Feat/nomad consul vault tokens #3222

Conversation

paladin-devops commented Apr 14, 2022

paladin-devops commented Apr 14, 2022

Consul

Vault

briancain left a comment

Choose a reason for hiding this comment

briancain Apr 14, 2022

Choose a reason for hiding this comment

paladin-devops Apr 15, 2022

Choose a reason for hiding this comment

briancain Apr 18, 2022

Choose a reason for hiding this comment

briancain commented Apr 14, 2022

xiaolin-ninja Apr 14, 2022 • edited Loading

Choose a reason for hiding this comment

xiaolin-ninja left a comment

Choose a reason for hiding this comment

briancain left a comment

Choose a reason for hiding this comment

briancain Apr 18, 2022

Choose a reason for hiding this comment

briancain Apr 18, 2022

Choose a reason for hiding this comment

briancain Apr 18, 2022

Choose a reason for hiding this comment

paladin-devops Apr 18, 2022

Choose a reason for hiding this comment

briancain left a comment

Choose a reason for hiding this comment

xiaolin-ninja Apr 14, 2022 •

edited

Loading