Backport of Fix non-JSON log messages when using -log-format JSON into release/1.13.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #24252 to be assessed for backporting due to the inclusion of the label backport/1.13.x.

🚨

Warning automatic cherry-pick of commits failed. If the first commit failed,
you will see a blank no-op commit below. If at least one commit succeeded, you
will see the cherry-picked commits up to, not including, the commit where
the merge conflict occurred.

The person who merged in the original PR is:
@peteski22
This person should manually cherry-pick the original PR into a new backport PR,
and close this one when the manual backport PR is merged in.

merge conflict error: POST https://api.github.com/repos/hashicorp/vault/merges: 409 Merge conflict []

The below text is copied from the body of the original PR.

---

The intention of this PR is that log entries generated by the consul-template library used by Vault Agent will now be in a format consistent with the supplied configuration.

Fixes: #21109

Steps to manually test:

1. Build Vault from PR/branch (make dev)
2. Run Vault in dev mode:

vault server -dev -dev-root-token-id="root"

3. Open another terminal window and set environment variables from $HOME directory (cd $HOME)

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

4. Create sample data file in $HOME directory

tee data.json -<<EOF
{
   "organization": "ACME Inc.",
   "customer_id": "ABXX2398YZPIE7391",
   "region": "US-West",
   "zip_code": "94105",
   "type": "premium",
   "contact_email": "[email protected]",
   "status": "active"
}
EOF

5. Write sample data to Vault:

vault kv put secret/customers/acme @data.json

6. Create Agent template

tee template.ctmpl -<<EOF
{{- with secret "secret/customers/acme"}}{"secret": "other",
{{- if .Data.data.region}}"region":"{{ .Data.data.region}}",{{- end }}
{{- if .Data.data.contact_email }}"contact_email":"{{ .Data.data.contact_email }}"{{- end }}}
{{- end }}
EOF

7. Create Agent configuration

tee agent-config.hcl -<<EOF
pid_file = "./pidfile"

vault {
   address = "http://127.0.0.1:8200"
   tls_skip_verify = true
}

auto_auth {
   method {
      type = "token_file"
      config = {
         token_file_path = "$HOME/.vault-token"
      }
   }
   sink "file" {
      config = {
            path = "$HOME/vault-token-via-agent"
      }
   }
}

template_config {
  exit_on_retry_failure = false
  static_secret_render_interval = 10
}

template {
  source      = "$HOME/template.ctmpl"
  destination = "$HOME/template-render.txt"
}
EOF

8. Start Agent:

vault agent -config=agent-config.hcl -log-file="$HOME/agent.log" -log-format=json -log-level=trace`

9. Check the logs (after 10, 20, 30s), they will be named with a timestamp suffix. e.g.

The output should be entirely JSON, so you can pipe it through jq.

cat agent-1701091968647204000.log | jq

Example content:

{
  "@level": "info",
  "@message": "(runner) rendered \"/Users/juan/template.ctmpl\" => \"/Users/juantemplate-render.txt\"",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653631Z"
}
{
  "@level": "debug",
  "@message": "(runner) diffing and updating dependencies",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653649Z"
}
{
  "@level": "debug",
  "@message": "(runner) vault.read(secret/customers/acme) is still needed",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653655Z"
}
{
  "@level": "debug",
  "@message": "(runner) watching 1 dependencies",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653665Z"
}
{
  "@level": "debug",
  "@message": "(runner) all templates rendered",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653671Z"
}

10. For completeness, check the rendered template

cat template-render.txt | jq

Example content:

{
  "secret": "other",
  "region": "US-West",
  "contact_email": "[email protected]"
}

---

Overview of commits

• 64dfff0

[hashicorp-cla]

Thank you for your submission! We require that all contributors sign our Contributor License Agreement ("CLA") before we can accept the contribution. Read and sign the agreement

Learn more about why HashiCorp requires a CLA and what the CLA includes

---

temp seems not to be a GitHub user.
You need a GitHub account to be able to sign the CLA. If you already have a GitHub account, please add the email address used for this commit to your account.

_{Have you signed the CLA already but the status is still pending? Recheck it.}

[github-actions]

CI Results:
All Go tests succeeded! ✅

[peteski22]

Not back-porting to 1.13.x