Backport of Fix non-JSON log messages when using -log-format JSON into release/1.15.x

[hc-github-team-secure-vault-core]

Backport

This PR is auto-generated from #24252 to be assessed for backporting due to the inclusion of the label backport/1.15.x.

The below text is copied from the body of the original PR.

---

The intention of this PR is that log entries generated by the consul-template library used by Vault Agent will now be in a format consistent with the supplied configuration.

Fixes: #21109

Steps to manually test:

1. Build Vault from PR/branch (make dev)
2. Run Vault in dev mode:

vault server -dev -dev-root-token-id="root"

3. Open another terminal window and set environment variables from $HOME directory (cd $HOME)

export VAULT_ADDR='http://127.0.0.1:8200'
export VAULT_TOKEN='root'

4. Create sample data file in $HOME directory

tee data.json -<<EOF
{
   "organization": "ACME Inc.",
   "customer_id": "ABXX2398YZPIE7391",
   "region": "US-West",
   "zip_code": "94105",
   "type": "premium",
   "contact_email": "[email protected]",
   "status": "active"
}
EOF

5. Write sample data to Vault:

vault kv put secret/customers/acme @data.json

6. Create Agent template

tee template.ctmpl -<<EOF
{{- with secret "secret/customers/acme"}}{"secret": "other",
{{- if .Data.data.region}}"region":"{{ .Data.data.region}}",{{- end }}
{{- if .Data.data.contact_email }}"contact_email":"{{ .Data.data.contact_email }}"{{- end }}}
{{- end }}
EOF

7. Create Agent configuration

tee agent-config.hcl -<<EOF
pid_file = "./pidfile"

vault {
   address = "http://127.0.0.1:8200"
   tls_skip_verify = true
}

auto_auth {
   method {
      type = "token_file"
      config = {
         token_file_path = "$HOME/.vault-token"
      }
   }
   sink "file" {
      config = {
            path = "$HOME/vault-token-via-agent"
      }
   }
}

template_config {
  exit_on_retry_failure = false
  static_secret_render_interval = 10
}

template {
  source      = "$HOME/template.ctmpl"
  destination = "$HOME/template-render.txt"
}
EOF

8. Start Agent:

vault agent -config=agent-config.hcl -log-file="$HOME/agent.log" -log-format=json -log-level=trace`

9. Check the logs (after 10, 20, 30s), they will be named with a timestamp suffix. e.g.

The output should be entirely JSON, so you can pipe it through jq.

cat agent-1701091968647204000.log | jq

Example content:

{
  "@level": "info",
  "@message": "(runner) rendered \"/Users/juan/template.ctmpl\" => \"/Users/juantemplate-render.txt\"",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653631Z"
}
{
  "@level": "debug",
  "@message": "(runner) diffing and updating dependencies",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653649Z"
}
{
  "@level": "debug",
  "@message": "(runner) vault.read(secret/customers/acme) is still needed",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653655Z"
}
{
  "@level": "debug",
  "@message": "(runner) watching 1 dependencies",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653665Z"
}
{
  "@level": "debug",
  "@message": "(runner) all templates rendered",
  "@module": "agent",
  "@timestamp": "2023-11-27T17:06:20.653671Z"
}

10. For completeness, check the rendered template

cat template-render.txt | jq

Example content:

{
  "secret": "other",
  "region": "US-West",
  "contact_email": "[email protected]"
}

---

Overview of commits

• 64dfff0

[github-actions]

Build Results:
All builds succeeded! ✅

[github-actions]

CI Results:
All Go tests succeeded! ✅