Datadog Integration

[natemollica-nm]

Changes proposed in this PR

• Expose several metrics specific features on consul-k8s to include:
  • agent enable_debug
  • agent telemetry.disable_hostname
  • agent telemetry.enable_host_metrics
  • agent telemetry.prefix_filter
  • agent telemetry.dogstatsd_addr
  • agent telemetry.dogstatsd_tags
• Introduce a means to ease the integration and operation of integrating with Datadog Agent metrics collection via fail-safe helm override value parameters. Overrides are intended to allow operators to easily configure 1 of the 3 following methods of monitoring Consul with Datadog on Kubernetes:
  • DogStatsD via one of either "UDP" or "UDS" transport protocols
  • OpenMetrics via Datadogs Autodiscovery feature to scrape the /v1/agent/metrics?format=prometheus endpoint
  • Datadog + Consul Integration Feature standard checks:
    • Serf events and member flaps
    • The Raft protocol
    • DNS performance
    • API Endpoints Health Checks:
      • /v1/agent/metrics?format=prometheus
      • /v1/agent/self
      • /v1/status/leader
      • /v1/status/peers
      • /v1/catalog/services
      • /v1/health/service
      • /v1/health/state/any
      • /v1/coordinate/datacenters
      • /v1/coordinate/nodes
• Introduces server-acl-init token creation for OpenMetrics and Datadog Consul Integration check methods allowing default minimal acl token permission generation for Datadog agent usage as necessary.

How I've tested this PR

• New ACL Token Testing as outline in the CONTRIBUTING.md steps.
• Deployment and testing of local consul-dev (main) and consul-k8s-control-plane-dev (datadog-integration branch) images on k3d test cluster for each scenario. Test repository here.
• Verification of helm templating for new value overrides added as instructed in CONTRIBUTING.md steps. bats ./charts/consul/test/unit --jobs 8 - ran successfully for all tests.

How I expect reviewers to test this PR

• Assess the need for additional unit testing creation and verification.
• If possible:
  • Reach out with any question/concerns or reasons for PR push-back.
  • Verification of fail-safe interlocks between the 3 methods of integration mentioned above.
  • Verification of ACL policy implementation.

Checklist

• Tests added
• CHANGELOG entry added

[curtbushko]

Closing in favour of #3407