Backport of Datadog Integration into release/1.2.x (cherry-pick)

[natemollica-nm]

Backport

This PR is manually generated from #3407 to be assessed for back porting due to the inclusion of the label backport/1.2.x and failure of auto-back porting.

The below text is copied from the body of the original PR.

---

Changes proposed in this PR

PR Resubmitted from Branch vice Fork for Automation Testing

• Expose several metrics specific features on consul-k8s to include:
  • agent enable_debug
  • agent telemetry.disable_hostname
  • agent telemetry.enable_host_metrics
  • agent telemetry.prefix_filter
  • agent telemetry.dogstatsd_addr
  • agent telemetry.dogstatsd_tags
• Introduce a means to ease the integration and operation of integrating with Datadog Agent metrics collection via fail-safe helm override value parameters. Overrides are intended to allow operators to easily configure 1 of the 3 following methods of monitoring Consul with Datadog on Kubernetes:
  • DogStatsD via one of either "UDP" or "UDS" transport protocols
  • OpenMetrics via Datadogs Autodiscovery feature to scrape the /v1/agent/metrics?format=prometheus endpoint
  • Datadog + Consul Integration Feature standard checks:
    • Serf events and member flaps
    • The Raft protocol
    • DNS performance
    • API Endpoints Health Checks:
      • /v1/agent/metrics?format=prometheus
      • /v1/agent/self
      • /v1/status/leader
      • /v1/status/peers
      • /v1/catalog/services
      • /v1/health/service
      • /v1/health/state/any
      • /v1/coordinate/datacenters
      • /v1/coordinate/nodes
• Introduces server-acl-init token creation for OpenMetrics and Datadog Consul Integration check methods allowing default minimal acl token permission generation for Datadog agent usage as necessary.

How I've tested this PR

• New ACL Token Testing as outline in the CONTRIBUTING.md steps.
• Deployment and testing of local consul-dev (main) and consul-k8s-control-plane-dev (datadog-integration branch) images on k3d test cluster for each scenario. Test repository here.
• Verification of helm templating for new value overrides added as instructed in CONTRIBUTING.md steps. bats ./charts/consul/test/unit --jobs 8 - ran successfully for all tests.

How I expect reviewers to test this PR

• Assess the need for additional unit testing creation and verification.
• If possible:
  • Reach out with any question/concerns or reasons for PR push-back.
  • Verification of fail-safe interlocks between the 3 methods of integration mentioned above.
  • Verification of ACL policy implementation.

Checklist

• Tests added
• CHANGELOG entry added

---

Overview of commits

• 8b6a9d8 - d27ea03 - 8a922e5 - e792599 - 3b27810 - f66f0f3 - 2dc237b - e4a234a - 6c7d488 - 8e7fd4c - f88bf52 - 96692a2 - ae4ac1a - d01568a - 307991b - a139223 - 2d3f39b - 5559f02 - 436884b - 0c4a8b4 - 85abcc8

[curtbushko]

LGTM!