Dvue/k8s otel alert policies #2558

douavue · 2024-09-17T17:28:32Z

Summary

A concise description of the changes being introduced. Please review the pre-merge checklist section to validate this pull request is ready for review and merge. If it is not ready, please mark the pull request as a draft.

The owners of this repo are not experts in the subject matter of the quickstarts. We review for the quickstart to be functional and for security risks. If you are seeking feedback on the content of the quickstart, please seek out a subject matter expert. If you are not an internal NR contributor, we can do our best to connect you with a content reviewer.

Pre merge checklist

Did you check you NRQL syntax? - Does it work?
Did you include a Data source and Documentation reference?
Are all documentation links publicly accessible?
Did you check your descriptive content for voice and tone?
Did you check your descriptive content for spelling and grammar errors?
Did you review your content with a subject matter expert? (e.g. a Browser agent quickstart is reviewed with a member of the Browser Agent team)

Dashboards

Does the PR contain a screenshot for each of your dashboards?
Do your screenshots show data?
Has the sanitization script been run on each dashboard?

Alerts

Did you check that your alerts actually work?
Are you trying to create standalone alerts? Standalone alerts are deprecated. They should only be included in quickstarts.

Release - 09/05/2024

CLAassistant · 2024-09-17T17:28:40Z

All committers have signed the CLA.

douavue · 2024-09-17T19:10:30Z

Marking as Ready, but pending additional changes.
DO NOT MERGE.

sjyothi54

@douavue I have added some comments and Could you please ensure consistency in the title "Kubernetes (OpenTelemetry)" throughout the quickstart.

sjyothi54 · 2024-09-18T08:36:31Z

quickstarts/kubernetes-opentelemetry/config.yml

+      Kubernetes is an open-source container-orchestration system for automating
+      computer application deployment, scaling, and management.
+    url: >-
+      https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration


Suggested change

https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration

https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/installation/k8s-otel/#install

I noticed that we have a Kubernetes OpenTelemetry-specific document.

sjyothi54 · 2024-09-18T08:43:27Z

quickstarts/kubernetes-opentelemetry/config.yml

+    url: >-
+      https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration
+dataSourceIds:
+  - kubernetes


Suggested change

- kubernetes

- kubernetes-opentelemetry

Since this quickstart is different from Kubernetes, you need to create a new datasource: kubernetes-opentelemetry. Here is the example: https://github.com/newrelic/newrelic-quickstarts/tree/release/data-sources/apigee-api

Hi @sjyothi54. Do the dataSourceId and dashboards properties require a value or can I remove them from the config?

Hey @douavue, dashboards are optional, but it is best practice to provide them in order to show the customer what are the metrics they get in dashboard. Since this is a new quickstart, datasourceIds is required. Please provide datasourceIds for the Kubernets opentelemetry, it is used to create install steps in production.

@sjyothi54 - I pushed an update here. We are not including a dashboard for now. Can you please review again? Thanks.

@sjyothi54 we don't provide a dashboard, because the K8s OTel instrumentation enables all the regular K8s UIs so no additional dashboards are needed.

@sjyothi54 we don't provide a dashboard, because the K8s OTel instrumentation enables all the regular K8s UIs so no additional dashboards are needed.

@mangulonr @douavue Thanks for the clarification

sjyothi54 · 2024-09-18T08:46:36Z

quickstarts/kubernetes-opentelemetry/config.yml

+dashboards:
+  - kubernetes


Suggested change

dashboards:

- kubernetes

dashboards:

- kubernetes

Since you attached the existing Kubernetes dashboard, will this dashboard include the metrics that we receive in Kubernetes OpenTelemetry?

sjyothi54

@douavue Looks good.

RamanaReddy8801

Hi @douavue, please add the icon attribute in the file quickstarts/kubernetes-opentelemetry/config.yml and make sure the logo image is related to Kubernetes.

RamanaReddy8801 · 2024-10-08T06:21:24Z

data-sources/kubernetes-opentelemetry/config.yml

+  primary:
+    link:
+      url: https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/installation/k8s-otel/#install
+icon: logo.png


Suggested change

icon: logo.png

icon: logo.svg

@RamanaReddy8801 - Updated here

douavue · 2024-10-08T13:55:19Z

@sjyothi54 - Updated the logo here.

What's required to for this step? Thanks for all your help.

zeitlerc · 2024-10-08T14:10:56Z

alert-policies/kubernetes-opentelemetry/ContainerCPUThrottling.yaml

+    # Operator used to compare against the threshold.
+    operator: ABOVE
+    # Value that triggers a violation
+    threshold: 90


The description says Alert when container is being throttled > 25% but the threshold value is 90. I think it should be 25?

zeitlerc · 2024-10-08T14:15:36Z

alert-policies/kubernetes-opentelemetry/ContainerHighMemUtil.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric max(container_memory_working_set_bytes) / filter(max(kube_pod_container_resource_limits), where resource = 'memory') where k8s.cluster.name in ('YOUR_CLUSTER_NAME') and k8s.namespace.name in ('YOUR_NAMESPACE_NAME') facet k8s.container.name, k8s.pod.name, k8s.namespace.name, k8s.cluster.name"


This is missing select. It should be from Metric select max(container_memory_working_set_bytes)

I can't get this one to work because I can't find Metrics where they contain container_memory_working_set_bytes and k8s.container.name at the same time. It could be a skill issue tho.

zeitlerc · 2024-10-08T14:49:17Z

alert-policies/kubernetes-opentelemetry/NodeHighAllocatableCPUUtil.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric select latest(k8s.node.cpu.utilization) where k8s.cluster.name in ('YOUR_CLUSTER_NAME') facet k8s.node.name, k8s.cluster.name"


I believe latest(k8s.node.cpu.utilization) would get you the last CPU utilization for the 5 minute window, which could be a brief spike where the other 4 minutes were relatively low. You might want average(k8s.node.cpu.utilization) based on the description.

zeitlerc · 2024-10-08T14:50:21Z

alert-policies/kubernetes-opentelemetry/NodeHighAllocatableMemUtil.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric select latest(k8s.node.memory.working_set) / filter(latest(kube_node_status_allocatable), WHERE resource = 'memory')  where k8s.cluster.name in ('YOUR_CLUSTER_NAME') facet k8s.node.name, k8s.cluster.name"


Same as CPU. Do you want average(k8s.node.memory.working_set)?

zeitlerc · 2024-10-08T14:53:09Z

alert-policies/kubernetes-opentelemetry/NodeHighAllocatableCPUUtil.yaml

+    # Operator used to compare against the threshold.
+    operator: ABOVE
+    # Value that triggers a violation
+    threshold: 90


The query returns a value 0 to 1. We would need to multiply the select value by 100 or divide this by 100.

zeitlerc · 2024-10-08T14:56:22Z

alert-policies/kubernetes-opentelemetry/NodeHighFSCapacityUtil.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric select max(k8s.node.filesystem.usage) / max(k8s.node.filesystem.capacity) where k8s.cluster.name in ('YOUR_CLUSTER_NAME') facet k8s.node.name, k8s.cluster.name"


The description says average but the query uses max.

zeitlerc · 2024-10-08T14:57:26Z

alert-policies/kubernetes-opentelemetry/NodeIsNotReady.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric if(latest(condition) = 'Ready', 0, 1) where k8s.cluster.name in ('YOUR_CLUSTER_NAME') facet k8s.node.name, k8s.cluster.name"


This is missing a select in from Metric select if(latest(condition)

zeitlerc · 2024-10-08T15:22:28Z

alert-policies/kubernetes-opentelemetry/NodePodCapacity.yaml

+
+# NRQL query
+nrql:
+  query: "FROM Metric select filter(uniqueCount(k8s.pod.name), where phase = 'Running' AND (metricName = 'kube_pod_status_phase' AND kube_pod_status_phase ['latest'] = 1) and created_by_kind != 'Job' ) / filter(latest(kube_node_status_allocatable), WHERE resource = 'cpu' ) * 100 as 'Pod Capacity %' where k8s.node.name != '' and k8s.node.name is not null and k8s.cluster.name in ('YOUR_CLUSTER_NAME') facet k8s.node.name, k8s.cluster.name"


I believe WHERE resource = 'cpu' should be WHERE resource = 'pods'

zeitlerc · 2024-10-08T15:24:59Z

alert-policies/kubernetes-opentelemetry/PodsFailingNamespace.yaml

+    # Operator used to compare against the threshold.
+    operator: ABOVE
+    # Value that triggers a violation
+    threshold: 0


The description says Alert when more than 5 pods so I think this should be 5.

zeitlerc · 2024-10-08T15:25:23Z

alert-policies/kubernetes-opentelemetry/StatefulsetPodsMissing.yaml

+
+# NRQL query
+nrql:
+  query: "from Metric latest(kube_statefulset_replicas) - latest(kube_statefulset_status_replicas_ready) where k8s.cluster.name in ('YOUR_CLUSTER_NAME') and k8s.namespace.name in ('YOUR_NAMESPACE_NAME') facet k8s.statefulset.name, k8s.namespace.name, k8s.cluster.name"


Missing select in from Metric latest(kube_statefulset_replicas)

RamanaReddy8801

LGTM

aswanson-nr and others added 2 commits September 5, 2024 10:27

Merge pull request newrelic#2532 from newrelic/release

ce64627

Release - 09/05/2024

feat: Add new quickstart and alert policies for k8s otel

40be710

douavue marked this pull request as ready for review September 17, 2024 19:28

sjyothi54 requested changes Sep 18, 2024

View reviewed changes

feat: Add datasource for k8s otel

da8ce23

sjyothi54 previously approved these changes Oct 8, 2024

View reviewed changes

RamanaReddy8801 requested changes Oct 8, 2024

View reviewed changes

douavue added 2 commits October 8, 2024 08:43

fix: Update image extension

ef5d5bf

fix: Update image for k8s-otel quickstart

d0572ac

zeitlerc reviewed Oct 8, 2024

View reviewed changes

douavue added 3 commits October 9, 2024 09:50

fix: Address CR feedback re nrql and configs

08de38e

feat: Update k8s otel description

4f54b21

fix: Add icon key/value

5439de3

RamanaReddy8801 approved these changes Oct 17, 2024

View reviewed changes

douavue dismissed sjyothi54’s stale review via 5439de3 October 17, 2024 15:43

nr-mlosier merged commit c5ed531 into newrelic:release Oct 17, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Dvue/k8s otel alert policies #2558

Dvue/k8s otel alert policies #2558

douavue commented Sep 17, 2024

CLAassistant commented Sep 17, 2024 •

edited

Loading

douavue commented Sep 17, 2024 •

edited

Loading

sjyothi54 left a comment

sjyothi54 Sep 18, 2024

sjyothi54 Sep 18, 2024

douavue Oct 1, 2024

sjyothi54 Oct 4, 2024

douavue Oct 7, 2024

mangulonr Oct 8, 2024

sjyothi54 Oct 10, 2024

sjyothi54 Sep 18, 2024

sjyothi54 left a comment

RamanaReddy8801 left a comment

RamanaReddy8801 Oct 8, 2024

douavue Oct 8, 2024

douavue commented Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

zeitlerc Oct 8, 2024

RamanaReddy8801 left a comment

	https://docs.newrelic.com/docs/integrations/host-integrations/host-integrations-list/kubernetes-monitoring-integration
	https://docs.newrelic.com/docs/kubernetes-pixie/kubernetes-integration/installation/k8s-otel/#install

Dvue/k8s otel alert policies #2558

Dvue/k8s otel alert policies #2558

Conversation

douavue commented Sep 17, 2024

Summary

Pre merge checklist

Dashboards

Alerts

CLAassistant commented Sep 17, 2024 • edited Loading

douavue commented Sep 17, 2024 • edited Loading

sjyothi54 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjyothi54 left a comment

Choose a reason for hiding this comment

RamanaReddy8801 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

douavue commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RamanaReddy8801 left a comment

Choose a reason for hiding this comment

CLAassistant commented Sep 17, 2024 •

edited

Loading

douavue commented Sep 17, 2024 •

edited

Loading