aks-node-termination-handler (aksnth)

Gracefully handle Azure Virtual Machine (Spot instance) shutdown within Kubernetes

Inspired by aws/aws-node-termination-handler and maksim-paskal/aks-node-termination-handler - kudos to you guys!

Installation

aksnth can be installed using helm

$ helm repo add aksnth https://fridayy.github.io/aks-node-termination-handler/
$ helm repo update
$ helm install aksnth aksnth/aks-node-termination-handler -n aksnth

Features

• Runs on Azure Spot Instance Kubernetes nodes
• Polls the Azure Metadata Service for Scheduled Terminal Events
• Creates a corresponding v1beta.Event Kubernetes event
• Safely drains the node

Event

If a scheduled event is received aksnth emits a kubernetes event in the following format:

LAST SEEN   TYPE      REASON               OBJECT                                          MESSAGE
9m59s       Warning   Unhealthy            node/aios                                       Node 'aios' will change into status 'Preempt' [Not before: 'Mon, 19 Sep 2016 18:29:47 GMT']

Draining the node

aksnth safely drains the affected node by

• Adding a NoSchedule taint on the node
• Cordoning the node
• Creating PodEviction objects

Metrics

aksnth exposes the following metrics on the /metrics endpoint:

# TYPE aksnth_spot_node_available gauge
aksnth_spot_node_available{
   container="aks-node-termination-handler", 
   endpoint="http", 
   instance="10.244.0.2:8080", 
   job="aksnth/aksnth-aks-node-termination-handler", 
   namespace="aksnth", 
   node="aks-somespotpool-38258973-vmss000001", 
   pod="aksnth-aks-node-termination-handler-9jztg"
} 1

aksnth_spot_node_available is 1 until the node receives an eviction event. After that the value will be 0.

# TYPE aksnth_spot_node_eviction_imminent gauge
aksnth_spot_node_eviction_imminent{
  container="aks-node-termination-handler", 
  endpoint="http", 
  instance="10.244.0.2:8080", 
  job="aksnth/aksnth-aks-node-termination-handler", 
  namespace="aksnth", 
  node="aks-dgsc4m16z1-38258973-vmss000001", 
  pod="aksnth-aks-node-termination-handler-9jztg"
} 0

aksnth_spot_node_eviction_imminent is 0 until the node receives an eviction event. After that the value will be 1.

Configuration

Name	Description	Value
mock.enabled	In mock mode scheduled preempt events can be simulated by using the /simulate-eviction endpoint	false
poll.interval	The interval (in milliseconds) used for polling the scheduled events API	1000
podMonitor.enabled	Specfies if a promtheus pod monitor should be created	true
podMonitor.interval	Configures the pod monitor scrape interval to be used	15s
podMonitor.labels	Additional pod monitor labels	{}