Modified: for 12.1.2



The SBC CNe uses PodDisruptionBudget for all deployments and statefulsets to ensure that the SBC service is not disrupted when the worker node is drained. A Pod Disruption Budget (PDB) limits the number of Pods of a replicated application that are down simultaneously from voluntary disruptions. Kubernetes workloads might become disrupted due to node maintenance or failures. When nodes are taken down for maintenance or experience failures, the pods running on them are terminated and rescheduled on other nodes. This can lead to decreased availability and potential data loss or downtime. 

Involuntary Disruptions

Examples of unavoidable hardware/software failures that might lead to pod disruption:

  • a hardware failure of the physical machine backing the node
  • cluster administrator deletes VM (instance) by mistake
  • cloud provider or hypervisor failure makes VM disappear
  • a kernel panic
  • the node disappears from the cluster due to cluster network partition
  • eviction of a pod due to the node being out-of-resources.

Voluntary Disruptions

Examples of disruptions voluntarily triggered by the application or cluster owner:

  • deleting the deployment or other controller that manages the pod
  • updating a deployment's pod template, causing a restart
  • directly deleting a pod (e.g., by accident)
  • Draining a node for repair or upgrade.
  • Draining a node from a cluster to scale the cluster down (learn about Cluster Autoscaling).
  • Removing a pod from a node to permit something else to fit on that node.


Pod Disruption Budgets allow a minimum number of Pods to remain alive for highly available applications so the service is not disrupted. For example, if we have four pods (replicas) for an application and need at least two for the service to perform normally, we can specify a PDB with a minimum available count of two. When any voluntary disruption occurs, Kubernetes deletes only two pods at once and deletes the other two after two new pods spawn in some other worker node and become ready. This ensures that at least two pods always remain available during the disruption.

PDB Template
# PodDisruptionBudget implementation for all pod types

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: {{ .Release.Name }}-<pod_type>-pdb
  namespace: {{ .Values.global.namespace }}
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: {{ .Release.Name }}-<pod_type>

maxUnavailable is set to 1 for all PDBs. Kubernetes will try to delete one pod at a time so that the number of available Pods is always greater than replica_count - 1.

This ensures that the number of active pods remains equal to the expected count. If any active pod is deleted, one standby pod will take its place. Thus, the number of standby pods is always "0" or one less than the expected count.

Not all voluntary disruptions respect PodDisruptionBudgets, such as a deployment or pod that is deleted with the "kubectl delete" command

Commands that respect PodDisruptionBudgets:

  • kubectl drain: (kubectl-drain-reference) can safely evict all of your pods from a node before performing maintenance on the node (e.g., kernel upgrade, hardware maintenance, etc.). Safe evictions allow the pod's containers to terminate gracefully and will respect the PodDisruptionBudgets you have specified.

  • Eviction API: (eviction-api-reference) Creates an Eviction object, which causes the API server to terminate the Pod. 

curl -v -H 'Content-type: application/json' http://localhost:8090/api/v1/namespaces/<namespace_name>/pods/<pod_name>/eviction -d "{\"apiVersion\": \"policy/v1\",\"kind\": \"Eviction\",\"metadata\": {\"name\": \"<pod_name>\",\"namespace\": \"<namespace_name>\"}}"