Kubernetes Pod Stuck in Pending

Question

A pod is stuck in Pending state. What are the common causes and troubleshooting steps?

Accepted Answer

When a pod is stuck in the Pending state, it indicates that the Kubernetes scheduler is unable to find a suitable node to place the pod. This is a common troubleshooting scenario, and its causes typically revolve around resource availability or scheduling constraints. **1. Insufficient Node Resources:** A primary reason is that no nodes have sufficient available CPU, memory, or ephemeral storage to satisfy the pod's `requests` specifications. The `kubectl describe pod ` command will often show `FailedScheduling` events with messages like '0/N nodes available: M insufficient cpu, P insufficient memory'. To troubleshoot, use `kubectl top nodes` to assess current resource usage across your cluster, `kubectl get nodes -o wide` to check allocatable resources, and then either adjust the pod's resource requests, add more nodes, or scale down other workloads. **2. Node Taints and Pod Tolerations Mismatch:** Nodes can have 'taints' applied, which repel pods that do not 'tolerate' them. If all eligible nodes have taints that the pending pod doesn't tolerate, it will remain unscheduled. `kubectl describe pod ` events may indicate `FailedScheduling` due to 'node(s) had taints that the pod didn't tolerate'. You should then examine node taints with `kubectl describe node ` and add the necessary `tolerations` to the pod's specification, or, if appropriate, remove the taint from the node. **3. PersistentVolumeClaim (PVC) and StorageClass Issues:** If the pod references a `PersistentVolumeClaim` that cannot be provisioned or bound to a `PersistentVolume`, the pod will remain in Pending state. This is especially true if the pod requests storage via a `StorageClass` that is misconfigured, doesn't exist, or if the underlying storage provisioner (e.g., cloud provider disk service) is experiencing issues. Troubleshooting involves checking `kubectl describe pvc ` for events and status, verifying the `kubectl get storageclass` configuration, and examining logs of the storage provisioner if dynamic provisioning is used. **4. Node Selector, Node Affinity, or Pod Affinity/Anti-Affinity Mismatches:** The pod's scheduling rules, such as `nodeSelector`, `nodeAffinity`, `podAffinity`, or `podAntiAffinity`, might prevent it from being scheduled on any available node. For instance, a `nodeSelector` might specify a label that no node currently possesses. The `kubectl describe pod ` events section is crucial here, providing insights into which scheduling rules are not being met. Verify that the specified labels and rules match existing nodes and other pods as intended. Effectively troubleshooting a Pending pod requires a systematic approach, heavily relying on `kubectl describe` for detailed event information. Proactive monitoring of cluster resources and regular review of pod and node specifications can help prevent many of these scheduling issues.

Kubernetes Pod Stuck in Pending

Ready to prove your Kubernetes administration knowledge?

Kubernetes Pod Stuck in Pending

K8s Readiness vs Liveness Probes

Kubernetes Pod Restart Loop Troubleshooting

Investigating High CPU Usage in Kubernetes Pods

Node Disk Full

EKS Cluster Upgrades: A Production Runbook for Platform Teams

From Terraform to GitOps: A Practical Migration Roadmap

GitOps: Monorepo vs Polyrepo - A Practical Comparison