Kubernetes Pod Stuck in Pending

Question

A pod has been stuck in Pending state for over 15 minutes. Walk me through how you would troubleshoot and resolve this issue.

Accepted Answer

A pod stuck in Pending for over 15 minutes indicates a scheduling issue. My approach would be systematic, starting with gathering information and progressively narrowing down potential causes. The goal is to quickly identify the bottleneck preventing the pod from being scheduled onto a node. **1. Initial Pod Inspection (kubectl describe pod):** My first step is to run `kubectl describe pod -n `. I'll meticulously examine the 'Events' section. This is the most crucial area for identifying scheduling-related errors. I'll look for messages like 'Unschedulable', 'No suitable nodes found', or specific reasons provided by the scheduler, such as insufficient CPU, memory, or persistent volume claims. **2. Node Capacity and Resource Availability:** If `kubectl describe pod` doesn't provide a clear error, I'd investigate node resources. I'd run `kubectl get nodes` to see the status of all nodes. Then, for nodes that *might* be candidates, I'd use `kubectl describe node ` to check their total allocatable resources (CPU, memory) and compare that against the pod's `resources.requests` and `resources.limits`. A common issue is that no node has enough available capacity for the pod's requested resources. **3. Taints, Tolerations, and Affinity/Anti-Affinity:** I'd then check for scheduling constraints. I'd review the `spec.tolerations` of the pod to see if it can run on nodes with specific taints. Conversely, I'd check the `spec.affinity` and `spec.nodeSelector` fields of the pod to ensure it's not being overly restrictive in where it can be scheduled. I'd also inspect the taints on potential nodes using `kubectl describe node `. Mismatched taints and tolerations or overly strict affinity rules can prevent scheduling. **4. Persistent Volume Claims (PVCs) and Storage:** If the pod requires persistent storage, I'd verify the status of its associated `PersistentVolumeClaim`. I'd run `kubectl get pvc -n ` and `kubectl describe pvc -n `. I'd check if the PVC is `Bound` to a `PersistentVolume` and if that PV is accessible by the nodes. Issues with storage provisioners, availability of the underlying storage, or incorrect `storageClassName` can cause pods to remain pending. **5. Scheduler and Controller-Manager Health:** As a last resort for persistent issues, I'd check the health of the Kubernetes control plane components. I'd ensure the `kube-scheduler` and `kube-controller-manager` pods are running and healthy in the `kube-system` namespace using `kubectl get pods -n kube-system`. Logs from these components, accessed via `kubectl logs -n kube-system`, can sometimes reveal deeper scheduling problems or misconfigurations. Once resolved, I would implement proactive monitoring and alerting for pod pending states. This includes setting up metrics and alerts for pods stuck in pending for extended periods, as well as regular checks on node resource utilization and scheduling health to prevent recurrence.

Kubernetes Pod Stuck in Pending

Ready to prove your Kubernetes administration knowledge?

Kubernetes Pod Stuck in Pending

K8s Readiness vs Liveness Probes

Kubernetes Pod Restart Loop Troubleshooting

Investigating High CPU Usage in Kubernetes Pods

Design an Online Code Execution Sandbox

EKS Cluster Upgrades: A Production Runbook for Platform Teams

From Terraform to GitOps: A Practical Migration Roadmap

GitOps: Monorepo vs Polyrepo - A Practical Comparison