Node Disk Full

Question

One of your production nodes is reporting 100% disk usage and workloads are failing. How do you investigate and resolve this?

Accepted Answer

A 100% disk usage on a production node is a critical incident that requires immediate and systematic investigation to restore service and prevent recurrence. My primary goal would be to quickly identify the root cause and implement a resolution with minimal downtime. **Initial Assessment & Filesystem Identification:** My first step would be to connect to the affected node, either via SSH or through Kubernetes' `kubectl exec` command into a privileged pod. I would immediately run `df -h` to get an overview of filesystem usage and pinpoint exactly which mount point is at 100%. This would tell me if it's the root filesystem, a specific data volume, or perhaps temporary storage. **Deep Dive with `du` and Log Analysis:** Once the full filesystem is identified, I'd use `du -sh /*` (and progressively drill down into subdirectories, e.g., `du -sh /var/*`) to find the largest directories and files. Simultaneously, I'd be checking application and system logs located in `/var/log` for any unusual spikes or errors that might indicate a runaway process generating excessive data. I'd also look for large temporary directories like `/tmp`. **Container-Specific Storage Examination:** In a Kubernetes environment, a significant portion of disk usage can be attributed to container runtime storage. I would investigate the container overlay filesystem (e.g., `/var/lib/docker/overlay2` or similar depending on the container runtime) for stale or unreferenced image layers and exited container volumes. Tools like `docker system prune -a` (with caution) or specific Kubernetes cleanup mechanisms would be considered. **Application & Workload Impact Analysis:** While identifying the source of the disk usage, I'd also be looking at the failing workloads. This involves checking pod logs via `kubectl logs ` and `kubectl describe pod ` for error messages related to disk I/O, timeouts, or specific application failures. This helps confirm that the disk issue is indeed the cause and not a symptom of a larger problem. **Resolution & Cleanup Strategy:** Based on the findings, the resolution would involve a combination of actions. This could include: deleting or rotating excessively large log files, removing stale container images or unused volumes, stopping and deleting pods that are causing excessive writes, or if it's a persistent volume issue, escalating to the storage administrator for resizing or performance tuning. If a specific application is identified as the culprit, I'd work with the development team to address the underlying issue (e.g., inefficient data writing, memory leaks). Resolving this critical issue is just the first step. To prevent future occurrences, I'd focus on implementing robust monitoring for disk utilization across all nodes and storage volumes, with automated alerting. Additionally, I'd work on establishing clear log rotation policies and automated cleanup mechanisms for container images and temporary files.

Node Disk Full

Ready to prove your Kubernetes administration knowledge?

du and df Show Different Disk Usage

Kubernetes Pod Stuck in Pending

Find Top CPU Processes in Linux

Troubleshooting High Memory in Linux

Find Large Files with Bash

Real-Time Monitoring with eBPF: Low-Overhead Insights for Linux & K8s

EKS Cluster Upgrades: A Production Runbook for Platform Teams

From Terraform to GitOps: A Practical Migration Roadmap