Node CPU Thrashing

Question

One node in your cluster shows 100% CPU usage with context switching spikes. How do you troubleshoot?

Accepted Answer

A 100% CPU usage with context switching spikes on a cluster node is a critical issue that requires immediate and systematic investigation. My approach would be to quickly isolate the root cause to minimize impact on the cluster's services. First, I'd immediately log into the affected node and use tools like `top` or `htop` to confirm the CPU saturation. I'd sort by CPU usage and look for any single process or a small group of processes consuming an unusually high percentage of CPU. Simultaneously, I'd monitor the context switching rate (often shown in `top`'s summary line or available via `vmstat -w`) to verify the spike. If `mpstat` is available, I'd use it to get a per-CPU breakdown and look for any particular cores being overutilized. Next, I would focus on identifying the cause of the high context switching. This often indicates a process that is repeatedly being scheduled and descheduled, or a high rate of interrupts. I'd examine the output of `vmstat` or `sar -w` for detailed context switch and interrupt statistics. High interrupt rates could point to I/O issues or network driver problems. With the potential culprit processes identified, I'd investigate them further. For runaway processes, I'd check their command line arguments, recent activity (e.g., file access, network connections using `lsof` or `strace`), and their resource consumption history if available. For 'noisy neighbors' (in a virtualized or containerized environment), I'd use the hypervisor or container orchestration tools to see if other VMs or containers on the same host are misbehaving and impacting the affected node. If a specific process is identified as the culprit, I'd investigate its logs for error messages or repeated operations that might be causing the excessive CPU. Depending on the service, I might try restarting the problematic service gracefully or, as a last resort in a critical situation, kill the offending process if its impact is severe and its nature is understood. I would also check for any recent configuration changes or deployments that might have introduced this behavior. Finally, I'd examine system-level metrics and logs for other potential indicators. This includes looking at disk I/O (`iostat`), network traffic (`nload`, `iftop`), and kernel logs (`dmesg`, `/var/log/syslog` or `journalctl`) for any correlated errors or warnings that might point to underlying hardware, network, or kernel-level issues contributing to the CPU load and context switching. Post-resolution, it's crucial to implement robust monitoring and alerting for CPU utilization and context switching rates. Proactive measures like regular performance tuning, resource planning, and automated anomaly detection can prevent such critical outages from recurring and ensure cluster stability.

Node CPU Thrashing

CPU Load Average Explained

Find Top CPU Processes in Linux

Troubleshooting High Memory in Linux

Find Large Files with Bash

Investigating High CPU Usage in Kubernetes Pods

Edge Computing Meets Edge Caching: Building Real-Time Applications at the Network Edge

Paginated vs. Bulk API Calls: Performance, Cost, and Reliability Trade‑offs

Real-Time Monitoring with eBPF: Low-Overhead Insights for Linux & K8s