Container OOMKilled Repeatedly

Question

A container is consistently OOMKilled under normal workload. How do you debug and fix it?

Accepted Answer

Encountering an OOMKilled container under normal load is a critical issue that requires systematic debugging. My approach would involve a multi-pronged investigation, starting from the Kubernetes level and drilling down into the container's internal processes. First, I would check Kubernetes events and system logs on the node where the container is running. Specifically, I'd look for `OOMKilled` events in `kubectl get events` and examine `/var/log/syslog` or `journalctl` on the node for relevant kernel messages (dmesg) that indicate the Out-Of-Memory killer action. Next, I'd meticulously review the container's resource requests and limits defined in its Kubernetes deployment manifest. I'd then compare these limits against actual memory usage observed over time, potentially using `kubectl top pod <pod-name> --containers` or metrics from a monitoring solution like Prometheus to identify if the container is consistently exceeding its allocated memory. To understand the application's memory consumption within the container, I would perform memory profiling. This could involve using tools specific to the application's language (e.g., `pprof` for Go, `jmap`/`jhat` for Java, `memory_profiler` for Python) to identify memory leaks or inefficient memory usage patterns. I might also enable detailed application logging for memory allocation and deallocation events. If the application's memory usage appears legitimate but the container is still OOMKilled, I would investigate potential memory fragmentation or the presence of memory-intensive background processes within the container that aren't directly part of the main application. This might involve exec'ing into a running container (if possible) and using tools like `top`, `htop`, or even `pmap` to analyze memory distribution. Finally, based on the findings, I would either increase the container's memory limit in the Kubernetes deployment if the workload genuinely requires more memory. If the issue is an application-level memory leak or inefficiency, I would work with the development team to optimize the application code. If memory fragmentation is suspected on the node, it might indicate a broader infrastructure issue requiring investigation by the platform team. Addressing OOMKilled events proactively involves setting appropriate resource requests and limits from the outset and implementing robust application-level memory profiling. Continuous monitoring of memory usage and alerting on deviations are crucial for preventing such issues from impacting production workloads.

Container OOMKilled Repeatedly

Ready to prove your Kubernetes administration knowledge?

Kubernetes Pod Stuck in Pending

Kubernetes Pod Stuck in Pending

Service in CrashLoopBackOff

Container Image Pulls Throttled by Registry

Docker vs Virtual Machines

EKS Cluster Upgrades: A Production Runbook for Platform Teams

From Terraform to GitOps: A Practical Migration Roadmap

GitOps: Monorepo vs Polyrepo - A Practical Comparison