Inode Exhaustion on Filesystem

Question

Disk usage looks low, but new files cannot be created and services fail with ENOSPC. How do you confirm inode exhaustion and fix it?

Accepted Answer

This is a critical situation indicating a lack of available inodes, which are essential for tracking file system objects. The first step is to confirm the problem precisely, followed by targeted actions to free up inodes and prevent recurrence. Confirming Inode Exhaustion: I would immediately use the `df -i` command. This will display inode usage for all mounted file systems. I'll look for partitions where 'IUse%' is at or near 100%. This command provides a clear, definitive confirmation of inode depletion. Identifying Directories with Excessive Files: Once confirmed, the next step is to pinpoint where these inodes are being consumed. I'd use a combination of `find` commands, such as `find /path/to/mountpoint -xdev -type f | wc -l` to count files recursively in suspected areas. A more granular approach would be `find /path/to/mountpoint -xdev -printf '%h ' | sort | uniq -c | sort -nr | head -n 20` to identify directories containing the most files. Immediate Cleanup of Temporary and Cache Data: The most common culprits for inode exhaustion are directories accumulating many small files, such as temporary directories (`/tmp`, `/var/tmp`), application caches, and log directories. I would prioritize cleaning these out by removing old or unnecessary files. This often involves using `rm -rf` with caution, or more controlled scripts to delete files older than a certain age. Log Rotation and Management: Specifically for log files, I would check the `logrotate` configuration (`/etc/logrotate.conf` and files in `/etc/logrotate.d/`). Ensuring log rotation is properly configured and active is crucial. If logs are growing excessively and not being rotated or compressed, this can quickly consume inodes. I might manually trigger rotation or adjust configurations if needed. Strategies for Bundling and Application Behavior Adjustments: If the issue stems from applications creating a high volume of very small, short-lived files (e.g., session files, temporary data), I'd investigate application-specific settings. This could involve adjusting application configurations to bundle smaller files into larger archives or to reduce the frequency of file creation. For critical systems, I might also consider archiving or consolidating existing directories with a very large number of small files if direct deletion isn't an option. Effectively diagnosing and resolving inode exhaustion requires both immediate corrective actions and proactive monitoring. Regularly reviewing inode usage with `df -i` and implementing robust log rotation and cache management policies are key to preventing future occurrences.

Inode Exhaustion on Filesystem

Node Disk Full

du and df Show Different Disk Usage

Check Disk Space in Linux

SSH Keys vs Password Authentication

Find Top CPU Processes in Linux

Real-Time Monitoring with eBPF: Low-Overhead Insights for Linux & K8s

Building a Security Operations Center (SOC) That Actually Works: A Practical Guide for Modern Organizations