How to Diagnose and Safely Fix Disk Space and Inode Exhaustion on Linux

Few Linux incidents feel more urgent than a server that suddenly refuses to write files. A web application may return 500 errors, SSH sessions may behave strangely, or a package upgrade may fail with a message such as No space left on device. The confusing part is that a quick df -h check can show several gigabytes free while the system still cannot create a small file.

This guide shows how to diagnose both common causes: a filesystem that has run out of storage blocks and one that has run out of inodes. The commands work across Ubuntu and Debian, RHEL and Fedora, and AlmaLinux, with clearly marked differences where package names or service paths vary. The goal is not merely to delete files, but to find the responsible workload, recover safely, and reduce the chance of a repeat incident.

TL;DR

• Check both storage capacity and inode capacity with df -hT and df -ih.
• Use du to locate large directories, but remember that deleted files held open by a process do not appear in directory totals.
• Check journal, log, cache, temporary, backup, and container data before removing anything.
• Use lsof +L1 to find deleted-but-open files, then restart the owning service during an approved maintenance window.
• Verify recovery with df, df -i, a test file, and the affected application.

Why disk and inode exhaustion matters

Linux filesystems store file content in data blocks and file metadata in inodes. A filesystem can therefore fail in two different ways. When blocks are full, large files or databases cannot grow. When inodes are full, even a zero-byte file cannot be created. Mail queues, session files, cache entries, and millions of tiny log files are frequent inode consumers.

Running at 100 percent is risky even when a service still appears healthy. Databases may fail to write transaction logs, systemd may be unable to record state, and package managers can leave a partial transaction. If the root filesystem is affected, avoid rebooting until you understand the situation; a reboot can expose problems that were hidden while processes still held open files.

Prerequisites and safety rules

• Have a shell with sudo access. Root access may be needed to inspect other users’ directories.
• Know the server’s maintenance and backup policy. Confirm that logs or backups can be removed or rotated.
• Capture evidence before changing anything: filesystem output, top directories, service status, and recent application errors.
• Use a production change window before restarting services or deleting data.

1. Diagnose before applying a fix

Start by identifying mounted filesystems, their types, and usage percentages:

df -hT

A typical result resembles:

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sda2      ext4   40G   38G  1.1G  98% /
tmpfs          tmpfs  2G     0    2G   0% /run/user/1000

The -h flag makes sizes readable, while -T displays the filesystem type. Focus on the mount point whose Use% is high. A full /var mount requires a different investigation from a full root filesystem.

Next check inode usage:

df -ih

Example:

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/sda2        2.5M  2.5M     0  100% /

Here, the block percentage may be low, but IUse% is 100 percent. That points to too many files rather than too much file content. If both values are high, solve the largest and most urgent issue first, while documenting every deletion.

Check whether the kernel reported a related event:

dmesg -T | tail -n 40
journalctl -p err..alert --since today --no-pager

dmesg -T prints kernel messages with readable timestamps. journalctl -p err..alert limits systemd journal entries to error through alert priority, and --no-pager prevents an interactive pager. Permission errors are normal on hardened systems; use sudo if required.

2. Locate large directories and files

For a full root filesystem, inspect one directory level at a time:

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /var 2>/dev/null | sort -h

In this command, du totals directory usage, -x stays on one filesystem, -h uses readable units, and -d1 limits the report to one level. Redirecting standard error hides routine permission warnings; do not use that redirection if you need a complete audit trail. sort -h sorts values such as 900M and 12G correctly.

Once you find a suspicious directory, descend into it. For example:

sudo du -xhd1 /var/log 2>/dev/null | sort -h
sudo du -xhd1 /var/lib 2>/dev/null | sort -h

To list unusually large regular files on the root filesystem, use:

sudo find / -xdev -type f -size +500M -printf '%s %p\n' 2>/dev/null | sort -n | tail -n 20

-xdev prevents crossing into other mounts, -type f excludes directories, and -size +500M selects files larger than 500 megabytes. The printed byte count is sortable, while the path identifies the candidate. A large database file is not automatically disposable; inspect its service and documentation first.

3. Check for deleted files still consuming space

A common surprise is a log file that was deleted while a process still had it open. Directory tools cannot see it, but the blocks remain allocated until the process closes the file. Search for such files:

sudo lsof +L1

Look for a large SIZE/OFF value and a path ending in (deleted). A result might look like:

COMMAND  PID USER   FD   TYPE DEVICE  SIZE/OFF NLINK NAME
java    2148 app    12w  REG  252,2 4294967296     0 /var/log/app.log (deleted)

The file is still held by PID 2148. Do not attempt to remove the displayed path again; it no longer exists. Find the owning service:

ps -p 2148 -o pid,ppid,user,cmd
sudo systemctl status your-service --no-pager

Restart the correct service only after confirming its identity and impact:

sudo systemctl restart your-service
sudo lsof -p 2148 2>/dev/null

The process ID may change after a restart. Re-run lsof +L1 and df -h to confirm the space returned. If a restart is not acceptable, ask the application owner whether it supports reopening logs, often through a documented signal or administrative command.

4. Safe cleanup by category

Systemd journal logs

Measure journal usage before reducing it:

sudo journalctl --disk-usage

To retain only the last seven days:

sudo journalctl --vacuum-time=7d

Alternatively, cap the journal by size:

sudo journalctl --vacuum-size=500M

These commands remove old journal files, not current entries. Retention may be required for compliance, so confirm policy first. To make a persistent limit, edit the systemd-journald configuration and restart the service:

sudoedit /etc/systemd/journald.conf
sudo systemctl restart systemd-journald

Set an appropriate value such as SystemMaxUse=1G, then verify with journalctl --disk-usage.

Package caches: Ubuntu and Debian

On Ubuntu and Debian, inspect and clean downloaded package archives:

sudo du -sh /var/cache/apt
sudo apt-get clean

apt-get clean removes cached package files. It does not remove installed software. Use sudo apt-get autoremove only after reviewing its proposed package list; it can remove packages no longer marked as dependencies.

Package caches: RHEL, Fedora, and AlmaLinux

Modern RHEL, Fedora, and AlmaLinux systems use DNF:

sudo du -sh /var/cache/dnf
sudo dnf clean all

Older installations may use YUM, where sudo yum clean all is the equivalent. Package cache cleanup is generally low risk, but keep repository and package-management evidence if the server is under audit.

Temporary files and application data

Systemd-managed temporary cleanup can be triggered with:

sudo systemd-tmpfiles --clean

Do not blindly empty /tmp. An active process may rely on a temporary file, and some applications use predictable names or sockets there. For application caches, use the application’s documented cleanup command. For container hosts, inspect usage before removing images:

sudo du -sh /var/lib/docker /var/lib/containers 2>/dev/null
sudo docker system df 2>/dev/null
sudo podman system df 2>/dev/null

Use Docker or Podman prune commands only after confirming that unused images, stopped containers, and unneeded volumes are genuinely disposable. Volumes can contain production data.

5. Find and resolve inode exhaustion

To locate directories containing many entries, count files at a controlled depth. Start with likely locations:

for d in /var /tmp /home; do
  printf '%s ' "$d"
  sudo find "$d" -xdev -type f 2>/dev/null | wc -l
done

This loop prints an approximate regular-file count. For a more focused directory, use:

sudo find /var/spool -xdev -type f -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -n | tail -n 20

Large counts in mail queues, session directories, cache trees, or application upload paths should be investigated with the responsible team. Delete through the application’s queue or retention mechanism where possible. Removing files directly can leave inconsistent metadata or cause duplicate processing.

6. Verify the repair

After each change, verify rather than assuming success:

df -hT
df -ih
sudo du -xhd1 /var 2>/dev/null | sort -h
sudo systemctl is-active your-service
sudo -u nobody sh -c 'touch /var/tmp/disk-check && rm /var/tmp/disk-check'

The service should report active, both usage percentages should have a safe margin, and the test user should be able to create and remove a file in the selected filesystem. Use an application-specific health check as well, such as a local HTTP request or a database connection test. Monitor logs for several minutes:

sudo journalctl -u your-service --since '10 minutes ago' --no-pager

Fallback troubleshooting

• Space did not return: run sudo lsof +L1. A deleted-open file is the most likely explanation.
• du is smaller than df: compare mount points with findmnt; also check deleted-open files and hidden files beneath a mount point.
• Cleanup says permission denied: inspect with sudo, but do not change ownership just to make a command work.
• The filesystem is read-only: run findmnt -no TARGET,OPTIONS /mount/path and inspect dmesg -T for I/O or filesystem errors. Do not force a repair on a mounted production filesystem.
• Inodes remain full: count directories rather than file sizes and check mail queues, application sessions, cache fan-out, and failed job output.
• The filesystem fills again: inspect rotation, retention, runaway debug logging, backups, core dumps, and container image policies instead of repeating manual deletion.

Common mistakes and best practices

The most damaging mistake is treating every large file as junk. Another is cleaning only the root filesystem while a separate /var mount is full. Running du / without -x can also produce misleading totals by crossing into network, container, or backup mounts.

Use log rotation rather than periodic deletion. On Ubuntu and Debian, inspect /etc/logrotate.conf and /etc/logrotate.d/; the same locations are common on RHEL, Fedora, and AlmaLinux. Test configuration with sudo logrotate -d /etc/logrotate.conf; the -d flag performs a dry run. For long-term reliability, alert at 70 or 80 percent block usage, alert separately on inode usage, reserve capacity for emergencies, and document ownership for every large data path.

Incident checklist

Conclusion

Full Linux filesystems are easier to resolve when you separate diagnosis from cleanup. First determine whether blocks or inodes are exhausted, then identify the exact mount and responsible workload. Clean package caches or retained logs only when policy allows it, handle deleted-open files through the owning process, and verify both operating-system and application health afterward. For related operational guidance, see AdminHint’s log rotation guide, systemd service troubleshooting guide, and Linux server monitoring basics.

FAQ

Why does df show more usage than du?

Usually because a process still has a deleted file open, or because du crossed different mount boundaries or skipped inaccessible content. Check lsof +L1, mount points, and permissions.

Is deleting journal files safe?

Vacuuming old journal entries is normally safe operationally, but it may violate retention or forensic requirements. Confirm the required retention period before using a vacuum command.

Should I reboot a full Linux server?

Not as a first response. Rebooting may release deleted-open files, but it can also expose boot, filesystem, or service failures. Recover space and identify the cause before rebooting whenever possible.

Which command fixes inode exhaustion?

There is no universal fix. Find the directory containing millions of small files, then use the application’s queue, cache, session, or retention mechanism. Deleting files blindly can cause data loss.