How to Diagnose and Safely Fix Full Linux Disks and Inodes

Few Linux alerts are more alarming than No space left on device. The message may appear while a database writes a record, SSH creates a session, or a package update is halfway through. Sometimes the disk really is full; sometimes the filesystem has run out of inodes, or a deleted log is still held open by a running process.

This guide gives beginners and intermediate sysadmins a cautious workflow: measure first, identify the filesystem and responsible directory, reclaim only known-safe data, and verify that the service and filesystem recover. The commands apply to Ubuntu and Debian, RHEL and Fedora, and AlmaLinux, with package-manager differences called out explicitly.

TL;DR

1. Run df -hT and df -ih to distinguish byte usage from inode exhaustion.
2. Use du to locate large directories, but stay on the affected filesystem with -x.
3. Check systemd journal usage, package caches, old files, and deleted files still held open.
4. Do not blindly delete files from /var/lib, /usr, databases, or active application directories.
5. Re-run the checks, inspect service health, and create a preventive retention policy.

Why this matters

A full root filesystem can prevent authentication, logging, package installation, temporary-file creation, and service restarts. Inode exhaustion is less obvious: a filesystem can report free gigabytes while refusing to create a tiny file because its file-entry supply is exhausted. The remedy depends on the diagnosis, so deleting random files is a poor first response.

For broader incident habits, pair this procedure with AdminHint’s systemd service troubleshooting guide and Linux log-file guide.

Prerequisites and safety rules

• Use an account with sudo privileges. A root shell is not required.
• Know whether the machine hosts a database, container runtime, web application, or other stateful service.
• Have a current backup before removing application data.
• Record the initial measurements and the files you intend to change.

1. Diagnose before changing anything

Start with the filesystem summary. The -h flag makes sizes human-readable, and -T displays the filesystem type.

df -hT

Typical output looks like this:

Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/vda2      ext4   40G   39G  600M  99% /
tmpfs          tmpfs  2G     0    2G   0% /run/user/1000

Focus on a mount at 90 percent or higher. A separate /var, /home, or data mount changes where you investigate. Do not assume that a large directory shown under /var consumes root space if /var is mounted separately.

Now check inode consumption:

df -ih

Example:

Filesystem     Inodes IUsed IFree IUse% Mounted on
/dev/vda2       2.5M  2.5M     0  100% /

If IUse% is 100 percent while df -hT still shows free space, find directories containing enormous numbers of small files. Mail queues, cache trees, temporary upload folders, and application session directories are common causes.

Also check memory-backed temporary filesystems. A full /run or /dev/shm is a different incident from a full disk:

df -h / /var /tmp /run /dev/shm 2>/dev/null

2. Locate the largest directories safely

For a full root filesystem, inspect only one level at a time. The -x option prevents du from crossing into other mounted filesystems, -s produces one total per argument, and -h uses readable units.

sudo du -xhd1 / 2>/dev/null | sort -h

You may see:

2.1G    /usr
8.7G    /home
25G     /var
39G     /

Continue down the largest path. For example:

sudo du -xhd1 /var 2>/dev/null | sort -h
sudo du -xhd1 /var/log 2>/dev/null | sort -h
sudo du -xhd1 /var/lib 2>/dev/null | sort -h

If the problem is inode exhaustion, count entries rather than sorting by byte size. This command reports immediate directory counts beneath /var:

sudo find /var -xdev -mindepth 1 -maxdepth 1 -type d -print0 \
  | sudo xargs -0 -r -n1 sh -c 'printf "%8s %s\n" "$(find "$1" -xdev -type f 2>/dev/null | wc -l)" "$1"' sh \
  | sort -n

The command is intentionally more expensive than du. Run it during a maintenance window on very large trees. A high count under a known cache or queue directory is a useful lead, but do not delete it until you understand which service owns it.

3. Check logs and the systemd journal

Logs are valuable during diagnosis, so inspect them before cleanup. List large regular files in the log directory:

sudo find /var/log -xdev -type f -printf '%s %p\n' 2>/dev/null \
  | sort -n | tail -20 \
  | numfmt --field=1 --to=iec

A journal using several gigabytes can be reduced with journalctl. First measure it:

sudo journalctl --disk-usage

Expected output:

Archived and active journals take up 3.2G in the file system.

Before deleting historical entries, preserve any records required by your incident, security, or compliance process. To retain only 500 megabytes of journal data:

sudo journalctl --vacuum-size=500M

Alternatively, retain the most recent seven days:

sudo journalctl --vacuum-time=7d

These options remove archived journal files; the currently active journal is not arbitrarily truncated. Verify the result:

sudo journalctl --disk-usage
sudo systemctl is-active systemd-journald

To prevent recurrence, inspect journal configuration:

sudo grep -R --line-number --no-messages '^SystemMaxUse\|^RuntimeMaxUse\|^MaxRetentionSec' /etc/systemd/journald.conf /etc/systemd/journald.conf.d 2>/dev/null

If no policy exists, create a drop-in rather than editing the vendor file:

sudo install -d -m 0755 /etc/systemd/journald.conf.d
sudo tee /etc/systemd/journald.conf.d/size-limit.conf >/dev/null <<'EOF'
[Journal]
SystemMaxUse=1G
SystemMaxFileSize=128M
EOF
sudo systemctl restart systemd-journald

Adjust the limit to the machine’s disk size and retention requirements. A small server may need much less than 1G; a production system may need centralized log forwarding instead of aggressive local deletion.

4. Check package-manager caches

Package caches are generally safer to clean than application data, but retain packages if your recovery process depends on offline reinstallations.

Ubuntu and Debian

sudo apt-get clean
sudo apt-get autoclean

clean removes downloaded package archives from the local cache. autoclean removes only archives that can no longer be downloaded. Verify cache size with:

sudo du -sh /var/cache/apt/archives

RHEL, Fedora, and AlmaLinux

sudo dnf clean packages
sudo dnf clean all

dnf clean packages removes cached RPM packages. dnf clean all also removes repository metadata and is more thorough; the metadata will be downloaded again. Older RHEL or CentOS-family systems may use yum:

sudo yum clean all

Do not confuse cache cleanup with removing installed packages. Never use apt autoremove or dnf autoremove automatically during an outage without reviewing the proposed package list.

5. Find deleted files that still consume space

Removing a log does not immediately free its blocks if a running process still has the file open. Search for deleted-open files:

sudo lsof +L1

Look for a large file whose name ends in (deleted). The SIZE/OFF column gives a rough size and COMMAND identifies the process. If a web server or application owns it, use that service’s supported log-reopen method. For a systemd service, a controlled restart may release the space:

sudo systemctl restart example.service
sudo systemctl status example.service --no-pager
sudo df -hT /

6. Handle temporary files and old data carefully

Systemd’s temporary-file policy can show what it manages:

sudo systemd-tmpfiles --cat-config | less
sudo systemd-tmpfiles --clean --dry-run

If the dry run is acceptable, run the cleanup without --dry-run:

sudo systemd-tmpfiles --clean

The exact behavior depends on age rules in configuration files. Do not manually empty /tmp while installers, editors, sockets, or active jobs are running. For application uploads or backups, identify ownership first:

sudo stat /path/to/suspect-file
sudo ls -lah /path/to/suspect-directory
sudo find /path/to/suspect-directory -xdev -type f -mtime +30 -print

find‘s -mtime +30 selects files whose modification time is older than approximately 30 days. Review the list, confirm the retention policy, and archive or remove only approved data. If you need to delete a reviewed list, prefer an interactive command:

sudo find /path/to/suspect-directory -xdev -type f -mtime +30 -ok rm {} \;

The -ok prompt requires confirmation for each file. For large, approved cleanups, use the application’s retention tool or a tested script rather than an unreviewed one-liner.

7. Verify recovery

After each cleanup category, measure again instead of assuming success:

df -hT
df -ih
sudo journalctl --disk-usage
sudo systemctl --failed

Write a harmless test file to the affected filesystem and remove it:

sudo sh -c 'printf "disk check\n" > /var/tmp/adminhint-disk-check'
sudo cat /var/tmp/adminhint-disk-check
sudo rm -f /var/tmp/adminhint-disk-check

Then verify the affected service. Replace the example unit with the real service:

sudo systemctl is-active example.service
sudo journalctl -u example.service -n 50 --no-pager

Expected healthy output from is-active is:

active

If a service failed while the filesystem was full, it may need a restart after space is available. Check application-level health as well: make a test request, confirm a database query succeeds, or verify that a scheduled job can create its output.

Fallback troubleshooting

There is no obvious large directory

Recheck mount boundaries with findmnt:

findmnt -D
findmnt /

A deleted-open file, reserved filesystem blocks, or a file hidden beneath a later-mounted filesystem can explain the difference between du and df. On ext4, reserved blocks may be visible to root but unavailable to ordinary users. Inspect them, but change the reservation only after confirming the filesystem and capacity plan:

sudo tune2fs -l /dev/vda2 | grep -E 'Block count|Reserved block count|Reserved block percentage'

Never run tune2fs against the wrong device. Use lsblk -f to confirm the device first.

Inodes are full

Find the directory containing millions of small files, then use the owning application’s queue, cache, or retention command. If the files are abandoned and backed up, remove them in batches during a controlled window. A filesystem cannot gain inodes without migration or reformatting, so long-term fixes may include a different filesystem layout or moving high-churn data to a dedicated volume.

The machine cannot boot normally

Use a rescue environment or emergency target, mount the affected filesystem read-write only after checking it, and remove one confirmed disposable item such as an old package cache or archived journal. Do not run filesystem repair on a mounted, active filesystem. For ext filesystems, use fsck from appropriate rescue media and follow your distribution’s recovery procedure.

Common mistakes

• Using du / without -x, which counts other mounts and misleads the investigation.
• Deleting the current log instead of rotating it, leaving the process holding deleted space.
• Assuming free bytes mean free inodes.
• Running package autoremove without reviewing dependencies.
• Deleting Docker, Podman, database, or virtual-machine data without checking its owner.
• Changing log retention without considering audit, legal, or incident-response requirements.
• Cleaning space but failing to fix the producer, such as a noisy service or missing log rotation.

Best practices and quick checklist

1. Alert before a filesystem reaches a crisis threshold, such as 80 percent.
2. Monitor both bytes and inodes.
3. Configure journald limits and verify logrotate policies.
4. Send important logs to a central system.
5. Give databases, container storage, backups, and uploads dedicated capacity where practical.
6. Test restoration and document which data may be removed during an outage.

Conclusion

Full-disk incidents become manageable when you separate measurement from remediation. df tells you which resource is exhausted, du and targeted searches identify the consumer, and service-aware cleanup prevents collateral damage. After reclaiming space, verify filesystem writes, service state, journal behavior, and the original application workflow. Prevention—sensible retention, monitoring, and dedicated storage—is more reliable than emergency deletion.

FAQ

Should I delete everything in /var/log?

No. Keep evidence and let logrotate or journald manage retention. If one log is huge, identify its service, rotate or reopen it safely, and investigate why it grew.

Why does df show more space used than du?

Common causes include deleted-open files, other mount points, reserved blocks, or files hidden beneath a mount. Check lsof +L1, findmnt, and the filesystem details.

Is apt-get clean safe?

It removes cached package archives, not installed packages. It is usually low risk, but offline recovery procedures may depend on those archives.

Can a full filesystem damage data?

It can interrupt writes and leave applications in an inconsistent state. Restore capacity first, then inspect service logs and application integrity before declaring recovery complete.