k3s Disk Usage: 9 Steps to Find and Clean Unused Images

k3s disk usage can grow quietly on busy worker nodes even when application data and logs look normal. In this case, a 67 GB root filesystem reached 74% utilization, but the real cause was not Prometheus retention, pod logs, or a runaway persistent volume. It was a large containerd cache containing more than one hundred old CI/CD images.

This guide follows the exact troubleshooting sequence used across four k3s workers. The investigation identified unused container images, preserved real application data, and recovered about 28 GB across three nodes. It also explains why the default kubelet image garbage-collection thresholds had not acted yet and how to tune them conservatively on modern k3s releases.

1. Start with filesystem and inode usage

Do not begin by deleting files. First confirm whether the problem is capacity, inode exhaustion, or a separate mounted filesystem.

df -h /
df -ih /

On the first worker, the root filesystem reported approximately 47 GB used out of 67 GB, or 74%. Inodes were only 31% utilized. That immediately ruled out the classic “millions of tiny files” failure mode and pointed to a smaller number of large directories or files.

This distinction matters. If disk blocks are exhausted, you look for large content. If inodes are exhausted, directory and file counts become more important than their total size.

2. Find the largest directories without crossing mounts

Walk the filesystem from the top down. The -x flag keeps du on the current filesystem, which prevents mounted volumes from distorting the result.

sudo du -xhd1 / 2>/dev/null | sort -h
sudo du -xhd1 /var 2>/dev/null | sort -h
sudo du -xhd1 /var/lib 2>/dev/null | sort -h

In this incident, /var led to /var/lib, and the largest k3s-related locations were:

• /var/lib/rancher/k3s/agent/containerd: about 31 GB
• /var/lib/rancher/k3s/storage: about 7.8 GB
• /var/log/pods: about 117 MB
• /var/log/containers: negligible

The numbers already suggested that containerd was the leading candidate, but they did not yet prove that its content was safe to remove.

Be careful when interpreting /var/lib/kubelet. It frequently contains bind mounts for volumes that are also visible under another path. Adding every du total together can therefore count the same data twice. Use findmnt -R /var/lib/kubelet when totals appear larger than the filesystem usage reported by df.

3. Separate containerd, PVC data, and logs

A useful k3s disk usage audit treats three data classes differently:

• Container images and snapshots may be cache data, but must be removed through the runtime.
• Persistent volume data belongs to applications and must not be deleted as cleanup.
• Logs and journals can grow, but should be measured before retention is changed.

Measure each class explicitly:

sudo du -sh /var/lib/rancher/k3s/agent/containerd
sudo du -h --max-depth=1 /var/lib/rancher/k3s/storage 2>/dev/null | sort -h
sudo du -sh /var/log/pods 2>/dev/null
sudo du -sh /var/log/containers 2>/dev/null
journalctl --disk-usage

The local-path storage investigation found that roughly 7.5 GB belonged to a Prometheus database. Its retention was configured for ten days with a 25 GB size ceiling. That was legitimate monitoring data, not disposable cache, so it was deliberately left alone.

If the node does not have a usable kubeconfig, run Kubernetes inventory commands from a control-plane node or an administrator workstation:

kubectl get pvc -A
kubectl get pv -o wide

Map any large local-path directory to its PV and PVC before changing it. A directory name that looks unfamiliar is not evidence that the data is stale.

4. Inventory container images before cleanup

Once containerd was identified as the largest consumer, the next step was to inspect the image inventory rather than manually touching its files.

sudo k3s crictl images
sudo k3s crictl images -q | wc -l
sudo k3s crictl ps -a
sudo du -xhd2 /var/lib/rancher/k3s/agent/containerd 2>/dev/null | sort -h | tail -30

The first affected worker held 123 images. Most were old development and test tags produced by repeated CI/CD deployments. The running containers used only a small subset of those image IDs.

Never remove files directly from containerd directories such as overlayfs/snapshots or content/blobs. Runtime metadata, shared layers, and snapshot references must remain consistent. Manual deletion can leave containerd in a broken state even if it appears to free space.

5. Prune only unused images

Record a before-state first:

df -h /
sudo du -sh /var/lib/rancher/k3s/agent/containerd
sudo k3s crictl images -q | wc -l

Confirm that the installed crictl supports prune:

sudo k3s crictl rmi --help | grep -i prune

Then ask the CRI runtime to remove images that are not used by containers:

sudo k3s crictl rmi --prune

Run the same measurements again:

df -h /
sudo du -sh /var/lib/rancher/k3s/agent/containerd 2>/dev/null
sudo k3s crictl images -q | wc -l

During active cleanup, du may briefly report that a snapshot directory disappeared while it was scanning. That can happen because containerd removed the directory after du discovered it but before the scan reached it. Wait a minute and repeat the command; do not treat the transient message as permission to delete more files manually.

6. Verify the result on every worker

The same audit was performed on four workers. Three had accumulated large numbers of old images; one was already healthy and required no cleanup.

Worker	Used before	Used after	Recovered	Images before → after
worker-a	47 GB / 74%	36 GB / 56%	about 11 GB	123 → 14
worker-b	42 GB / 67%	32 GB / 50%	about 10 GB	142 → 14
worker-c	30 GB / 47%	23 GB / 36%	about 7 GB	72 → 12
worker-d	13 GB / 14%	unchanged	not needed	15

The total recovery was about 28 GB. More importantly, the comparison proved that the growth pattern came from stale image cache, not from application PVCs or logging.

7. Understand why kubelet GC waited

The cache could grow because none of the affected nodes had crossed the default high threshold. Kubernetes documents a default imageGCHighThresholdPercent of 85 and a default imageGCLowThresholdPercent of 80. Image garbage collection starts above the high threshold and removes the least recently used images until usage reaches the low threshold.

A node at 74% can therefore contain more than one hundred old deployment images while kubelet still considers the filesystem below its trigger point. That behavior is expected, not evidence that garbage collection is disabled.

The official Kubernetes garbage collection documentation also advises against external garbage-collection tools for routine maintenance because they can interfere with kubelet-managed lifecycle behavior.

8. Tune k3s disk usage with image garbage collection

For relatively small worker root disks, waiting until 85% can leave too little operational headroom. A conservative starting point for this environment was 75% high and 65% low.

On k3s v1.32 and newer, the official k3s configuration documentation recommends kubelet drop-in files under:

/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/

Inspect the directory first and leave 00-k3s-defaults.conf unchanged. Add a later drop-in such as 10-image-gc.conf:

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageGCHighThresholdPercent: 75
imageGCLowThresholdPercent: 65

The official KubeletConfiguration reference requires the high value to be greater than the low value. Apply and verify the change one worker at a time:

sudo systemctl restart k3s-agent
sudo systemctl status k3s-agent --no-pager

From an administrator workstation, confirm that the node returns to Ready before moving to the next worker:

kubectl get nodes

Kubernetes also provides imageMaximumGCAge for age-based collection. Its default is 0s, which disables age-based cleanup. That option can suit fast-moving development clusters, but it was intentionally deferred here. Threshold tuning is easier to observe and less aggressive; age-based cleanup can be added later if old images still accumulate.

9. Turn the fix into a safe runbook

A repeatable k3s disk usage runbook should contain four phases:

1. Measure: record filesystem, inode, containerd, PVC, and log usage.
2. Classify: separate cache from persistent application data and bind-mounted duplicates.
3. Clean safely: use the CRI runtime for a one-time unused-image prune; never delete containerd internals.
4. Prevent recurrence: tune kubelet thresholds, restart one node at a time, and monitor the trend.

Disk maintenance is also part of recovery readiness. Before changing node configuration, verify that the cluster has a credible backup path. The k3s etcd backup audit covers the control-plane side, while Kubernetes disaster recovery needs more than an etcd snapshot explains why PVCs and application-native backups remain separate concerns. Add the same measurements to routine monitoring so that k3s disk usage trends are visible before a node reaches eviction pressure.

The important lesson from this incident is simple: a nearly full k3s worker does not automatically mean “delete logs” or “expand the disk.” Trace usage from the filesystem downward, preserve real data, let containerd remove its own unused images, and tune kubelet only after the cause is proven.