What goes into a storage
Every storage has a Content list — what it may hold. Proxmox sorts files into subdirectories itself, and in the UI each type is its own tab on the storage.
| Content | What it is | Where it lives in the local directory |
|---|---|---|
Disk image (images) |
virtual machine disks | /var/lib/vz/images/<id>/ |
Container (rootdir) |
container root disks | /var/lib/vz/images/<id>/ |
ISO image (iso) |
installation images | /var/lib/vz/template/iso/ |
Container template (vztmpl) |
container templates and OCI images | /var/lib/vz/template/cache/ |
VZDump backup file (backup) |
vzdump backups | /var/lib/vz/dump/ |
Snippets (snippets) |
your own cloud-init files and hook scripts | /var/lib/vz/snippets/ |
Import (import) |
images to import: cloud images, OVA, disks from ESXi | /var/lib/vz/import/ |
Since 9.1 the installer enables the import type on local: put the Ubuntu cloud image there and make a VM disk
from it (VM template with cloud-init).
What the installer gives you
The layout is chosen in the installer, with the Options button next to the disk (Installing Proxmox). Two main variants:
ext4 (or xfs) on one disk ZFS (RAID0, RAID1 mirror, RAIDZ…)
LVM, volume group pve pool rpool
├─ root ext4, the host's "/" ├─ rpool/ROOT/pve-1 the host's "/"
│ └─ /var/lib/vz → storage local ├─ /var/lib/vz → storage local
├─ swap └─ rpool/data → storage local-zfs
└─ data thin pool → storage local-lvm
local is always there: it’s a directory on the host’s root, for files. Guest disks go to the second storage:
local-lvm on ext4 or local-zfs on ZFS. Each guest disk there is a separate volume: vm-101-disk-0.
Storage types
| Type | What it is | Snapshots | Thin disks | Shared in a cluster | At home — for what |
|---|---|---|---|---|---|
| LVM-thin | an LVM thin pool on a disk | yes | yes | no | guest disks on a single SSD |
| ZFS (local) | a ZFS pool: volumes and datasets | yes | yes | no, but replicates to another node | guest disks on a mirror of two SSDs |
| Directory | a folder on any file system | qcow2 only | qcow2 and raw | no | ISOs, templates, backups on a second disk |
| NFS, SMB/CIFS | a NAS share | qcow2 only | qcow2 | yes | backups and ISOs on a NAS |
| LVM (plain) | thick LVM volumes | preview since 9.0 | no | yes, on a shared SAN | rarely needed at home |
| Ceph RBD | distributed storage on cluster nodes | yes | yes | yes | three nodes and a fast network or more |
| Proxmox Backup Server | a separate backup server | — | — | yes | backups only, with deduplication |
A thin disk takes space as data is written: the VM was given 100 GB, wrote 12 — the pool lost 12. So guests can be given more than there is space. It works until the guests all start writing at once: a full pool stops every guest on it (see “If it doesn’t work”). Space a guest frees on its side returns to the pool only with discard (Disk discard).
qcow2 or raw. On a directory or NFS a VM disk is a file. qcow2 does snapshots and thin provisioning itself; raw is a few percent faster, but raw on a directory has no snapshots. On LVM-thin and ZFS the format is always raw: the storage itself provides snapshots and thin provisioning.
ext4 with LVM-thin, or ZFS
ext4 and LVM-thin
one disk- the default layout, nothing to know in advance
- only the kernel’s regular page cache takes memory for caching
- snapshots and thin disks are there
- no checksums: only the guest notices a corrupted block
- no mirror in the installer: when the disk dies, restore from backup
ZFS
better with two disks or more- a mirror or RAIDZ right in the installer
- a checksum on every block; in a mirror a corrupted block is repaired from the other disk
- compression is on (
compression=on, i.e. lz4): systems and text compress noticeably, video and archives don’t - snapshots, replication to another node,
zfs sendfor copies - asks for memory for ARC and writes more to the SSD than ext4
The rule of thumb is simple. One SSD, 8–16 GB of memory — ext4 and LVM-thin. Two identical SSDs and 16 GB of memory or more — a ZFS mirror: one disk dying stops being an event. For ZFS, use SSDs with write endurance to spare (TBW) and no hardware RAID (Hardware for Proxmox).
ZFS takes memory for the ARC cache. Since 8.1 the installer writes a limit to /etc/modprobe.d/zfs.conf: 10 % of
memory but no more than 16 GiB. On a 16 GB host that’s 1.6 GB — the guests keep almost everything.
Snapshots
A snapshot freezes the state of the guest’s disk: from then on only changes are written, and a rollback returns the disk to the moment of the snapshot. For a VM, memory can be included in the snapshot too (Include RAM) — after a rollback the VM continues from the same point instead of booting again.
How snapshots work depends on the storage:
- LVM-thin — a snapshot becomes a separate thin volume with shared blocks. You can roll back to any snapshot.
- ZFS — native ZFS snapshots: created instantly and cost almost nothing. Rollback only to the latest one.
- qcow2 — snapshots inside the disk file. A long chain of them slows the disk down.
A snapshot lives on the same storage as the disk. When the disk dies, the snapshots are gone, which is why a snapshot doesn’t count as a backup (A snapshot isn’t a backup).
A second disk for backups and ISOs
Connect an empty disk from the UI: node › Disks › Directory › Create Directory. Proxmox creates the file system, a
systemd mount unit and a directory storage itself. In Content pick backup, iso, vztmpl.
The same for a ZFS disk: node › Disks › ZFS › Create: ZFS, and the pool immediately becomes a storage for guests.
A NAS share — Datacenter › Storage › Add › NFS or SMB/CIFS. Proxmox mounts it itself at /mnt/pve/<name> and
remounts it after a reboot. In the console pvesm does the same:
pvesm add nfs nas-backup --server 10.0.0.20 --export /volume1/proxmox --content backup,iso
pvesm status # list of storages, type, used and free
If it doesn’t work
VMs froze all at once or report I/O errors, and lvs shows Data% 100.00 on data
- Cause
- The
local-lvmthin pool is full: guests were given more disk than the pool has, and they used it up. Snapshots are another cause: they hold old blocks. - Fix
- Free up space: delete old snapshots, enable discard and run
fstrim -avin the guests (Disk discard). Watchlvs pve/data— keepData%andMeta%under 80–90 %.
Snapshot rollback on ZFS: can't rollback, 'before-upgrade' is not most recent snapshot
- Cause
- ZFS rolls back only to the latest snapshot: newer ones would have to be deleted, and Proxmox doesn’t do that on its own.
- Fix
- Delete the snapshots newer than the one you need (Snapshots › Remove) and roll back. LVM-thin and qcow2 have no such limit.
Can’t upload an ISO to local-lvm — there’s no ISO Images tab
- Cause
- A storage accepts only the content types enabled in its Content. An LVM thin pool holds only disks; it never contains files.
- Fix
- ISOs, templates and backups go to a directory storage:
localor your own on a second disk. What a storage accepts — Datacenter › Storage, the Content column.
The UI throws errors, and df -h / shows 100 %
- Cause
- The
localstorage is a directory on the host’s root file system. ISOs, templates and backups in it filled the whole root, and thepve-clusterservice can’t write the/etc/pvedatabase. - Fix
- Delete what’s unneeded from
/var/lib/vz/dumpand/var/lib/vz/template/iso. From then on, backups go to a separate disk or a NAS (Backups in Proxmox).
A ZFS host has almost all memory in use, though guests were given less than half
- Cause
- ZFS keeps its ARC cache in memory. Since 8.1 the installer limits it to 10 % of memory but no more than 16 GiB; hosts installed earlier have no limit in
/etc/modprobe.d/zfs.conf. - Fix
- Cache size —
arc_summary -s arc. A limit of, say, 4 GiB:options zfs zfs_arc_max=4294967296in/etc/modprobe.d/zfs.conf, thenupdate-initramfs -u -k alland a reboot.