basforge

Search titles, text and the “If it doesn’t work” cases.

    RU
    OverviewHomelabProxmox

    Proxmox storage: local, local-lvm, ZFS — what lives where and what it can do

    A storage in Proxmox is where guest disks, ISOs, templates and backups go. After an ext4 install there are two: local — the /var/lib/vz directory for files — and local-lvm — an LVM thin pool for guest disks. What a storage can do — snapshots, thin disks, shared access from a cluster — is decided by its type, not by the disk underneath.

    What goes into a storage

    Every storage has a Content list — what it may hold. Proxmox sorts files into subdirectories itself, and in the UI each type is its own tab on the storage.

    Content What it is Where it lives in the local directory
    Disk image (images) virtual machine disks /var/lib/vz/images/<id>/
    Container (rootdir) container root disks /var/lib/vz/images/<id>/
    ISO image (iso) installation images /var/lib/vz/template/iso/
    Container template (vztmpl) container templates and OCI images /var/lib/vz/template/cache/
    VZDump backup file (backup) vzdump backups /var/lib/vz/dump/
    Snippets (snippets) your own cloud-init files and hook scripts /var/lib/vz/snippets/
    Import (import) images to import: cloud images, OVA, disks from ESXi /var/lib/vz/import/

    Since 9.1 the installer enables the import type on local: put the Ubuntu cloud image there and make a VM disk from it (VM template with cloud-init).

    What the installer gives you

    The layout is chosen in the installer, with the Options button next to the disk (Installing Proxmox). Two main variants:

    ext4 (or xfs) on one disk                 ZFS (RAID0, RAID1 mirror, RAIDZ…)
    LVM, volume group pve                     pool rpool
      ├─ root  ext4, the host's "/"             ├─ rpool/ROOT/pve-1  the host's "/"
      │    └─ /var/lib/vz  → storage local      ├─ /var/lib/vz       → storage local
      ├─ swap                                   └─ rpool/data        → storage local-zfs
      └─ data  thin pool → storage local-lvm

    local is always there: it’s a directory on the host’s root, for files. Guest disks go to the second storage: local-lvm on ext4 or local-zfs on ZFS. Each guest disk there is a separate volume: vm-101-disk-0.

    Storage types

    Type What it is Snapshots Thin disks Shared in a cluster At home — for what
    LVM-thin an LVM thin pool on a disk yes yes no guest disks on a single SSD
    ZFS (local) a ZFS pool: volumes and datasets yes yes no, but replicates to another node guest disks on a mirror of two SSDs
    Directory a folder on any file system qcow2 only qcow2 and raw no ISOs, templates, backups on a second disk
    NFS, SMB/CIFS a NAS share qcow2 only qcow2 yes backups and ISOs on a NAS
    LVM (plain) thick LVM volumes preview since 9.0 no yes, on a shared SAN rarely needed at home
    Ceph RBD distributed storage on cluster nodes yes yes yes three nodes and a fast network or more
    Proxmox Backup Server a separate backup server — — yes backups only, with deduplication

    A thin disk takes space as data is written: the VM was given 100 GB, wrote 12 — the pool lost 12. So guests can be given more than there is space. It works until the guests all start writing at once: a full pool stops every guest on it (see “If it doesn’t work”). Space a guest frees on its side returns to the pool only with discard (Disk discard).

    qcow2 or raw. On a directory or NFS a VM disk is a file. qcow2 does snapshots and thin provisioning itself; raw is a few percent faster, but raw on a directory has no snapshots. On LVM-thin and ZFS the format is always raw: the storage itself provides snapshots and thin provisioning.

    ext4 with LVM-thin, or ZFS

    ext4 and LVM-thin

    one disk
    • the default layout, nothing to know in advance
    • only the kernel’s regular page cache takes memory for caching
    • snapshots and thin disks are there
    • no checksums: only the guest notices a corrupted block
    • no mirror in the installer: when the disk dies, restore from backup

    ZFS

    better with two disks or more
    • a mirror or RAIDZ right in the installer
    • a checksum on every block; in a mirror a corrupted block is repaired from the other disk
    • compression is on (compression=on, i.e. lz4): systems and text compress noticeably, video and archives don’t
    • snapshots, replication to another node, zfs send for copies
    • asks for memory for ARC and writes more to the SSD than ext4

    The rule of thumb is simple. One SSD, 8–16 GB of memory — ext4 and LVM-thin. Two identical SSDs and 16 GB of memory or more — a ZFS mirror: one disk dying stops being an event. For ZFS, use SSDs with write endurance to spare (TBW) and no hardware RAID (Hardware for Proxmox).

    ZFS takes memory for the ARC cache. Since 8.1 the installer writes a limit to /etc/modprobe.d/zfs.conf: 10 % of memory but no more than 16 GiB. On a 16 GB host that’s 1.6 GB — the guests keep almost everything.

    Snapshots

    A snapshot freezes the state of the guest’s disk: from then on only changes are written, and a rollback returns the disk to the moment of the snapshot. For a VM, memory can be included in the snapshot too (Include RAM) — after a rollback the VM continues from the same point instead of booting again.

    How snapshots work depends on the storage:

    • LVM-thin — a snapshot becomes a separate thin volume with shared blocks. You can roll back to any snapshot.
    • ZFS — native ZFS snapshots: created instantly and cost almost nothing. Rollback only to the latest one.
    • qcow2 — snapshots inside the disk file. A long chain of them slows the disk down.

    A snapshot lives on the same storage as the disk. When the disk dies, the snapshots are gone, which is why a snapshot doesn’t count as a backup (A snapshot isn’t a backup).

    A second disk for backups and ISOs

    Connect an empty disk from the UI: node › Disks › Directory › Create Directory. Proxmox creates the file system, a systemd mount unit and a directory storage itself. In Content pick backup, iso, vztmpl.

    The same for a ZFS disk: node › Disks › ZFS › Create: ZFS, and the pool immediately becomes a storage for guests.

    A NAS share — Datacenter › Storage › Add › NFS or SMB/CIFS. Proxmox mounts it itself at /mnt/pve/<name> and remounts it after a reboot. In the console pvesm does the same:

    pvesm add nfs nas-backup --server 10.0.0.20 --export /volume1/proxmox --content backup,iso
    pvesm status        # list of storages, type, used and free

    If it doesn’t work

    VMs froze all at once or report I/O errors, and lvs shows Data% 100.00 on data

    Cause
    The local-lvm thin pool is full: guests were given more disk than the pool has, and they used it up. Snapshots are another cause: they hold old blocks.
    Fix
    Free up space: delete old snapshots, enable discard and run fstrim -av in the guests (Disk discard). Watch lvs pve/data — keep Data% and Meta% under 80–90 %.

    Snapshot rollback on ZFS: can't rollback, 'before-upgrade' is not most recent snapshot

    Cause
    ZFS rolls back only to the latest snapshot: newer ones would have to be deleted, and Proxmox doesn’t do that on its own.
    Fix
    Delete the snapshots newer than the one you need (Snapshots › Remove) and roll back. LVM-thin and qcow2 have no such limit.

    Can’t upload an ISO to local-lvm — there’s no ISO Images tab

    Cause
    A storage accepts only the content types enabled in its Content. An LVM thin pool holds only disks; it never contains files.
    Fix
    ISOs, templates and backups go to a directory storage: local or your own on a second disk. What a storage accepts — Datacenter › Storage, the Content column.

    The UI throws errors, and df -h / shows 100 %

    Cause
    The local storage is a directory on the host’s root file system. ISOs, templates and backups in it filled the whole root, and the pve-cluster service can’t write the /etc/pve database.
    Fix
    Delete what’s unneeded from /var/lib/vz/dump and /var/lib/vz/template/iso. From then on, backups go to a separate disk or a NAS (Backups in Proxmox).

    A ZFS host has almost all memory in use, though guests were given less than half

    Cause
    ZFS keeps its ARC cache in memory. Since 8.1 the installer limits it to 10 % of memory but no more than 16 GiB; hosts installed earlier have no limit in /etc/modprobe.d/zfs.conf.
    Fix
    Cache size — arc_summary -s arc. A limit of, say, 4 GiB: options zfs zfs_arc_max=4294967296 in /etc/modprobe.d/zfs.conf, then update-initramfs -u -k all and a reboot.