VM storage for etcd

Tomas

Tomas

In Try Fedora 45 we installed f45-min, the VM the rest of the series runs in. Before it becomes a hypervisor of its own, it's worth deciding where its disks live and how they're attached, because every VM inside it, and every etcd inside those, inherits that choice.

etcd is the reason. It doesn't acknowledge a write until the write is on disk, so every commit waits for an fdatasync. etcd's own guidance is that the 99th percentile of that wait should stay under 10 ms. Above that, leader elections start timing out and the API server slows with them.

This post measures that wait the way etcd's maintainers recommend, through every layer between etcd and the disk, and ends with a prescription. The commands are Fedora's, but everything measured here is Linux: the block layer, QEMU and KVM.

The test

etcd's documentation suggests one fio job that looks like etcd's write-ahead log: small sequential writes, each followed by fdatasync.

$ fio --name=etcd --directory=/path/to/test --rw=write --ioengine=sync \
--fdatasync=1 --size=22m --bs=2300

fio reports the latency of the sync calls as percentiles. The two that matter are p50, the typical commit, and p99, the one etcd's 10 ms budget is about. Every number below is p50 / p99 in milliseconds, from three or more runs on an idle machine.

What one fsync costs

A guest's fdatasync ends as a flush on the host's disk. How many flushes, and how expensive each one is, are the whole story:

A flush on the host costs whatever the drive and the host filesystem charge for it.
A journaling filesystem in the guest (ext4, XFS or btrfs) writes its journal too, so one fdatasync in the guest usually becomes two flushes below it. A raw device with no filesystem needs one.
Each layer of virtualisation adds the time QEMU and KVM take to pass the request down.

So: guest fsync ≈ flushes per fsync × cost of one flush on the host. Tuning the guest's journal (logbsize, an external log, notreelog) doesn't change the count, and doesn't move the numbers.

The host filesystem

Measured directly on the host, on a desktop with a consumer NVMe drive:

Host storage	p50	p99
The partition itself, no filesystem	0.41	0.70–0.84
XFS	1.06	1.25
ext4	1.03–1.11	1.30–4.75
btrfs, `nodatacow`	1.17	1.40
btrfs, copy-on-write	1.29	1.55–4.75

On the host itself the differences are small. They grow once VMs are on top, because a VM's disk image is a file on that filesystem and every guest flush becomes a flush of that file:

A raw image on…	one flush inside the VM (p50 / p99)
a partition passed straight to the VM	0.53 / 0.70
XFS	0.53 / 0.70
btrfs, `nodatacow`	1.24 / 1.5
btrfs, copy-on-write, qcow2, default cache	1.43 / 1.8

A raw image on XFS costs nothing over handing the VM the partition. btrfs roughly doubles every flush, and nodatacow doesn't fix it: btrfs still commits its own metadata tree on every flush of the file.

Keep the OS on whatever you like, but put VM images on their own XFS volume: a partition or a logical volume, mounted noatime. XFS needs no tuning for this: the mkfs.xfs defaults are right, and there are no file attributes to set. Leave discard off the mount and let fstrim.timer trim once a week.

The image and how it's attached

Three settings decide what a flush costs on its way out of the VM:

Format: raw. A raw file is the disk, byte for byte. qcow2 adds a mapping layer that's useful for snapshots and backing files, and preallocated qcow2 on XFS measured the same as raw. The one pairing to avoid is XFS inside the guest on a qcow2 image: p99 7–9 ms, close to etcd's limit on its own.
Preallocation: falloc. qemu-img create -f raw -o preallocation=falloc reserves the file's blocks up front, so a write never waits for the host filesystem to find space. Writing zeros (preallocation=full) measured the same and takes much longer.
Cache and I/O mode: cache=none, io=native. cache=none opens the image with O_DIRECT, so the guest's writes skip the host's page cache, and a guest flush becomes exactly one flush on the host. io=native submits them through Linux AIO instead of a thread pool.

In libvirt that's one line per disk:

<driver name='qemu' type='raw' cache='none' io='native'/>

and in virt-install, --disk path,format=raw,cache=none,io=native.

Inside the guest, use ext4 or XFS on raw images, and ext4 on a qcow2 overlay. On an otherwise ideal stack, the first fio run on a freshly made guest filesystem is slower (ext4: 2.5 ms p50) while it settles. The runs after it are not.

Nesting

The lab nests: f45-min is a VM, and the machines the cluster runs on are VMs inside it. Every level costs time on each flush, so the number etcd sees is the number at the bottom.

On the desktop, the prescribed stack against the defaults, with ext4 in each guest:

Level	Defaults: qcow2 on btrfs, default cache	Prescribed: raw on XFS, `cache=none`, `io=native`
L1, a VM	2.9 / 3.7–5.1	1.2 / 1.5–2.0
L2, a VM inside that VM	4.2–5.0 / 6.3–9.0	1.7–1.8 / 2.8–3.0

The defaults put etcd at its 10 ms limit with nothing else running. The prescribed stack leaves it two thirds of its budget.

The ~0.3 ms that each level adds is mostly the CPU. A request from an L2 guest exits to the L1 hypervisor, and handling that exit makes the L1 hypervisor read and write its virtual-machine control structures. Without VMCS shadowing, each of those reads and writes traps again, to the host. Check yours:

$ grep -o -w 'vmx\|ept\|shadow_vmcs' /proc/cpuinfo | sort -u

On a two-socket server with VMCS shadowing and two SATA SSDs in RAID 1, the lab as the rest of the series builds it measures:

Level	p50	p99
L0, the host's XFS volume	0.40–0.42	0.56–0.64
L1, f45-min, ext4 root on raw	0.97–1.07	1.43–1.56
L1, f45-min, XFS `/storage` on raw	0.73–1.09	1.03–1.58
L2, a lab machine, ext4 on a qcow2 overlay	1.3–3.7	4.1–6.2

The host's flush is faster than the desktop's NVMe, and L1 is the same story one level down. L2 is where the format shows: the lab machines' roots are thin qcow2 overlays on a shared base image, and a write to a block the overlay hasn't allocated yet costs qcow2 a metadata update before the data lands. That's within etcd's budget, with more than a third of it to spare, and it's what makes a clone cost megabytes instead of gigabytes. For etcd's own data on a machine that matters, give it a raw, preallocated disk of its own instead, attached the same way: that's the 1.7 / 2.8 ms row above.

Not just fsync: IOPS

etcd waits on one flush at a time, which is why fsync latency is its number. Most other workloads, a database's reads or a container image being unpacked, care how many small random operations the storage can do. fio measures that too, with 4 KiB random reads and writes that bypass the page cache (--direct=1), on a 512 MiB file it lays out first:

$ fio --name=iops --directory=/path/to/test --rw=randread --bs=4k --ioengine=libaio \
--direct=1 --iodepth=32 --size=512m --runtime=20 --time_based

at two queue depths: 1, a single request at a time, which is what one waiting writer sees, and 32, as many in flight as the stack will take. On the same server, the same levels, three 20 s runs each:

Level	random read, QD1	random read, QD32	random write, QD1	random write, QD32
L0, the host's XFS volume	7,300–7,500	87,000	9,400–10,000	51,000
L1, f45-min, ext4 root on raw	4,100–4,200	68,000–70,000	5,900–6,000	38,000–39,000
L1, f45-min, XFS `/storage` on raw	4,100–4,300	70,000–73,000	6,200	50,000
L2, a lab machine, ext4 on a qcow2 overlay	1,500–1,700	19,000–24,000	1,500–2,100	19,000–24,000

At queue depth 1, IOPS is just one over the latency of a single request, so it shows what each level adds: about 0.1 ms going from the host into f45-min, and about 0.4 ms more into the lab machines. At queue depth 32 the levels overlap their work, and the prescribed stack keeps most of the host: f45-min's XFS volume does 80 percent of the host's random reads and all of its random writes. ext4 on f45-min's root does a quarter fewer writes than XFS, through the same raw image and cache mode.

The lab machines lose three quarters or more, except for random writes at depth 32, where they keep about 40 percent. They're two levels down, and their roots are qcow2 overlays: every request goes through qcow2's mapping on its way to the backing file. That's still 20,000 random operations a second on each lab machine, which no part of this series comes close to needing, and the same advice as for etcd applies: a disk that has to be fast gets its own raw image.

The prescription

On the hypervisor:

VM images on their own XFS volume, mounted noatime, fstrim.timer on.
Raw images created with preallocation=falloc.
Every disk attached with cache=none and io=native.

In the guests:

ext4 or XFS on raw images; ext4 on qcow2 overlays.
Never XFS on qcow2.
etcd's data on a raw, preallocated disk when it has to be fast; an overlay when cheap clones matter more.

At every level of nesting, the same again. The rest of the series follows it from here: f45-min's disks are raw, its /storage is XFS, and the lab machines are ext4 on overlays attached the same way.

Raw results

Every number above comes from these files, as the lab wrote them, with only the private host names and the home network's addresses changed: study-clean.log, study-extra.log, study-server.log, and the server round again from a fresh lab with the IOPS runs, study-server-2.log.