Ring 0 · Core

Base image Fedora 45

Cover Image for Base image Fedora 45
Tomas
Tomas

In Libvirt Setup, f45-min became a hypervisor with a storage pool. This post fills that pool with a base image: one Fedora install that every machine in the lab starts from. Each machine is a thin clone of it, a disk that stores only what is different, and each one becomes its own machine the first time it boots.

This is a refresh of the 2020 base image. Then, every clone was the same machine: same hostname, same SSH host keys, the same user baked into the image, and growing the disk meant resizing partitions by hand. Now the base image is sealed, with nothing in it that belongs to one machine, and cloud-init fills in the rest on first boot.

All commands below run on f45-min, against the system connection:

$ export LIBVIRT_DEFAULT_URI=qemu:///system

A sealed base image

The kickstart starts from the minimal one in Try Fedora 45, and changes what makes an image a template rather than a machine:

  • No user, and root locked. Users come from cloud-init, per clone.
  • cloud-init and cloud-utils-growpart are installed, and cloud-init's services are enabled. Fedora 45 ships cloud-init 26, which split its old cloud-init service into cloud-init-main, cloud-init-local and cloud-init-network. A kickstart that enables the old name stops the installer at "Configuring installed system".
  • ext4 on a single partition, so that a clone's root filesystem can grow to fill a bigger disk. The clones' disks will be qcow2 overlays, and VM storage for etcd measured ext4 as the journaling filesystem that suits them: XFS on qcow2 is the one pairing to avoid.
  • The %post section seals the image. It empties /etc/machine-id, deletes the SSH host keys, clears the hostname and cloud-init's own state, and tells cloud-init to look only for a NoCloud seed, the kind virt-install attaches. Each clone then generates its own identity on first boot.
services --enabled=sshd,chronyd,cloud-init-main,cloud-init-local,cloud-init-network,cloud-config,cloud-final
rootpw --lock
autopart --type=plain --fstype=ext4 --nohome --noswap

%post
echo "%wheel ALL=(ALL) NOPASSWD:ALL" > /etc/sudoers.d/wheel
cat > /etc/cloud/cloud.cfg.d/99-nocloud.cfg <<EOF
datasource_list: [NoCloud, None]
EOF
echo 'network: {config: disabled}' > /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg
echo "" > /etc/hostname
truncate -s 0 /etc/machine-id
rm -f /etc/ssh/ssh_host_*
cloud-init clean --logs
%end

cloud-init is also told to leave the network alone. NetworkManager already brings it up, and cloud-init would otherwise rewrite the connection on every boot.

The install goes straight into the images pool, on libvirt's default NAT network. When it powers off, the VM definition is removed and only its disk is kept: the base image is never booted again, because booting it would give it an identity.

$ virt-install --name f45-base --memory 4096 --vcpus 2 --os-variant fedora-unknown --graphics none --noreboot \
  --network network=default \
  --disk=/storage/images/f45-base.vda.x86_64.qcow2,bus=virtio,format=qcow2,size=7,cache=none,io=native \
  --location=https://download.fedoraproject.org/pub/fedora/linux/releases/45/Everything/x86_64/os/ \
  --initrd-inject ./ks-base.cfg \
  --extra-args="inst.ks=file:/ks-base.cfg console=ttyS0 net.ifnames=0 biosdevname=0"
$ virsh undefine f45-base
$ virsh vol-list images
 Name                        Path
------------------------------------------------------------------------
 f45-base.vda.x86_64.qcow2   /storage/images/f45-base.vda.x86_64.qcow2

Thin clones

A clone needs two things: a disk that starts as a copy of the base, and a seed that tells cloud-init who it is.

The seed is two small files. user-data is the same for every clone: the user, its SSH key and sudo, and a request to grow the root filesystem to fill the disk. meta-data holds what makes the clone unique, its name:

$ cat seed/user-data
#cloud-config
users:
  - name: fedora
    groups: [wheel]
    sudo: "ALL=(ALL) NOPASSWD:ALL"
    lock_passwd: true
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPEBEPdH4p6zFVxm/Sajh3i3VDlYiMXQ4ePYFyg2HgHq fedora@try-fedora
growpart:
  mode: auto
  devices: ["/"]
resize_rootfs: true
$ cat seed/saturn.meta-data
instance-id: saturn
local-hostname: saturn

A DHCP reservation on the default network gives the clone an address known in advance, so we can reach it without looking it up:

$ virsh net-update default add ip-dhcp-host \
  "<host mac='52:54:00:45:00:10' name='saturn' ip='192.168.122.10'/>" --live --config

The disk is an overlay: a 30 GB qcow2 volume whose backing file is the base. It starts empty and stores only the blocks the clone changes, while every read of an unchanged block goes to the base. It's created through the pool rather than with qemu-img, because the base image is owned by root and readable only by root. libvirt can read it; the fedora user can't.

$ virsh vol-create-as images saturn.vda.x86_64.qcow2 30G --format qcow2 \
  --backing-vol f45-base.vda.x86_64.qcow2 --backing-vol-format qcow2

virt-install imports the overlay instead of installing anything, and its --cloud-init option builds the seed into a small disk and attaches it. The overlay is attached like every disk in the lab, with cache=none and io=native:

$ virt-install --name saturn --memory 1536 --vcpus 1 --os-variant fedora-unknown --graphics none \
  --network network=default,mac=52:54:00:45:00:10 \
  --disk=/storage/images/saturn.vda.x86_64.qcow2,bus=virtio,format=qcow2,cache=none,io=native \
  --cloud-init user-data=./seed/user-data,meta-data=./seed/saturn.meta-data \
  --import --noautoconsole

virt-install warns that 1536 MB is less than the 2048 MB it recommends for Fedora. That figure is for a desktop install, and a minimal clone runs comfortably in less.

jupiter is made the same way, with its own meta-data, MAC address and 192.168.122.11. The overlay's backing chain shows the two layers: 30 GB to the clone, about 2.8 GB actually stored, nearly all of it in the base.

$ sudo qemu-img info -U --backing-chain /storage/images/saturn.vda.x86_64.qcow2 | grep -E 'image:|virtual size|disk size|backing file:'
image: /storage/images/saturn.vda.x86_64.qcow2
virtual size: 30 GiB (32212254720 bytes)
disk size: 35 MiB
backing file: /storage/images/f45-base.vda.x86_64.qcow2
image: /storage/images/f45-base.vda.x86_64.qcow2
virtual size: 7 GiB (7516192768 bytes)
disk size: 2.78 GiB

First boot

On first boot, cloud-init reads the seed: it sets the hostname, creates the user, generates new SSH host keys and grows the root filesystem. systemd generates a machine-id.

The seed authorizes the same demo key as f45-min's own kickstart, and the clones are logged in to from f45-min, so f45-min needs the private half. From your machine:

$ scp -P 2222 -i id_ed25519 id_ed25519 fedora@localhost:.ssh/id_ed25519

A minute later, each clone is its own machine:

$ ssh fedora@192.168.122.10
[fedora@saturn ~]$ hostnamectl hostname
saturn
[fedora@saturn ~]$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda3        28G  1.3G   26G   5% /
[fedora@saturn ~]$ cloud-init status
status: done

The root filesystem is 28 GB, grown from the 5 GB partition in the base. saturn and jupiter come from the same base, but each has its own machine-id and its own SSH host key:

saturn   6851993e2c644ea891e98d2ca1d34d1c   SHA256:FckfstIdpNAAVvNVMLh6U8QXLUPCjUoiZEjaiLL/1bQ
jupiter  afe4516dcc3d4ef7b9f41f072f0c08e8   SHA256:8taz9MR3Tcov3XvPFbxU9nDK2+bWE6mA4qUK72lowrU

And the overlays are still small. Each clone has written a few tens of megabytes of its own (logs, mostly), on top of a 2.8 GB base the two share:

$ sudo du -h /storage/images/*.qcow2
2.8G	/storage/images/f45-base.vda.x86_64.qcow2
35M	/storage/images/jupiter.vda.x86_64.qcow2
39M	/storage/images/saturn.vda.x86_64.qcow2

Two things to watch for:

  • A clone started without a seed has no SSH host keys, because the base image has none. sshd won't start, and connections are refused. If ssh to a new clone says "connection refused", check that --cloud-init was there.
  • cloud-init status --wait exits with 2, not 0, on Fedora 45, although it prints status: done. cloud-init's earliest stage runs hostnamectl before D-Bus is up, the call fails, and a later stage sets the hostname successfully. The failed attempt stays on record as a recoverable error, which cloud-init status --long lists. The clone is fine, but a script that waits with cloud-init status --wait && … will think it isn't.

f45-min now has a base image and two machines made from it. Next, they stop sharing a NAT network: open networks, so that machines on different hosts can reach each other by their real addresses.

back