Running fedora coreos with incus

After using proxmox for years, incus is my new goto as hypervisor. As I like the lazyness of a self updating os, here is a way to work with Fedora CoreOS with incus.

For those who don’t know, incus is a modern and secure system container and virtual machine manager. Fedora CoreOS (fcos) is an auto-updating, minimal-operating-system designed specifically for running containerized workloads with streamlined management via Ignition.

Building the image#

There is no fcos image build for incus, thus we have to build it ourselves:

Download the current qemu qcow2 from the download page
Decompress the qcow as incus requires a special archive format
Create a metadata.yaml file containing the required fields
Build the archive as described here

You can find and example script here.

At the end of the setup, you should be able to use the image to start the VM:

To run this asciicast without javascript, use asciinema play https://agnomi.net/casts/incus/fedora-coreos/create-vm.cast with Asciinema

Great, we now can run coreos on incus !

Sparking things off#

Fcos is a somewhat particular distro. As mentioned above, its configuration is done using ignition. As fcos is itself immutable, we should write an ignition file describing the full setup of our host. Once booted up, the VM will apply it’s configuration and be ready to go.

Building an ignition file#

Ignition instructions consumed by fcos are json formatted. However, a more human friendly syntax called Butane can be used to generate ignition files.

As an example, we will use the First Ignition config via Butane provided by the project itself:

variant: fcos
version: 1.6.0
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure`
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: |
          tutorial

As you guessed, this ignition file will autolog the console as the sysadmin core and set the hostname as tutorial.

Converted to json using butane, we get:

{
  "ignition": {
    "version": "3.5.0"
  },
  "storage": {
    "files": [
      {
        "path": "/etc/hostname",
        "contents": {
          "compression": "",
          "source": "data:,tutorial%0A"
        },
        "mode": 420
      }
    ]
  },
  "systemd": {
    "units": [
      {
        "dropins": [
          {
            "contents": "[Service]\n# Override Execstart in main unit\nExecStart=\n# Add new Execstart with `-` prefix to ignore failure`\nExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM\n",
            "name": "autologin-core.conf"
          }
        ],
        "name": "serial-getty@ttyS0.service"
      }
    ]
  }
}

Now you get why we deal with a file conversion instead of writing raw json…

Wrapping up in a brand new /var/lib/incus/ignition/fcos02.ign, we now can create a second VM using the ignition file.

incus create fcos/stable fcos02

Make it use the ignition file.

incus config set fcos02 raw.qemu=" -fw_cfg name=opt/com.coreos/config,file=/var/lib/incus/ignition/fcos02.ign"

By default (and at least on debian), the incus VMs gets apparmor rules making them enable to read our ignition file; thus we append a line to the vm config to fix this issue.

incus config set fcos02 raw.apparmor="/var/lib/incus/ignition/fcos02.ign r,"

After a short spinup, we got our host up, running and set up:

Fedora CoreOS 43.20251024.3.0
Kernel 6.17.1-300.fc43.x86_64 on x86_64 (ttyS0)

SSH host key: SHA256:oujHKzq8ZEJIG93lmG6i2Y/uNxwVGl/8S7g8jFSBc0w (ECDSA)
SSH host key: SHA256:NE++l0H9FldZw1FadV3yUUls0zJt/qLc1rEAvc2da6k (ED25519)
SSH host key: SHA256:UpyZEhAMH/EZPZ6UMJDzv9JW+YwOs4apvJXKZrmQMS4 (RSA)
enp5s0: 10.97.5.203 fe80::2e10:b6ef:4fea:496e
Ignition: ran on 2025/11/15 21:12:21 UTC (this boot)
Ignition: user-provided config was applied
No SSH authorized keys provided by Ignition or Afterburn
Ignition: ran on 2025/11/15 21:12:21 UTC (this boot)
Ignition: user-provided config was applied
Try contacting this VM's SSH server via 'ssh vsock%946744013' from host.

Fedora CoreOS 43.20251024.3.0
Kernel 6.17.1-300.fc43.x86_64 on x86_64 (ttyS0)

tutorial login: core (automatic login)

Fedora CoreOS 43.20251024.3.0
[core@tutorial ~]$

Working with persistent storage#

In the fcos mindset, vms should be recreated each time there is a new configuration release. This becomes tedious when our fcos has to handle persistent data.

Incus makes it possible to make custom volumes, which are basicaly extra disks attached to instances:

incus storage volume create default mydata --type block
incus storage volume attach default mydata fcos02

The drive is detected as /dev/sdb, but most importantly, its serial number is marked as incus_mydata:

$ ls -l /dev/disk/by-id/
total 0
lrwxrwxrwx. 1 root root  9 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_mydata -> ../../sdb
lrwxrwxrwx. 1 root root  9 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_root -> ../../sda
lrwxrwxrwx. 1 root root 10 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_root-part1 -> ../../sda1
lrwxrwxrwx. 1 root root 10 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_root-part2 -> ../../sda2
lrwxrwxrwx. 1 root root 10 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_root-part3 -> ../../sda3
lrwxrwxrwx. 1 root root 10 Nov 15 22:08 scsi-0QEMU_QEMU_HARDDISK_incus_root-part4 -> ../../sda4

This means that we can add some storage properties to our ignition setup:

variant: fcos
version: 1.6.0
systemd:
  units:
    - name: serial-getty@ttyS0.service
      dropins:
      - name: autologin-core.conf
        contents: |
          [Service]
          # Override Execstart in main unit
          ExecStart=
          # Add new Execstart with `-` prefix to ignore failure`
          ExecStart=-/usr/sbin/agetty --autologin core --noclear %I $TERM
storage:
  files:
    - path: /etc/hostname
      mode: 0644
      contents:
        inline: |
          tutorial
  filesystems:
    - path: /var/data
      device: /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_incus_mydata
      format: xfs
      with_mount_unit: true

After pushing the new ignition file, we can delete fcos02, and ignite our brand new fcos03:

# Clearing the path
incus stop --force fcos02
incus rm fcos02
incus storage volume rm default mydata

# Building fcos03
incus create fcos/stable fcos03
incus config set fcos03 raw.qemu=" -fw_cfg name=opt/com.coreos/config,file=/var/lib/incus/ignition/fcos03.ign"
incus config set fcos03 raw.apparmor="/var/lib/incus/ignition/fcos03.ign r,"
incus storage volume create default mydata --type block
incus storage volume attach default mydata fcos03
incus start fcos03

[core@tutorial ~]$ findmnt --real
TARGET           SOURCE                                       FSTYPE OPTIONS
/var             /dev/sda4[/ostree/deploy/fedora-coreos/var]  xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,prjquota
└─/var/data      /dev/sdb                                     xfs    rw,relatime,seclabel,attr2,inode64,logbufs=8,logbsize=32k,noquota

We see that our custom drive is mounted in /var/data. To be verify that our disk is remounted as is after vm rebuild, we leave a message in the directory:

[core@tutorial ~]$ echo "From fcos03" | sudo tee /var/data/hello

Creating a new release of our fcos config, we delete and rebuild the vm such as:

# Clearing the path
incus stop --force fcos03
incus rm fcos03

# Building fcos04
incus create fcos/stable fcos04
incus config set fcos04 raw.qemu=" -fw_cfg name=opt/com.coreos/config,file=/var/lib/incus/ignition/fcos04.ign"
incus config set fcos04 raw.apparmor="/var/lib/incus/ignition/fcos04.ign r,"
incus storage volume attach default mydata fcos04
incus start fcos04

After this ultimate boot, we can see our hello file, unchanged:

[core@tutorial ~]$ cat /var/data/hello 
From fcos03

Conclusion#

In this post, we saw that incus and fedora coreos can be a good fit to work together. Using the excellent incus tofu provider, those tools has been a blessing for my personnal and associative activities.

However, nothing is perfect and I am still searching for solutions about some quirks I met:

Find a way not to have to scp the ignition files (maybe using a volume containing them all ?)
If scp is unavoidable, how could I not have to spread all ignition files to the cluster mumbers ?