update fedora server virtualization instructions

infrastructure/graduated/fedora/fedora-server.md

@@ -16,6 +16,7 @@
   - [Common Storage Mounts](#common-storage-mounts)
   - [Network Bridge](#network-bridge)
   - [Virtualization](#virtualization)
+    - [Troubleshooting](#troubleshooting)
     - [QEMU Images](#qemu-images)
   - [Firewalld](#firewalld)
   - [Backups](#backups)
@@ -33,7 +34,7 @@
     - [LVM Thin Provisioning](#lvm-thin-provisioning)
     - [Set eui64 on network interface](#set-eui64-on-network-interface)
     - [Install and Enable Cockpit](#install-and-enable-cockpit)
-  - [Troubleshooting](#troubleshooting)
+  - [Troubleshooting](#troubleshooting-1)
     - [Cockpit Terminal Unusable or Weird Colors](#cockpit-terminal-unusable-or-weird-colors)
     - [Chroot into a mounted disk](#chroot-into-a-mounted-disk)
     - [Resize Last Partition to Fill Available Space](#resize-last-partition-to-fill-available-space)
@@ -73,12 +74,12 @@ On the operator:
 
 ```bash
 export SSH_HOST=kube
-ssh-keygen -t rsa -b 4096 -C ducoterra@"$SSH_HOST".reeselink.com -f ~/.ssh/id_"$SSH_HOST"_rsa
+ssh-keygen -C ducoterra@${SSH_HOST}.reeselink.com -f ~/.ssh/id_${SSH_HOST}_rsa
 
 # Note: If you get "too many authentication failures" it's likely because you have too many private
 # keys in your ~/.ssh directory. Use `-o PubkeyAuthentication` to fix it.
-ssh-copy-id -o PubkeyAuthentication=no -i ~/.ssh/id_"$SSH_HOST"_rsa.pub ducoterra@${SSH_HOST}.reeselink.com
+ssh-copy-id -o PubkeyAuthentication=no -i ~/.ssh/id_${SSH_HOST}_rsa.pub ducoterra@${SSH_HOST}.reeselink.com
-ssh -i ~/.ssh/id_"$SSH_HOST"_rsa ducoterra@${SSH_HOST}.reeselink.com
+ssh -i ~/.ssh/id_${SSH_HOST}_rsa ducoterra@${SSH_HOST}.reeselink.com
 # Copy authorized_keys to root
 sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys
 exit
@@ -88,9 +89,6 @@ cat <<EOF >> ~/.ssh/config
 Host ${SSH_HOST}
     Hostname ${SSH_HOST}.reeselink.com
     User root
-    ProxyCommand none
-    ForwardAgent no
-    ForwardX11 no
     Port 22
     KeepAlive yes
     IdentityFile ~/.ssh/id_${SSH_HOST}_rsa
@@ -177,7 +175,7 @@ mount -a --mkdir
 
 <https://en.opensuse.org/openSUSE:Snapper_Tutorial>
 
-Note: This requires you set up the mount point at `/btrfs` correctly! See [above](#btrfs-parent-volumes).
+<http://snapper.io/manpages/snapper-configs.html>
 
 We'll be using snapper, a tool for automating and controlling snapshot behavior.
 
@@ -207,6 +205,9 @@ snapper -c root create --description "test snapshot"
 snapper -c root delete 1
 ```
 
+Note - you probably don't want to keep yearly snapshots.
+Edit `/etc/snapper/configs/root` and change `TIMELINE_LIMIT_YEARLY=` to `0`.
+
 ## TPM2 Luks Decryption
 
 Mostly taken from here:
@@ -263,9 +264,6 @@ done
 ##### END OPTIONAL #####
 ########################
 
-# Reenroll
-systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
-
 # Append to command line args
 echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
 dracut -f
@@ -274,8 +272,18 @@ dracut -f
 Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX
 
 ```bash
+# Update Grub
 grub2-mkconfig -o /boot/grub2/grub.cfg
 reboot
+# Cross your fingers that you don't have to go type in the password manually.
+# Yes, 60 full seconds is too long. Go type your password in.
+```
+
+If you need to reenroll for some reason:
+
+```bash
+# Reenroll
+systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
 ```
 
 ## Change your password
@@ -316,6 +324,38 @@ nmcli connection modify bridge0 ipv6.addr-gen-mode eui64
 
 ## Virtualization
 
+Don't forget to add a btrfs subvolume for `/var/lib/libvirt`
+
+```bash
+# Since we already created our /btrfs mountpoint, this volume will show up automatically
+# at /btrfs/libvirt
+btrfs sub create /btrfs/libvirt
+```
+
+Now create an fstab entry that mounts the volume at /var/lib/libvirt
+
+```bash
+UUID=... /var/lib/libvirt    btrfs   subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0
+```
+
+Mount the libvirt volume:
+
+```bash
+systemctl daemon-reload
+mount -a --mkdir
+# Check that the mount was successful. This will print something if our mount worked.
+mount | grep -i /var/lib/libvirt
+```
+
+Create a snapshot schedule for libvirt.
+
+```bash
+snapper -c libvirt create-config /var/lib/libvirt
+# Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt
+```
+
+Install and enable the virtualization service.
+
 ```bash
 dnf group install --with-optional virtualization
 systemctl enable --now libvirtd
@@ -323,6 +363,17 @@ systemctl enable --now libvirtd
 
 Install the cockpit machines application.
 
+### Troubleshooting
+
+```bash
+# Oops, I did this after I installed virtualization
+rsync -av /var/lib/libvirt/ /btrfs/libvirt/
+rm -rf /var/lib/libvirt
+# Find the path to your btrfs volume
+lsblk
+mount -o subvol=libvirt /dev/mapper/luks-... /var/lib/libvirtd
+```
+
 ### QEMU Images
 
 ```bash