the great migration from truenas to fedora and all its collatoral

2025-04-08 12:40:42 -04:00
parent 9a3382862d
commit 9417e711a9
54 changed files with 1533 additions and 519 deletions
--- a/infrastructure/graduated/fedora/fedora-server.md
+++ b/infrastructure/graduated/fedora/fedora-server.md
@@ -2,17 +2,42 @@

 - [Fedora Server](#fedora-server)
  - [Installation](#installation)
-  - [Resize logical volume](#resize-logical-volume)
  - [Setup SSH](#setup-ssh)
  - [DNF](#dnf)
  - [Fail2Ban](#fail2ban)
+  - [BTRFS Parent Volumes](#btrfs-parent-volumes)
+  - [BTRFS Snapshots](#btrfs-snapshots)
+  - [TPM2 Luks Decryption](#tpm2-luks-decryption)
+  - [Change your password](#change-your-password)
  - [Automatic Updates](#automatic-updates)
-  - [Disable Swap](#disable-swap)
-  - [Selinux](#selinux)
+  - [Monitoring](#monitoring)
+    - [Disk Usage](#disk-usage)
+    - [Disk Wear](#disk-wear)
+  - [Common Storage Mounts](#common-storage-mounts)
+  - [Network Bridge](#network-bridge)
+  - [Virtualization](#virtualization)
+    - [QEMU Images](#qemu-images)
  - [Firewalld](#firewalld)
-  - [Docker with Podman as Runtime](#docker-with-podman-as-runtime)
-  - [Extras](#extras)
-  - [Downgrading Kernel](#downgrading-kernel)
+  - [Backups](#backups)
+    - [Quick Backup](#quick-backup)
+    - [Regular Backup to an NFS Share](#regular-backup-to-an-nfs-share)
+  - [Optional Steps](#optional-steps)
+    - [Docker with Podman as Runtime](#docker-with-podman-as-runtime)
+    - [Vanilla Docker](#vanilla-docker)
+    - [Extra Software](#extra-software)
+    - [Disable Swap](#disable-swap)
+    - [Disable Selinux](#disable-selinux)
+    - [Downgrading Kernel](#downgrading-kernel)
+    - [Resize logical volume](#resize-logical-volume)
+    - [Create XFS LVM](#create-xfs-lvm)
+    - [LVM Thin Provisioning](#lvm-thin-provisioning)
+    - [Set eui64 on network interface](#set-eui64-on-network-interface)
+    - [Install and Enable Cockpit](#install-and-enable-cockpit)
+  - [Troubleshooting](#troubleshooting)
+    - [Cockpit Terminal Unusable or Weird Colors](#cockpit-terminal-unusable-or-weird-colors)
+    - [Chroot into a mounted disk](#chroot-into-a-mounted-disk)
+    - [Resize Last Partition to Fill Available Space](#resize-last-partition-to-fill-available-space)
+    - [LUKS performance](#luks-performance)

 <https://docs.fedoraproject.org/en-US/fedora-server/installation/postinstallation-tasks/#_manage_system_updates>

@@ -23,22 +48,27 @@ and the operator will store information about each server.

 ## Installation

-1. Create an administrator. We'll give ssh root access later, but this gives you a cockpit user.
-2. Ensure IPV6 connection is set to "eui64".
-3. Set hostname
-
-## Resize logical volume
-
-```bash
-# Replace /dev/sda2 with whatever your disks are
-# This assumes xfs
-pvresize /dev/sda2
-lvextend /dev/mapper/root -l+100%FREE
-xfs_growfs -d /dev/mapper/root
-```
+1. Configure network first
+   1. Set a hostname
+   2. Disable ipv6 privacy extensions
+2. Software Selection
+   1. Headless Management
+3. User Creation
+   1. Set a simple password, we'll change it later
+4. Disk partitioning
+   1. Select manual (blivet) partitioning
+   2. Create a 1GB EFI system partition and mount it at `/boot/efi`
+   3. Create a 1GB ext4 partition and mount it at `/boot`
+   4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it
+   5. Create a btrfs subvolume called "root" and mount it at `/`
+   6. Create any other btrfs subvolumes you might need
+5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time.
+6. Install and reboot

 ## Setup SSH

+In this setup we'll allow ssh to the root user via key and keep the admin user for cockpit.
+
 On the operator:

 ```bash
@@ -47,7 +77,11 @@ ssh-keygen -t rsa -b 4096 -C ducoterra@"$SSH_HOST".reeselink.com -f ~/.ssh/id_"$

 # Note: If you get "too many authentication failures" it's likely because you have too many private
 # keys in your ~/.ssh directory. Use `-o PubkeyAuthentication` to fix it.
-ssh-copy-id -o PubkeyAuthentication=no -i ~/.ssh/id_"$SSH_HOST"_rsa.pub ducoterra@"$SSH_HOST".reeselink.com
+ssh-copy-id -o PubkeyAuthentication=no -i ~/.ssh/id_"$SSH_HOST"_rsa.pub ducoterra@${SSH_HOST}.reeselink.com
+ssh -i ~/.ssh/id_"$SSH_HOST"_rsa ducoterra@${SSH_HOST}.reeselink.com
+# Copy authorized_keys to root
+sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys
+exit

 cat <<EOF >> ~/.ssh/config

@@ -61,26 +95,17 @@ Host ${SSH_HOST}
    KeepAlive yes
    IdentityFile ~/.ssh/id_${SSH_HOST}_rsa
 EOF
-```

-On the server:
-
-```bash
-# Copy authorized_keys to root
-sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys
-
-# Change your password
-passwd
-
-sudo su -
+ssh ${SSH_HOST}
+# Disable password auth
 echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/01-prohibit-password.conf
-echo '%wheel    ALL=(ALL)   NOPASSWD: ALL' > /etc/sudoers.d/01-nopasswd-wheel
 systemctl restart sshd
-```

-On the operator:
+# OPTIONAL: Disable sudo password
+echo '%wheel    ALL=(ALL)   NOPASSWD: ALL' > /etc/sudoers.d/01-nopasswd-wheel
+
+exit

-```bash
 # Test if you can SSH with a password
 ssh -o PubkeyAuthentication=no ducoterra@${SSH_HOST}.reeselink.com

@@ -95,7 +120,9 @@ Configure dnf to use the fastest mirror:
 ```bash
 echo 'fastestmirror=1' >> /etc/dnf/dnf.conf
 dnf clean all
-dnf update
+dnf update --refresh -y
+# libdnf5 is required for ansible to work
+dnf install -y glances tmux vim python3-libdnf5
 ```

 ## Fail2Ban
@@ -103,6 +130,9 @@ dnf update
 On the server:

 ```bash
+# Run tmux session
+tmux
+
 dnf install -y fail2ban

 # Setup initial rules
@@ -123,45 +153,198 @@ enabled = true
 EOF

 systemctl enable fail2ban --now
+
+# OPTIONAL: follow logs
 tail -f /var/log/fail2ban.log
 ```

+## BTRFS Parent Volumes
+
+In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see
+all subvolumes.
+
+```conf
+UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
+UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
+```
+
+```bash
+systemctl daemon-reload
+mount -a --mkdir
+```
+
+## BTRFS Snapshots
+
+<https://en.opensuse.org/openSUSE:Snapper_Tutorial>
+
+Note: This requires you set up the mount point at `/btrfs` correctly! See [above](#btrfs-parent-volumes).
+
+We'll be using snapper, a tool for automating and controlling snapshot behavior.
+
+```bash
+dnf install snapper dnf-plugin-snapper
+
+# Allow selinux management
+semanage permissive -a snapperd_t
+
+# Note, if you mess something up you can run snapper -c root delete-config to delete
+# System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper
+snapper -c root create-config /
+snapper -c data create-config /path/to/other/data
+
+# Enable automatic snapshots
+systemctl enable --now snapper-timeline.timer
+# Enable automatic cleanup
+systemctl enable --now snapper-cleanup.timer
+# Enable snapshots on boot
+systemctl enable --now snapper-boot.timer
+
+# List snapshots
+snapper -c root list
+# Create snapshot manually
+snapper -c root create --description "test snapshot"
+# Delete first snapshot
+snapper -c root delete 1
+```
+
+## TPM2 Luks Decryption
+
+Mostly taken from here:
+<https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>
+
+PCR reference for `--tpm2-pcrs` args
+
+```text
+0: System firmware executable
+2: Kernel
+4: Bootloader
+7: Secure boot state
+8: Cmdline
+9: Initrd
+```
+
+Note, if your threat vector is people trying to get data off your old disks after throwing them
+away, you can set `--tpm2-pcrs=""`. Someone could gain access to your encrypted partition if they
+can access your machine physically by manipulating the boot parameters but you're guaranteed to
+unlock despite updates and upgrades.
+
+Basic commands:
+
+```bash
+# Run tmux session
+tmux
+
+# Show tpm2 devices
+systemd-cryptenroll --tpm2-device=list
+# Show crypto luks block devices
+blkid -t TYPE=crypto_LUKS
+
+# Enroll the tpm2 device with systemd-cryptenroll
+systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs=""
+
+####################
+##### OPTIONAL #####
+####################
+# If you have lots of devices to decrypt (like a btrfs raid array), use these commands.
+# Get all crypto luks partitions
+blkid | grep crypto_LUKS
+# List them all space-separated and drop the '/dev'
+LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1"
+
+# Check that your list is good
+for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done
+
+# Enroll
+for dev in $LUKS_DEVS; do \
+echo "Enrolling /dev/$dev"; \
+systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \
+done
+########################
+##### END OPTIONAL #####
+########################
+
+# Reenroll
+systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
+
+# Append to command line args
+echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
+dracut -f
+```
+
+Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX
+
+```bash
+grub2-mkconfig -o /boot/grub2/grub.cfg
+reboot
+```
+
+## Change your password
+
+In Cockpit navigate to Accounts -> user -> Set password
+
 ## Automatic Updates

-On the server:
+In Cockpit navigate to software updates -> automatic updates -> install -> security updates only
+
+## Monitoring
+
+In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics
+
+### Disk Usage
+
+TODO
+
+### Disk Wear
+
+TODO
+
+## Common Storage Mounts
+
+Note: mount these before you install the relavant package!
+
+1. For virtual machines: `/var/lib/libvirt`
+2. For podman: `/var/lib/containers`
+3. For docker: `/var/lib/docker`
+
+## Network Bridge
+
+Networking -> Add bridge -> add network interface and save

 ```bash
-dnf install dnf-automatic -y
-
-systemctl enable --now dnf-automatic.timer
+nmcli connection modify bridge0 ipv6.addr-gen-mode eui64
 ```

-Edit the configuration to only do security updates.
-
-## Disable Swap
+## Virtualization

 ```bash
-swapoff -a
-zramctl --reset /dev/zram0
-dnf -y remove zram-generator-defaults
+dnf group install --with-optional virtualization
+systemctl enable --now libvirtd
 ```

-## Selinux
+Install the cockpit machines application.

-By default selinux will be enforcing. You can set it to permissive with
+### QEMU Images

 ```bash
-setenforce 0
+# Grow an image to 2TB
+qemu-img resize nextcloud_aio-fcfgp.qcow2 2T
 ```

-And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.
+```bash
+# Convert OVA to img
+qemu-img convert -f vmdk -O raw in.vmdk out.img
+
+# Convert qcow2 to img
+qemu-img convert -f qcow2 -O raw in.raw out.img
+```

 ## Firewalld

 Set the default firewalld zone to `public`

 ```bash
-firewall-cmd --set-default-zone=public
+# Note, you probably don't have to do this. Check Cockpit Network -> Firewall
+# firewall-cmd --set-default-zone=public
 ```

 Firewalld will be on and blocking by default. You can check the zone and allowed ports with:
@@ -178,7 +361,50 @@ firewall-cmd --permanent --zone=public --add-port=9090/tcp
 firewall-cmd --reload
 ```

-## Docker with Podman as Runtime
+## Backups
+
+### Quick Backup
+
+```bash
+rsync -av / \
+--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} \
+/mnt/root/
+```
+
+### Regular Backup to an NFS Share
+
+1. Create a new dataset called "<server>_backup"
+2. Remove "other" read/exec permissions from the dataset
+3. Create a new NFS share for that dataset with maproot user and group set to root
+4. Mount the NFS share to your server at `/backup`
+5. Copy the following script into /root/backup.sh
+
+```bash
+#!/bin/bash
+
+BACKUP_PATH="/backup"
+EXCLUDE_DIR='{"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}'
+SOURCE_DIR="/"
+
+rsync -aAXv ${SOURCE_DIR} --exclude=${EXCLUDE_DIR} ${BACKUP_PATH}
+
+if [ $? -eq 0 ]; then
+    echo "Backup completed successfully"
+else
+    echo "Some error occurred during backup"
+fi
+```
+
+6. `chmod +x /root/backup.sh`
+7. `crontab -e`
+
+```cron
+0  2  *  *  *  bash /root/backup.sh >> /root/backup.log
+```
+
+## Optional Steps
+
+### Docker with Podman as Runtime

 Note, you'll need to ssh into the server as the user in order to start the user's systemd session.

@@ -191,9 +417,18 @@ docker context create podman --docker host=unix://$XDG_RUNTIME_DIR/podman/podman
 docker context use podman
 ```

-## Extras
+### Vanilla Docker

-On the server:
+<https://docs.docker.com/engine/install/fedora/>
+
+```bash
+dnf -y install dnf-plugins-core
+dnf-3 config-manager --add-repo https://download.docker.com/linux/fedora/docker-ce.repo
+dnf install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
+systemctl enable --now docker
+```
+
+### Extra Software

 ```bash
 # Set vim as the default editor
@@ -202,38 +437,30 @@ dnf install -y vim-default-editor  --allowerasing
 # Install glances for system monitoring
 dnf install -y glances

-# Install zsh with autocomplete and suggestions
-dnf install -y zsh zsh-autosuggestions zsh-syntax-highlighting
-
-cat <<EOF > ~/.zshrc
-# History
-HISTFILE=~/.zsh_history
-HISTSIZE=10000
-SAVEHIST=10000
-setopt appendhistory
-
-# Basic settings
-autoload bashcompinit && bashcompinit
-autoload -U compinit; compinit
-zstyle ':completion:*' menu select
-
-# Prompt settings
-autoload -Uz promptinit
-promptinit
-prompt redhat
-PROMPT_EOL_MARK=
-
-# Syntax Highlighting
-source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
-source /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh
-
-### Custom Commands and Aliases ###
-EOF
-
+# ZSH
+dnf install -y zsh
 chsh -s $(which zsh) && chsh -s $(which zsh) ducoterra
 ```

-## Downgrading Kernel
+### Disable Swap
+
+```bash
+swapoff -a
+zramctl --reset /dev/zram0
+dnf -y remove zram-generator-defaults
+```
+
+### Disable Selinux
+
+By default selinux will be enforcing. You can set it to permissive with
+
+```bash
+setenforce 0
+```
+
+And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.
+
+### Downgrading Kernel

 ```bash
 dnf install koji
@@ -243,3 +470,153 @@ cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3

 reboot
 ```
+
+### Resize logical volume
+
+```bash
+# Replace /dev/sda2 with whatever your disks are
+# This assumes xfs
+pvresize /dev/sda2
+lvextend /dev/mapper/root -l+100%FREE
+xfs_growfs -d /dev/mapper/root
+```
+
+### Create XFS LVM
+
+<https://www.linuxtechi.com/how-to-create-lvm-partition-in-linux/>
+<https://www.golinuxcloud.com/lvcreate-command-in-linux/#How_to_install_lvcreate>
+
+If you get the error "Not creating system devices file due to existing VGs."
+Run `vgimportdevices -a` and check `/etc/lvm/devices/system.devices`
+
+1. Create a new partition for the Physical Volume (fdisk)
+
+```bash
+# Create the physical volume 
+pvcreate /dev/vda4
+# Create the volume group (vgcreate <vg_name>  <pv>)
+vgcreate nextcloud_data /dev/vda4
+# Create the logical volume (lvcreate -L <Size-of-LV> -n <LV-Name>   <VG-Name>)
+# Or lvcreate -l 100%FREE
+lvcreate -l 100%FREE -n nextcloud_data_vol nextcloud_data
+
+# list the PV, VG, LV
+pvs
+vgs
+lvs
+
+# Format lv
+mkfs.btrfs /dev/nextcloud_data/nextcloud_data_vol
+```
+
+### LVM Thin Provisioning
+
+<https://linuxconfig.org/introduction-to-lvm-thin-provisioning>
+
+If you get the error "Not creating system devices file due to existing VGs."
+Run `vgimportdevices -a` and check `/etc/lvm/devices/system.devices`
+
+Thin provisioning allows you to overprovision your storage drives to make the filesystem
+think it has more data than it does.
+
+```bash
+# Create the physical volume
+pvcreate /dev/vda4
+# Create the volume group
+vgcreate vg0 /dev/vda4
+
+# Create the thin pool - the volume with real data that will hold our thing volumes with fake data
+lvcreate -l 100%FREE -T vg0/thinpool
+
+# Create the thin volumes with fake data
+lvcreate -T -V 2T vg0/thinpool -n local-path-provisioner
+lvcreate -T -V 2T vg0/thinpool -n docker-data
+
+# Format the fake volumes
+mkfs.xfs /dev/mapper/vg0-local--path--provisioner
+mkfs.xfs /dev/mapper/vg0-docker--data
+```
+
+### Set eui64 on network interface
+
+```bash
+nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64
+```
+
+### Install and Enable Cockpit
+
+<https://cockpit-project.org/running>
+
+```bash
+dnf install cockpit
+systemctl enable --now cockpit.socket
+firewall-cmd --add-service=cockpit
+firewall-cmd --add-service=cockpit --permanent
+```
+
+## Troubleshooting
+
+### Cockpit Terminal Unusable or Weird Colors
+
+Make sure you give canvas access to the browser (especially in librewolf)
+
+### Chroot into a mounted disk
+
+This lets you run grub2-mkconfig among other things.
+
+```bash
+# Mount root
+mount /dev/mapper/vg0-root /mnt
+
+# Mount proc, sys, and dev
+mount -t proc /proc proc/
+mount --rbind /sys sys/
+mount --rbind /dev dev/
+
+# Mount boot and efi
+mount /dev/vdb2 /mnt/boot
+mount /dev/vdb1 /mnt/boot/efi
+
+chroot /mnt
+```
+
+### Resize Last Partition to Fill Available Space
+
+```bash
+parted /dev/vdb
+# to resize /dev/vdb3 to fill 100% of the disk, for example
+resizepart 3 100%
+quit
+# Resize the physical volume to match the partition
+pvresize /dev/vdb3
+```
+
+### LUKS performance
+
+```bash
+cryptsetup benchmark
+```
+
+Should output something like:
+
+```bash
+#     Algorithm |       Key |      Encryption |      Decryption
+        aes-cbc        128b      1409.1 MiB/s      3627.9 MiB/s
+    serpent-cbc        128b       146.5 MiB/s       981.4 MiB/s
+    twofish-cbc        128b       289.8 MiB/s       613.3 MiB/s
+        aes-cbc        256b      1100.2 MiB/s      3448.2 MiB/s
+    serpent-cbc        256b       150.3 MiB/s       982.1 MiB/s
+    twofish-cbc        256b       294.3 MiB/s       590.8 MiB/s
+        aes-xts        256b      4423.5 MiB/s      4561.2 MiB/s
+    serpent-xts        256b       874.9 MiB/s       883.7 MiB/s
+    twofish-xts        256b       557.8 MiB/s       559.4 MiB/s
+        aes-xts        512b      4551.2 MiB/s      4669.6 MiB/s
+    serpent-xts        512b       890.8 MiB/s       860.5 MiB/s
+    twofish-xts        512b       557.5 MiB/s       564.2 MiB/s
+```
+
+Which will tell you how fast you can theoretically write/read to encrypted drives.
+
+The default encryption used by most modern operating systems is AES-XTS.
+
+You can see your system's cipher and key with `cryptsetup luksDump /dev/nvme0n1p1  | grep -i cipher`