# Foobar These docs frequently reference an "operator" and a "client". The operator is your laptop, computer, pipeline, or any device will be connecting to, or configuring, the OS during/after install. The client is the device you're installing the OS onto. - [Foobar](#foobar) - [Installation](#installation) - [Network](#network) - [Firewalld Configuration](#firewalld-configuration) - [Setup SSH](#setup-ssh) - [Fail2Ban](#fail2ban) - [Set eui64 on network interface](#set-eui64-on-network-interface) - [Set up Network Bridge](#set-up-network-bridge) - [Storage](#storage) - [BTRFS Parent Volumes](#btrfs-parent-volumes) - [BTRFS Snapshots](#btrfs-snapshots) - [BTRFS Maintenance](#btrfs-maintenance) - [TPM2 Luks Decryption](#tpm2-luks-decryption) - [Users](#users) - [Change your password](#change-your-password) - [Updates](#updates) - [Automatic Updates](#automatic-updates) - [Monitoring](#monitoring) - [Disk Usage](#disk-usage) - [Disk Wear](#disk-wear) - [Backups](#backups) - [Downgrading Kernel](#downgrading-kernel) - [Apps](#apps) - [Package Manager](#package-manager) - [Install and Enable Cockpit](#install-and-enable-cockpit) - [Install and Enable Virtualization](#install-and-enable-virtualization) - [Install and Enable Containers](#install-and-enable-containers) - [Troubleshooting](#troubleshooting) - [Disable Swap](#disable-swap) - [Disable Selinux](#disable-selinux) ## Installation 1. Configure network first 1. Set a hostname 2. Disable ipv6 privacy extensions 2. Software Selection 1. Headless Management 3. User Creation 1. Set a simple password, we'll change it later 4. Disk partitioning 1. Select manual (blivet) partitioning 2. Create a 1GB EFI system partition and mount it at `/boot/efi` 3. Create a 1GB ext4 partition and mount it at `/boot` 4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it 5. Create a btrfs subvolume called "root" and mount it at `/` 6. Create any other btrfs subvolumes you might need 5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time. 6. Install and reboot ## Network ### Firewalld Configuration Set the default firewalld zone to `public` ```bash # Note, you probably don't have to do this. Check Cockpit Network -> Firewall # firewall-cmd --set-default-zone=public ``` Firewalld will be on and blocking by default. You can check the zone and allowed ports with: ```bash firewall-cmd --zone=public --list-ports firewall-cmd --zone=public --list-services ``` Allow Cockpit with ```bash firewall-cmd --permanent --zone=public --add-port=9090/tcp firewall-cmd --reload ``` ### Setup SSH See [README](/README.md#ssh-key-generation) ### Fail2Ban On the server: ```bash # Run tmux session tmux dnf install -y fail2ban # Setup initial rules cat < /etc/fail2ban/jail.local # Jail configuration additions for local installation # Adjust the default configuration's default values [DEFAULT] # Optional enter an trusted IP never to ban # ignoreip = 2600:1700:1e6c:a81f::0/64 bantime = 6600 backend = auto # The main configuration file defines all services but # deactivates them by default. We have to activate those neeeded [sshd] enabled = true EOF systemctl enable fail2ban --now # OPTIONAL: follow logs tail -f /var/log/fail2ban.log ``` Checking, banning, unbanning ```bash # See banned clients fail2ban-client banned # See jails (sshd should be one of them) fail2ban-client status # Unban a client from the sshd jail fail2ban-client set sshd unbanip ``` ### Set eui64 on network interface Ensures consistent mac-based IPv6 address. ```bash nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64 ``` ### Set up Network Bridge Networking -> Add bridge -> add network interface and save ```bash nmcli connection modify bridge0 ipv6.addr-gen-mode eui64 ``` ## Storage ### BTRFS Parent Volumes In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see all subvolumes. ```conf UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root btrfs subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0 UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data btrfs subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0 ``` ```bash systemctl daemon-reload mount -a --mkdir ``` ### BTRFS Snapshots We'll be using snapper, a tool for automating and controlling snapshot behavior. ```bash dnf install snapper dnf-plugin-snapper # Allow selinux management semanage permissive -a snapperd_t # Note, if you mess something up you can run snapper -c root delete-config to delete # System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper snapper -c root create-config / snapper -c data create-config /path/to/other/data # Enable automatic snapshots systemctl enable --now snapper-timeline.timer # Enable automatic cleanup systemctl enable --now snapper-cleanup.timer # Enable snapshots on boot systemctl enable --now snapper-boot.timer # List snapshots snapper -c root list # Create snapshot manually snapper -c root create --description "test snapshot" # Delete first snapshot snapper -c root delete 1 ``` Note - you probably don't want to keep yearly snapshots. Edit `/etc/snapper/configs/root` and change `TIMELINE_LIMIT_YEARLY=` to `0`. ### BTRFS Maintenance ```bash # Start a scrub in the foreground (-B) at / btrfs scrub start -B / ``` ### TPM2 Luks Decryption Mostly taken from here: PCR reference for `--tpm2-pcrs` args ```text 0: System firmware executable 2: Kernel 4: Bootloader 7: Secure boot state 8: Cmdline 9: Initrd ``` Note, if your threat vector is people trying to get data off your old disks after throwing them away, you can set `--tpm2-pcrs=""`. Someone could gain access to your encrypted partition if they can access your machine physically by manipulating the boot parameters but you're guaranteed to unlock despite updates and upgrades. Basic commands: ```bash # Run tmux session tmux # Show tpm2 devices systemd-cryptenroll --tpm2-device=list # Show crypto luks block devices blkid -t TYPE=crypto_LUKS # Enroll the tpm2 device with systemd-cryptenroll systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs="" #################### ##### OPTIONAL ##### #################### # If you have lots of devices to decrypt (like a btrfs raid array), use these commands. # Get all crypto luks partitions blkid | grep crypto_LUKS # List them all space-separated and drop the '/dev' LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1" # Check that your list is good for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done # Enroll for dev in $LUKS_DEVS; do \ echo "Enrolling /dev/$dev"; \ systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \ done ######################## ##### END OPTIONAL ##### ######################## # Append to command line args echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf dracut -f ``` Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX ```bash # Update Grub grub2-mkconfig -o /boot/grub2/grub.cfg reboot # Cross your fingers that you don't have to go type in the password manually. # Yes, 60 full seconds is too long. Go type your password in. ``` If you need to reenroll for some reason: ```bash # Reenroll systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs="" ``` ## Users ### Change your password In Cockpit navigate to Accounts -> user -> Set password ## Updates ### Automatic Updates In Cockpit navigate to software updates -> automatic updates -> install -> security updates only ## Monitoring In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics ### Disk Usage TODO ### Disk Wear TODO ## Backups See [borg.md](/active/systemd_borg/borg.md) ### Downgrading Kernel ```bash dnf install koji # Note: format is kernel-version.fedora-version cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3-300.fc41 && dnf install ./* reboot ``` ## Apps ### Package Manager Configure dnf to use the fastest mirror: ```bash echo 'fastestmirror=1' >> /etc/dnf/dnf.conf dnf clean all dnf update --refresh -y # libdnf5 is required for ansible to work dnf install -y glances tmux vim python3-libdnf5 ``` ### Install and Enable Cockpit ```bash dnf install cockpit systemctl enable --now cockpit.socket firewall-cmd --add-service=cockpit firewall-cmd --add-service=cockpit --permanent ``` ## Install and Enable Virtualization Don't forget to add a btrfs subvolume for `/var/lib/libvirt` ```bash # Since we already created our /btrfs mountpoint, this volume will show up automatically # at /btrfs/libvirt btrfs sub create /btrfs/libvirt ``` Now create an fstab entry that mounts the volume at /var/lib/libvirt ```bash UUID=... /var/lib/libvirt btrfs subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0 ``` Mount the libvirt volume: ```bash systemctl daemon-reload mount -a --mkdir # Check that the mount was successful. This will print something if our mount worked. mount | grep -i /var/lib/libvirt ``` Create a snapshot schedule for libvirt. ```bash snapper -c libvirt create-config /var/lib/libvirt # Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt ``` Install and enable the virtualization service. ```bash dnf group install --with-optional virtualization systemctl enable --now libvirtd ``` Install the cockpit machines application. ## Install and Enable Containers ## Troubleshooting ### Disable Swap ```bash swapoff -a zramctl --reset /dev/zram0 dnf -y remove zram-generator-defaults ``` ### Disable Selinux By default selinux will be enforcing. You can set it to permissive with ```bash setenforce 0 ``` And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.