homelab/templates/os/foobar.md

Foobar

These docs frequently reference an "operator" and a "client". The operator is your laptop, computer, pipeline, or any device will be connecting to, or configuring, the OS during/after install. The client is the device you're installing the OS onto.

• Foobar
  • Installation
  • Network
    • Firewalld Configuration
    • Setup SSH
    • Fail2Ban
    • Set eui64 on network interface
    • Set up Network Bridge
  • Storage
    • BTRFS Parent Volumes
    • BTRFS Snapshots
    • BTRFS Maintenance
    • TPM2 Luks Decryption
  • Users
    • Change your password
  • Updates
    • Automatic Updates
  • Monitoring
    • Disk Usage
    • Disk Wear
  • Backups
    • Downgrading Kernel
  • Apps
    • Package Manager
    • Install and Enable Cockpit
  • Install and Enable Virtualization
  • Install and Enable Containers
  • Troubleshooting
    • Disable Swap
    • Disable Selinux

Installation

1. Configure network first
   1. Set a hostname
   2. Disable ipv6 privacy extensions
2. Software Selection
   1. Headless Management
3. User Creation
   1. Set a simple password, we'll change it later
4. Disk partitioning
   1. Select manual (blivet) partitioning
   2. Create a 1GB EFI system partition and mount it at /boot/efi
   3. Create a 1GB ext4 partition and mount it at /boot
   4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it
   5. Create a btrfs subvolume called "root" and mount it at /
   6. Create any other btrfs subvolumes you might need
5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time.
6. Install and reboot

Network

Firewalld Configuration

Set the default firewalld zone to public

# Note, you probably don't have to do this. Check Cockpit Network -> Firewall
# firewall-cmd --set-default-zone=public

Firewalld will be on and blocking by default. You can check the zone and allowed ports with:

firewall-cmd --zone=public --list-ports
firewall-cmd --zone=public --list-services

Allow Cockpit with

firewall-cmd --permanent --zone=public --add-port=9090/tcp
firewall-cmd --reload

Setup SSH

See README

Fail2Ban

On the server:

# Run tmux session
tmux

dnf install -y fail2ban

# Setup initial rules
cat <<EOF > /etc/fail2ban/jail.local
# Jail configuration additions for local installation

# Adjust the default configuration's default values
[DEFAULT]
# Optional enter an trusted IP never to ban
# ignoreip = 2600:1700:1e6c:a81f::0/64
bantime  = 6600
backend = auto

# The main configuration file defines all services but
# deactivates them by default. We have to activate those neeeded
[sshd]
enabled = true
EOF

systemctl enable fail2ban --now

# OPTIONAL: follow logs
tail -f /var/log/fail2ban.log

Checking, banning, unbanning

# See banned clients
fail2ban-client banned
# See jails (sshd should be one of them)
fail2ban-client status
# Unban a client from the sshd jail
fail2ban-client set sshd unbanip <IP address>

Set eui64 on network interface

Ensures consistent mac-based IPv6 address.

nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64

Set up Network Bridge

Networking -> Add bridge -> add network interface and save

nmcli connection modify bridge0 ipv6.addr-gen-mode eui64

Storage

BTRFS Parent Volumes

In /etc/fstab, add the parent volumes for your disks mounted with subvolid=5 at /btrfs so you can see all subvolumes.

UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0

systemctl daemon-reload
mount -a --mkdir

BTRFS Snapshots

https://en.opensuse.org/openSUSE:Snapper_Tutorial

http://snapper.io/manpages/snapper-configs.html

We'll be using snapper, a tool for automating and controlling snapshot behavior.

dnf install snapper dnf-plugin-snapper

# Allow selinux management
semanage permissive -a snapperd_t

# Note, if you mess something up you can run snapper -c root delete-config to delete
# System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper
snapper -c root create-config /
snapper -c data create-config /path/to/other/data

# Enable automatic snapshots
systemctl enable --now snapper-timeline.timer
# Enable automatic cleanup
systemctl enable --now snapper-cleanup.timer
# Enable snapshots on boot
systemctl enable --now snapper-boot.timer

# List snapshots
snapper -c root list
# Create snapshot manually
snapper -c root create --description "test snapshot"
# Delete first snapshot
snapper -c root delete 1

Note - you probably don't want to keep yearly snapshots. Edit /etc/snapper/configs/root and change TIMELINE_LIMIT_YEARLY= to 0.

BTRFS Maintenance

# Start a scrub in the foreground (-B) at /
btrfs scrub start -B /

TPM2 Luks Decryption

Mostly taken from here: https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95

PCR reference for --tpm2-pcrs args

0: System firmware executable
2: Kernel
4: Bootloader
7: Secure boot state
8: Cmdline
9: Initrd

Note, if your threat vector is people trying to get data off your old disks after throwing them away, you can set --tpm2-pcrs="". Someone could gain access to your encrypted partition if they can access your machine physically by manipulating the boot parameters but you're guaranteed to unlock despite updates and upgrades.

Basic commands:

# Run tmux session
tmux

# Show tpm2 devices
systemd-cryptenroll --tpm2-device=list
# Show crypto luks block devices
blkid -t TYPE=crypto_LUKS

# Enroll the tpm2 device with systemd-cryptenroll
systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs=""

####################
##### OPTIONAL #####
####################
# If you have lots of devices to decrypt (like a btrfs raid array), use these commands.
# Get all crypto luks partitions
blkid | grep crypto_LUKS
# List them all space-separated and drop the '/dev'
LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1"

# Check that your list is good
for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done

# Enroll
for dev in $LUKS_DEVS; do \
echo "Enrolling /dev/$dev"; \
systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \
done
########################
##### END OPTIONAL #####
########################

# Append to command line args
echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
dracut -f

Finally, vim /etc/default/grub and add rd.luks.options=tpm2-device=auto to GRUB_CMDLINE_LINUX

# Update Grub
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
# Cross your fingers that you don't have to go type in the password manually.
# Yes, 60 full seconds is too long. Go type your password in.

If you need to reenroll for some reason:

# Reenroll
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""

Users

Change your password

In Cockpit navigate to Accounts -> user -> Set password

Updates

Automatic Updates

In Cockpit navigate to software updates -> automatic updates -> install -> security updates only

Monitoring

In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics

Disk Usage

TODO

Disk Wear

TODO

Backups

See borg.md

Downgrading Kernel

dnf install koji

# Note: format is kernel-version.fedora-version
cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3-300.fc41 && dnf install ./*

reboot

Apps

Package Manager

Configure dnf to use the fastest mirror:

echo 'fastestmirror=1' >> /etc/dnf/dnf.conf
dnf clean all
dnf update --refresh -y
# libdnf5 is required for ansible to work
dnf install -y glances tmux vim python3-libdnf5

Install and Enable Cockpit

https://cockpit-project.org/running

dnf install cockpit
systemctl enable --now cockpit.socket
firewall-cmd --add-service=cockpit
firewall-cmd --add-service=cockpit --permanent

Install and Enable Virtualization

Don't forget to add a btrfs subvolume for /var/lib/libvirt

# Since we already created our /btrfs mountpoint, this volume will show up automatically
# at /btrfs/libvirt
btrfs sub create /btrfs/libvirt

Now create an fstab entry that mounts the volume at /var/lib/libvirt

UUID=... /var/lib/libvirt    btrfs   subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0

Mount the libvirt volume:

systemctl daemon-reload
mount -a --mkdir
# Check that the mount was successful. This will print something if our mount worked.
mount | grep -i /var/lib/libvirt

Create a snapshot schedule for libvirt.

snapper -c libvirt create-config /var/lib/libvirt
# Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt

Install and enable the virtualization service.

dnf group install --with-optional virtualization
systemctl enable --now libvirtd

Install the cockpit machines application.

Install and Enable Containers

Troubleshooting

Disable Swap

swapoff -a
zramctl --reset /dev/zram0
dnf -y remove zram-generator-defaults

Disable Selinux

By default selinux will be enforcing. You can set it to permissive with

setenforce 0

And then make it permanent by editing /etc/selinux/config and inserting SELINUX=permissive.