homelab/templates/os/foobar.md

# Foobar

These docs frequently reference an "operator" and a "client". The operator is
your laptop, computer, pipeline, or any device will be connecting to, or
configuring, the OS during/after install. The client is the device you're
installing the OS onto.

- [Foobar](#foobar)
  - [Installation](#installation)
  - [Network](#network)
    - [Firewalld Configuration](#firewalld-configuration)
    - [Setup SSH](#setup-ssh)
    - [Fail2Ban](#fail2ban)
    - [Set eui64 on network interface](#set-eui64-on-network-interface)
    - [Set up Network Bridge](#set-up-network-bridge)
  - [Storage](#storage)
    - [BTRFS Parent Volumes](#btrfs-parent-volumes)
    - [BTRFS Snapshots](#btrfs-snapshots)
    - [BTRFS Maintenance](#btrfs-maintenance)
    - [TPM2 Luks Decryption](#tpm2-luks-decryption)
  - [Users](#users)
    - [Change your password](#change-your-password)
  - [Updates](#updates)
    - [Automatic Updates](#automatic-updates)
  - [Monitoring](#monitoring)
    - [Disk Usage](#disk-usage)
    - [Disk Wear](#disk-wear)
  - [Backups](#backups)
    - [Downgrading Kernel](#downgrading-kernel)
  - [Apps](#apps)
    - [Package Manager](#package-manager)
    - [Install and Enable Cockpit](#install-and-enable-cockpit)
  - [Install and Enable Virtualization](#install-and-enable-virtualization)
  - [Install and Enable Containers](#install-and-enable-containers)
  - [Troubleshooting](#troubleshooting)
    - [Disable Swap](#disable-swap)
    - [Disable Selinux](#disable-selinux)

## Installation

1. Configure network first
   1. Set a hostname
   2. Disable ipv6 privacy extensions
2. Software Selection
   1. Headless Management
3. User Creation
   1. Set a simple password, we'll change it later
4. Disk partitioning
   1. Select manual (blivet) partitioning
   2. Create a 1GB EFI system partition and mount it at `/boot/efi`
   3. Create a 1GB ext4 partition and mount it at `/boot`
   4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it
   5. Create a btrfs subvolume called "root" and mount it at `/`
   6. Create any other btrfs subvolumes you might need
5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time.
6. Install and reboot

## Network

### Firewalld Configuration

Set the default firewalld zone to `public`

```bash
# Note, you probably don't have to do this. Check Cockpit Network -> Firewall
# firewall-cmd --set-default-zone=public
```

Firewalld will be on and blocking by default. You can check the zone and allowed ports with:

```bash
firewall-cmd --zone=public --list-ports
firewall-cmd --zone=public --list-services
```

Allow Cockpit with

```bash
firewall-cmd --permanent --zone=public --add-port=9090/tcp
firewall-cmd --reload
```

### Setup SSH

See [README](/README.md#ssh-key-generation)

### Fail2Ban

On the server:

```bash
# Run tmux session
tmux

dnf install -y fail2ban

# Setup initial rules
cat <<EOF > /etc/fail2ban/jail.local
# Jail configuration additions for local installation

# Adjust the default configuration's default values
[DEFAULT]
# Optional enter an trusted IP never to ban
# ignoreip = 2600:1700:1e6c:a81f::0/64
bantime  = 6600
backend = auto

# The main configuration file defines all services but
# deactivates them by default. We have to activate those neeeded
[sshd]
enabled = true
EOF

systemctl enable fail2ban --now

# OPTIONAL: follow logs
tail -f /var/log/fail2ban.log
```

Checking, banning, unbanning

```bash
# See banned clients
fail2ban-client banned
# See jails (sshd should be one of them)
fail2ban-client status
# Unban a client from the sshd jail
fail2ban-client set sshd unbanip <IP address>
```

### Set eui64 on network interface

Ensures consistent mac-based IPv6 address.

```bash
nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64
```

### Set up Network Bridge

Networking -> Add bridge -> add network interface and save

```bash
nmcli connection modify bridge0 ipv6.addr-gen-mode eui64
```

## Storage

### BTRFS Parent Volumes

In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see
all subvolumes.

```conf
UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
```

```bash
systemctl daemon-reload
mount -a --mkdir
```

### BTRFS Snapshots

<https://en.opensuse.org/openSUSE:Snapper_Tutorial>

<http://snapper.io/manpages/snapper-configs.html>

We'll be using snapper, a tool for automating and controlling snapshot behavior.

```bash
dnf install snapper dnf-plugin-snapper

# Allow selinux management
semanage permissive -a snapperd_t

# Note, if you mess something up you can run snapper -c root delete-config to delete
# System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper
snapper -c root create-config /
snapper -c data create-config /path/to/other/data

# Enable automatic snapshots
systemctl enable --now snapper-timeline.timer
# Enable automatic cleanup
systemctl enable --now snapper-cleanup.timer
# Enable snapshots on boot
systemctl enable --now snapper-boot.timer

# List snapshots
snapper -c root list
# Create snapshot manually
snapper -c root create --description "test snapshot"
# Delete first snapshot
snapper -c root delete 1
```

Note - you probably don't want to keep yearly snapshots.
Edit `/etc/snapper/configs/root` and change `TIMELINE_LIMIT_YEARLY=` to `0`.

### BTRFS Maintenance

```bash
# Start a scrub in the foreground (-B) at /
btrfs scrub start -B /
```

### TPM2 Luks Decryption

Mostly taken from here:
<https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>

PCR reference for `--tpm2-pcrs` args

```text
0: System firmware executable
2: Kernel
4: Bootloader
7: Secure boot state
8: Cmdline
9: Initrd
```

Note, if your threat vector is people trying to get data off your old disks after throwing them
away, you can set `--tpm2-pcrs=""`. Someone could gain access to your encrypted partition if they
can access your machine physically by manipulating the boot parameters but you're guaranteed to
unlock despite updates and upgrades.

Basic commands:

```bash
# Run tmux session
tmux

# Show tpm2 devices
systemd-cryptenroll --tpm2-device=list
# Show crypto luks block devices
blkid -t TYPE=crypto_LUKS

# Enroll the tpm2 device with systemd-cryptenroll
systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs=""

####################
##### OPTIONAL #####
####################
# If you have lots of devices to decrypt (like a btrfs raid array), use these commands.
# Get all crypto luks partitions
blkid | grep crypto_LUKS
# List them all space-separated and drop the '/dev'
LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1"

# Check that your list is good
for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done

# Enroll
for dev in $LUKS_DEVS; do \
echo "Enrolling /dev/$dev"; \
systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \
done
########################
##### END OPTIONAL #####
########################

# Append to command line args
echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
dracut -f
```

Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX

```bash
# Update Grub
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
# Cross your fingers that you don't have to go type in the password manually.
# Yes, 60 full seconds is too long. Go type your password in.
```

If you need to reenroll for some reason:

```bash
# Reenroll
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
```

## Users

### Change your password

In Cockpit navigate to Accounts -> user -> Set password

## Updates

### Automatic Updates

In Cockpit navigate to software updates -> automatic updates -> install -> security updates only

## Monitoring

In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics

### Disk Usage

TODO

### Disk Wear

TODO

## Backups

See [borg.md](/active/systemd_borg/borg.md)

### Downgrading Kernel

```bash
dnf install koji

# Note: format is kernel-version.fedora-version
cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3-300.fc41 && dnf install ./*

reboot
```

## Apps

### Package Manager

Configure dnf to use the fastest mirror:

```bash
echo 'fastestmirror=1' >> /etc/dnf/dnf.conf
dnf clean all
dnf update --refresh -y
# libdnf5 is required for ansible to work
dnf install -y glances tmux vim python3-libdnf5
```

### Install and Enable Cockpit

<https://cockpit-project.org/running>

```bash
dnf install cockpit
systemctl enable --now cockpit.socket
firewall-cmd --add-service=cockpit
firewall-cmd --add-service=cockpit --permanent
```

## Install and Enable Virtualization

Don't forget to add a btrfs subvolume for `/var/lib/libvirt`

```bash
# Since we already created our /btrfs mountpoint, this volume will show up automatically
# at /btrfs/libvirt
btrfs sub create /btrfs/libvirt
```

Now create an fstab entry that mounts the volume at /var/lib/libvirt

```bash
UUID=... /var/lib/libvirt    btrfs   subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0
```

Mount the libvirt volume:

```bash
systemctl daemon-reload
mount -a --mkdir
# Check that the mount was successful. This will print something if our mount worked.
mount | grep -i /var/lib/libvirt
```

Create a snapshot schedule for libvirt.

```bash
snapper -c libvirt create-config /var/lib/libvirt
# Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt
```

Install and enable the virtualization service.

```bash
dnf group install --with-optional virtualization
systemctl enable --now libvirtd
```

Install the cockpit machines application.

## Install and Enable Containers

## Troubleshooting

### Disable Swap

```bash
swapoff -a
zramctl --reset /dev/zram0
dnf -y remove zram-generator-defaults
```

### Disable Selinux

By default selinux will be enforcing. You can set it to permissive with

```bash
setenforce 0
```

And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.