homelab/active/os_fedora/fedora-server.md

# Fedora Server

- [Fedora Server](#fedora-server)
  - [Installation](#installation)
  - [Power Profiles with Tuned](#power-profiles-with-tuned)
  - [Setup SSH](#setup-ssh)
  - [DNF](#dnf)
  - [Fail2Ban](#fail2ban)
  - [BTRFS Parent Volumes](#btrfs-parent-volumes)
  - [BTRFS Snapshots](#btrfs-snapshots)
    - [Snapper Installation](#snapper-installation)
    - [Snapper Cleanup](#snapper-cleanup)
  - [BTRFS Maintenance](#btrfs-maintenance)
  - [TPM2 Luks Decryption](#tpm2-luks-decryption)
  - [Change your password](#change-your-password)
  - [Automatic Updates](#automatic-updates)
  - [Monitoring](#monitoring)
    - [Glances](#glances)
    - [Disk Usage](#disk-usage)
    - [Disk Wear](#disk-wear)
  - [Common Storage Mounts](#common-storage-mounts)
  - [Network Bridge](#network-bridge)
  - [Virtualization](#virtualization)
    - [Virtualization Troubleshooting](#virtualization-troubleshooting)
    - [QEMU Images](#qemu-images)
    - [Shared directory with VM Guest](#shared-directory-with-vm-guest)
  - [Firewalld](#firewalld)
  - [Backups](#backups)
    - [Connect to the ISCSI Backup Target](#connect-to-the-iscsi-backup-target)
      - [Connect to Backup Target with Cockpit](#connect-to-backup-target-with-cockpit)
      - [Connect to Backup Target with iscsiadm](#connect-to-backup-target-with-iscsiadm)
      - [Format backup disk](#format-backup-disk)
      - [Troubleshooting Backup ISCSI Connection](#troubleshooting-backup-iscsi-connection)
    - [Quick Backup](#quick-backup)
    - [Regular Backups with Borg](#regular-backups-with-borg)
  - [Version Upgrades](#version-upgrades)
  - [Optional Steps](#optional-steps)
    - [Disable Swap](#disable-swap)
    - [Disable Selinux](#disable-selinux)
    - [Downgrading Kernel](#downgrading-kernel)
    - [Resize logical volume](#resize-logical-volume)
    - [Create XFS LVM](#create-xfs-lvm)
    - [LVM Thin Provisioning](#lvm-thin-provisioning)
    - [Set eui64 on network interface](#set-eui64-on-network-interface)
    - [Install and Enable Cockpit](#install-and-enable-cockpit)
  - [Troubleshooting](#troubleshooting)
    - [Cockpit Terminal Unusable or Weird Colors](#cockpit-terminal-unusable-or-weird-colors)
    - [Chroot into a mounted disk](#chroot-into-a-mounted-disk)
    - [Resize Last Partition to Fill Available Space](#resize-last-partition-to-fill-available-space)
    - [LUKS performance](#luks-performance)

<https://docs.fedoraproject.org/en-US/fedora-server/installation/postinstallation-tasks/#_manage_system_updates>

Note these instructions differentiate between an `operator` and a `server`. The operator can be
any machine that configure the server. A pipeline, laptop, dedicated server, etc. are all options.
The server can be its own operator, though that's not recommended since servers should be ephemeral
and the operator will store information about each server.

## Installation

1. Configure network first
   1. Set a hostname
   2. Disable ipv6 privacy extensions
2. Software Selection
   1. Headless Management
3. User Creation
   1. Set a simple password, we'll change it later
4. Disk partitioning
   1. Select manual (blivet) partitioning
   2. Create a 1GB EFI system partition and mount it at `/boot/efi`
   3. Create a 1GB ext4 partition and mount it at `/boot`
   4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it
   5. Create a btrfs subvolume called "root" and mount it at `/`
   6. Create any other btrfs subvolumes you might need
5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time.
6. Install and reboot

## Power Profiles with Tuned

1. `dnf install tuned`
2. `systemctl enable --now tuned`
3. `tuned-adm profile virtual-host`

## Setup SSH

See [README](/README.md#ssh-setup)

## DNF

Configure dnf to use the fastest mirror:

```bash
echo 'fastestmirror=1' >> /etc/dnf/dnf.conf
dnf clean all
dnf update --refresh -y
# libdnf5 is required for ansible to work
dnf install -y glances tmux vim python3-libdnf5
```

## Fail2Ban

On the server:

```bash
# Run tmux session
tmux

dnf install -y fail2ban

# Setup initial rules
cat <<EOF > /etc/fail2ban/jail.local
# Jail configuration additions for local installation

# Adjust the default configuration's default values
[DEFAULT]
# Optional enter an trusted IP never to ban
# ignoreip = 2600:1700:1e6c:a81f::0/64
bantime  = 6600
backend = auto

# The main configuration file defines all services but
# deactivates them by default. We have to activate those neeeded
[sshd]
enabled = true
EOF

systemctl enable fail2ban --now

# OPTIONAL: follow logs
tail -f /var/log/fail2ban.log
```

Checking, banning, unbanning

```bash
# See banned clients
fail2ban-client banned
# See jails (sshd should be one of them)
fail2ban-client status
# Unban a client from the sshd jail
fail2ban-client set sshd unbanip <IP address>
```

## BTRFS Parent Volumes

In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see
all subvolumes.

```conf
UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
```

```bash
systemctl daemon-reload
mount -a --mkdir
```

## BTRFS Snapshots

<https://en.opensuse.org/openSUSE:Snapper_Tutorial>

<http://snapper.io/manpages/snapper-configs.html>

### Snapper Installation

We'll be using snapper, a tool for automating and controlling snapshot behavior.

```bash
dnf install snapper dnf-plugin-snapper

# Allow selinux management
semanage permissive -a snapperd_t

# Note, if you mess something up you can run snapper -c root delete-config to delete
# System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper
snapper -c root create-config /
snapper -c data create-config /path/to/other/data

# Enable automatic snapshots
systemctl enable --now snapper-timeline.timer
# Enable automatic cleanup
systemctl enable --now snapper-cleanup.timer
# Enable snapshots on boot
systemctl enable --now snapper-boot.timer
```

### Snapper Cleanup

```bash
# List snapshots
snapper -c root list
# Create snapshot manually
snapper -c root create --description "test snapshot"
# Delete first snapshot
snapper -c root delete 1
# Delete snapshots between 655-857
snapper -c root delete 655-857  
```

Note - you probably don't want to keep yearly snapshots.
Edit `/etc/snapper/configs/root` and change `TIMELINE_LIMIT_YEARLY=` to `0`.

## BTRFS Maintenance

```bash
# Start a scrub with low impact/priority at / (good for servers)
btrfs scrub start -c idle /

# Start a scrub in the foreground and monitor
btrfs scrub start -c idle -B -d /

# Check for errors
dmesg -T | grep btrfs
```

## TPM2 Luks Decryption

Mostly taken from here:
<https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>

PCR reference for `--tpm2-pcrs` args

```text
0: System firmware executable
2: Kernel
4: Bootloader
7: Secure boot state
8: Cmdline
9: Initrd
```

Note, if your threat vector is people trying to get data off your old disks after throwing them
away, you can set `--tpm2-pcrs=""`. Someone could gain access to your encrypted partition if they
can access your machine physically by manipulating the boot parameters but you're guaranteed to
unlock despite updates and upgrades.

Basic commands:

```bash
# Run tmux session
tmux

# Show tpm2 devices
systemd-cryptenroll --tpm2-device=list
# Show crypto luks block devices
blkid -t TYPE=crypto_LUKS

# Enroll the tpm2 device with systemd-cryptenroll
systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs=""

####################
##### OPTIONAL #####
####################
# If you have lots of devices to decrypt (like a btrfs raid array), use these commands.
# Get all crypto luks partitions
blkid | grep crypto_LUKS
# List them all space-separated and drop the '/dev'
LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1"

# Check that your list is good
for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done

# Enroll
for dev in $LUKS_DEVS; do \
echo "Enrolling /dev/$dev"; \
systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \
done
########################
##### END OPTIONAL #####
########################

# Append to command line args
echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
dracut -f
```

Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX

```bash
# Update Grub
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot
# Cross your fingers that you don't have to go type in the password manually.
# Yes, 60 full seconds is too long. Go type your password in.
```

If you need to reenroll for some reason:

```bash
# Reenroll
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
```

## Change your password

In Cockpit navigate to Accounts -> user -> Set password

## Automatic Updates

In Cockpit navigate to software updates -> automatic updates -> install -> security updates only

## Monitoring

In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics

### Glances

```bash
dnf install -y glances python3-jinja2
systemctl enable --now glances
firewall-cmd --permanent --zone=FedoraServer --add-port=61208/tcp
firewall-cmd --reload
```

### Disk Usage

```bash
# Show size of folder exclude snapshots
du --exclude .snapshots -sh .

# Show size of all files in your current dir
for folder in $(ls); do du --exclude .snapshots -sh $folder; done

# Calculate all folder sizes in current dir 
alias {dudir,dud}='du -h --max-depth 1 | sort -h'

# Calculate all file sizes in current dir
alias {dufile,duf}='ls -lhSr'
```

### Disk Wear

TODO

## Common Storage Mounts

Note: mount these before you install the relevant package!

1. For virtual machines: `/var/lib/libvirt`
2. For podman: `/var/lib/containers`
3. For docker: `/var/lib/docker`

## Network Bridge

Networking -> Add bridge -> add network interface and save

```bash
nmcli connection modify bridge0 ipv6.addr-gen-mode eui64
```

## Virtualization

Don't forget to add a btrfs subvolume for `/var/lib/libvirt`

```bash
# Since we already created our /btrfs mountpoint, this volume will show up automatically
# at /btrfs/libvirt
btrfs sub create /btrfs/libvirt
```

Now create an fstab entry that mounts the volume at /var/lib/libvirt

```bash
UUID=... /var/lib/libvirt    btrfs   subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0
```

Mount the libvirt volume:

```bash
systemctl daemon-reload
mount -a --mkdir
# Check that the mount was successful. This will print something if our mount worked.
mount | grep -i /var/lib/libvirt
```

Create a snapshot schedule for libvirt.

```bash
snapper -c libvirt create-config /var/lib/libvirt
# Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt
```

Install and enable the virtualization service.

```bash
dnf group install --with-optional virtualization
systemctl enable --now libvirtd
```

Install the cockpit machines application.

### Virtualization Troubleshooting

```bash
# Oops, I did this after I installed virtualization
rsync -av /var/lib/libvirt/ /btrfs/libvirt/
rm -rf /var/lib/libvirt
# Find the path to your btrfs volume
lsblk
mount -o subvol=libvirt /dev/mapper/luks-... /var/lib/libvirtd
```

### QEMU Images

```bash
# Grow an image to 2TB
qemu-img resize nextcloud_aio-fcfgp.qcow2 2T
```

```bash
# Convert OVA to img
qemu-img convert -f vmdk -O raw in.vmdk out.img

# Convert qcow2 to img
qemu-img convert -f qcow2 -O raw in.raw out.img
```

### Shared directory with VM Guest

```bash
mount -t virtiofs [mount tag] [mount point]
```

## Firewalld

Set the default firewalld zone to `public`

```bash
# Note, you probably don't have to do this. Check Cockpit Network -> Firewall
# firewall-cmd --set-default-zone=public
```

Firewalld will be on and blocking by default. You can check the zone and allowed ports with:

```bash
firewall-cmd --get-active-zones
firewall-cmd --get-default-zone
firewall-cmd --zone=public --list-ports
firewall-cmd --zone=public --list-services
```

Allow Cockpit with

```bash
firewall-cmd --permanent --zone=public --add-port=9090/tcp
firewall-cmd --reload
```

Remove cockpit with

```bash
firewall-cmd --permanent --zone=public --remove-port=9090/tcp
```

Add a custom source for a service

```bash
sudo firewall-cmd --new-zone=home --permanent
sudo firewall-cmd --zone=home --add-source=10.2.0.0/24 --permanent
sudo firewall-cmd --zone=home --add-port=10700/tcp --permanent
sudo firewall-cmd --reload
```

## Backups

Note: this assumes you've set up [an iscsi backup disk](/active/os_truenas/truenas.md#iscsi-backup-volumes)

### Connect to the ISCSI Backup Target

#### Connect to Backup Target with Cockpit

1. Storage -> Hamburger menu -> Add iSCSI portal
2. Type your portal address, username, and password

#### Connect to Backup Target with iscsiadm

<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/5/html/online_storage_reconfiguration_guide/iscsi-replacements_timeout#iscsi-replacements_timeout>

```bash
# Set username and password for discovered nodes
# Optionally you can add "-T <target name>" to specify which target has the username/password
iscsiadm -m node \
-o update \
-n node.session.auth.username -v username \
-n node.session.auth.password -v password

# Set replacement_timeout to 10 minutes in case server reboots
iscsiadm -m node \
-o update \
-n node.session.timeo.replacement_timeout -v 600

systemctl restart iscsid

# Discover targets
iscsiadm -m discovery -t st -p drivework.reeselink.com

# Login to all nodes
iscsiadm -m node -l
```

#### Format backup disk

```bash
# list disks
lsblk

# Create partition
fdisk /dev/sdx

# Format partition with btrfs
mkfs.btrfs /dev/sdx1

# Get the UUID
blkid /dev/sdx1
```

Update /etc/fstab with the iscsi disk details.

Note:

- `x-systemd.automount` which only mounts the device when it's accessed.
- `x-systemd.mount-timeout=30` allows a 30 second timeout
- `_netdev` ensures the device won't be mounted until after the network is available

```conf
UUID=... /btrfs/some-name    btrfs   subvolid=5,compress=zstd:1,x-systemd.automount,x-systemd.mount-timeout=30,_netdev 0 0
```

#### Troubleshooting Backup ISCSI Connection

```bash
# List targets
iscsiadm -m node

# Delete node
iscsiadm -m node -o delete -T iqn.2022-01.com.reeselink:driveripper:iqn.2022-01.com.reeselink:driveripper

# List discovered targets
iscsiadm -m discovery

# Delete from discovery db
iscsiadm -m discoverydb -t sendtargets -p driveripper.reeselink.com -o delete
```

### Quick Backup

```bash
rsync -av --progress --exclude '.snapshots' /btrfs/yellow/root /btrfs/backup-yellow --dry-run
```

### Regular Backups with Borg

See [borg.md](/active/systemd_borg/borg.md)

## Version Upgrades

```bash
# Make sure to be fully up to date first
dnf upgrade --refresh
reboot

# Set the releasever to the version you want to upgrade to
dnf system-upgrade download --releasever=43
dnf system-upgrade reboot
```

## Optional Steps

### Disable Swap

```bash
swapoff -a
zramctl --reset /dev/zram0
dnf -y remove zram-generator-defaults
```

### Disable Selinux

By default selinux will be enforcing. You can set it to permissive with

```bash
setenforce 0
```

And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.

### Downgrading Kernel

```bash
dnf install koji

# Note: format is kernel-version.fedora-version
cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3-300.fc41 && dnf install ./*

reboot
```

### Resize logical volume

```bash
# Replace /dev/sda2 with whatever your disks are
# This assumes xfs
pvresize /dev/sda2
lvextend /dev/mapper/root -l+100%FREE
xfs_growfs -d /dev/mapper/root
```

### Create XFS LVM

<https://www.linuxtechi.com/how-to-create-lvm-partition-in-linux/>
<https://www.golinuxcloud.com/lvcreate-command-in-linux/#How_to_install_lvcreate>

If you get the error "Not creating system devices file due to existing VGs."
Run `vgimportdevices -a` and check `/etc/lvm/devices/system.devices`

1. Create a new partition for the Physical Volume (fdisk)

```bash
# Create the physical volume 
pvcreate /dev/vda4
# Create the volume group (vgcreate <vg_name>  <pv>)
vgcreate nextcloud_data /dev/vda4
# Create the logical volume (lvcreate -L <Size-of-LV> -n <LV-Name>   <VG-Name>)
# Or lvcreate -l 100%FREE
lvcreate -l 100%FREE -n nextcloud_data_vol nextcloud_data

# list the PV, VG, LV
pvs
vgs
lvs

# Format lv
mkfs.btrfs /dev/nextcloud_data/nextcloud_data_vol
```

### LVM Thin Provisioning

<https://linuxconfig.org/introduction-to-lvm-thin-provisioning>

If you get the error "Not creating system devices file due to existing VGs."
Run `vgimportdevices -a` and check `/etc/lvm/devices/system.devices`

Thin provisioning allows you to overprovision your storage drives to make the filesystem
think it has more data than it does.

```bash
# Create the physical volume
pvcreate /dev/vda4
# Create the volume group
vgcreate vg0 /dev/vda4

# Create the thin pool - the volume with real data that will hold our thing volumes with fake data
lvcreate -l 100%FREE -T vg0/thinpool

# Create the thin volumes with fake data
lvcreate -T -V 2T vg0/thinpool -n local-path-provisioner
lvcreate -T -V 2T vg0/thinpool -n docker-data

# Format the fake volumes
mkfs.xfs /dev/mapper/vg0-local--path--provisioner
mkfs.xfs /dev/mapper/vg0-docker--data
```

### Set eui64 on network interface

```bash
nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64
nmcli connection modify Wired\ connection\ 1 ipv6.ip6-privacy disabled
systemctl restart NetworkManager
```

### Install and Enable Cockpit

<https://cockpit-project.org/running>

```bash
dnf install cockpit
systemctl enable --now cockpit.socket
firewall-cmd --add-service=cockpit
firewall-cmd --add-service=cockpit --permanent
```

## Troubleshooting

### Cockpit Terminal Unusable or Weird Colors

Make sure you give canvas access to the browser (especially in librewolf)

### Chroot into a mounted disk

This lets you run grub2-mkconfig among other things.

```bash
# Mount root
mount /dev/mapper/vg0-root /mnt

# Mount proc, sys, and dev
mount -t proc /proc proc/
mount --rbind /sys sys/
mount --rbind /dev dev/

# Mount boot and efi
mount /dev/vdb2 /mnt/boot
mount /dev/vdb1 /mnt/boot/efi

chroot /mnt
```

### Resize Last Partition to Fill Available Space

```bash
parted /dev/vdb
# to resize /dev/vdb3 to fill 100% of the disk, for example
resizepart 3 100%
quit
# Resize the physical volume to match the partition
pvresize /dev/vdb3
```

### LUKS performance

```bash
cryptsetup benchmark
```

Should output something like:

```bash
#     Algorithm |       Key |      Encryption |      Decryption
        aes-cbc        128b      1409.1 MiB/s      3627.9 MiB/s
    serpent-cbc        128b       146.5 MiB/s       981.4 MiB/s
    twofish-cbc        128b       289.8 MiB/s       613.3 MiB/s
        aes-cbc        256b      1100.2 MiB/s      3448.2 MiB/s
    serpent-cbc        256b       150.3 MiB/s       982.1 MiB/s
    twofish-cbc        256b       294.3 MiB/s       590.8 MiB/s
        aes-xts        256b      4423.5 MiB/s      4561.2 MiB/s
    serpent-xts        256b       874.9 MiB/s       883.7 MiB/s
    twofish-xts        256b       557.8 MiB/s       559.4 MiB/s
        aes-xts        512b      4551.2 MiB/s      4669.6 MiB/s
    serpent-xts        512b       890.8 MiB/s       860.5 MiB/s
    twofish-xts        512b       557.5 MiB/s       564.2 MiB/s
```

Which will tell you how fast you can theoretically write/read to encrypted drives.

The default encryption used by most modern operating systems is AES-XTS.

You can see your system's cipher and key with `cryptsetup luksDump /dev/nvme0n1p1  | grep -i cipher`