homelab/k3s/FedoraServer.md

# Fedora Server

Fedora server is an awesome container hosting OS. It has a lot built in, and setup is pretty
quick.

- [Fedora Server](#fedora-server)
  - [Initialization](#initialization)
  - [Disable swap](#disable-swap)
  - [Network](#network)
  - [Enable ISCSI](#enable-iscsi)
  - [Disable Firewalld](#disable-firewalld)
  - [Set SELinux to Permissive](#set-selinux-to-permissive)
  - [Install K3S](#install-k3s)
    - [Database Backups](#database-backups)
  - [Expanding Root Partition](#expanding-root-partition)
  - [Arping IP Address](#arping-ip-address)
  - [Optional Steps](#optional-steps)
    - [Certbot for Cockpit](#certbot-for-cockpit)

## Initialization

1. `dnf install vim pwgen wireguard-tools`
2. `hostnamectl hostname node1`
3. Set a static IP through the web interface
4. Allow wheel group members to sudo without password

## Disable swap

    ```bash
    swapoff -a
    dnf remove zram-generator-defaults
    ```

    mask <systemd-zram-setup@zram0.service>

## Network

1. Set MTU to 9000

If your network supports it, use 9000 as your mtu to allow more data per packet between
servers. Note! For bridge interfaces you must set both the physical interface and bridge
interface to 9000 - setting one but not the other can cause connectivity problems.

## Enable ISCSI

```bash
# Install the following system packages
dnf install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath

# Enable multipathing
mpathconf --enable --with_multipathd y

# Ensure that iscsid and multipathd are running
systemctl enable --now iscsid multipathd

# Test that discovery works
iscsiadm -m discovery -t st -p democratic-csi-server.reeselink.com
# Remove them - democratic-csi will populate this
rm -rf /var/lib/iscsi/nodes/

# Start and enable iscsi
systemctl enable --now iscsi
```

## Disable Firewalld

<https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos--fedora>

Disable firewalld. You could add rules for each service but every time you open a port
from a container you'd need to run a firewalld rule.

You can disable firewalld from the web interface.

## Set SELinux to Permissive

K3S is more than capable of running with SELinux set to enforcing. We won't be doing
that, however. We'll set it to permissive and you can reenable it once you've added all
the rules you need to keep your services running.

Set SELinux to permissive by editing `/etc/selinux/config`

    SELINUX=permissive

## Install K3S

<https://docs.k3s.io/installation/requirements>

We're going to be tweaking some installation parameters so if you already have k3s
installed you can either uninstall it or skip these steps.

This installation disables Traefik, local-storage, and Klipper. We'll replace them with
our own components.

1. Generate a secure token for each node to use when connecting

    umask 077
    echo -n $(pwgen 16 4) | sed 's/ /-/g' > token.txt

2. Create the cluster

    export SECRET=$(cat token.txt)

    curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - \
        "--cluster-init" \
        "--flannel-backend=wireguard-native" \
        "--disable" \
        "traefik" \
        "--disable" \
        "local-storage" \
         "--disable" \
        "servicelb" \
        "--disable" \
        "coredns" \
        "--cluster-dns" \
        "10.43.0.10" \
        "--tls-san" \
        "kube.reeselink.com" \
        "--tls-san" \
        "kube.reeseapps.com" \

3. Join each server node

    export SECRET=$(cat token.txt)

    curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - server \
        --server https://node1.reeselink.com:6443 \
        --flannel-backend=wireguard-native \
        "--disable" \
        "traefik" \
        "--disable" \
        "local-storage" \
        "--disable" \
        "coredns" \
        "--disable" \
        "servicelb" \
        "--cluster-dns" \
        "10.43.0.10" \
        "--tls-san" \
        "kube.reeselink.com" \

Now you can change the ownership of (and copy) the k3s.yaml file:

    chown ducoterra /etc/rancher/k3s/k3s.yaml

    scp /etc/rancher/k3s/k3s.yaml ~/.kube/config

Edit ~/.kube/config and change 127.0.0.1 to containers.reeselink.com

### Database Backups

<https://docs.k3s.io/cli/etcd-snapshot>

Note, you must backup `/var/lib/rancher/k3s/server/token`
and use the contents as the token when restoring the backup as data is encrypted with that token.

## Expanding Root Partition

```bash
dnf install cloud-utils-growpart
growpart /dev/sda 3
lvextend -l +100%FREE fedora
xfs_growfs /dev/mapper/fedora-root
```

## Arping IP Address

```bash
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
arping -I bridge0 -U -s 10.1.2.102 10.1.0.1
```

## Optional Steps

### Certbot for Cockpit

During this process you'll pick one node to act as your manager for your other nodes.
You'll only need to cert a single node and then it will connect via ssh over your local
network to the other nodes.

Create an AWS user which will have route53 access. This is required for certbot's route53
validation.

```bash
aws iam create-user --user-name replicator
```

You'll also need a policy which allows the user to modify the selected hosted zone:

(list with `aws route53 list-hosted-zones`)

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "route53:ListHostedZones",
                "route53:GetChange"
            ],
            "Resource": [
                "*"
            ]
        },
        {
            "Effect" : "Allow",
            "Action" : [
                "route53:ChangeResourceRecordSets"
            ],
            "Resource" : [
                "arn:aws:route53:::hostedzone/Z012820733346FJ0U4FUF",
                "arn:aws:route53:::hostedzone/Z0092652G7L97DSINN18",
                "arn:aws:route53:::hostedzone/Z04612891U5Q2JRHUZ11T"
            ]
        }
    ]
}
```

Attach the policy to the user:

```bash
aws iam attach-user-policy \
    --user-name replicator \
    --policy-arn arn:aws:iam::892236928704:policy/certbot-route53-reeseapps
```

Generate credentials:

```bash
aws iam create-access-key --user-name replicator
```

On the host machine:

```bash
mkdir ~/.aws
vim ~/.aws/config
```

```conf
[profile default]
region=us-east-2
```

```bash
vim ~/.aws/credentials
```

```conf
[default]
aws_access_key_id=<key>
aws_secret_access_key=<key>
```

Install the aws cli v2 on the manager node:

```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```

Test your credentials with `aws route53 list-hosted-zones`. You should see as list of your
hosted zones.

Now install certbot and acquire a cert using those credentials:

```bash
sudo dnf install certbot python3-certbot-dns-route53
sudo certbot certonly --dns-route53 -d containers.reeselink.com

sudo cp /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
sudo cp /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
```

Test the renewal process with:

```bash
sudo certbot renew --cert-name containers.reeselink.com --dry-run
```

Create a renewal script in /usr/lib/scripts/certbot-renew.sh

/usr/lib/scripts/certbot-renew.sh (chmod +x)

```bash
#!/bin/bash

/usr/bin/certbot renew --cert-name containers.reeselink.com
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
```

Now create a systemd oneshot service to run the script

/etc/systemd/system/certbot-renew.service

```conf
[Unit]
Description=Certbot Renewal

[Service]
Type=oneshot
ExecStart=/usr/lib/scripts/certbot-renew.sh
```

/etc/systemd/system/certbot-renew.timer

```conf
[Unit]
Description=Timer for Certbot Renewal

[Timer]
OnBootSec=300
OnUnitActiveSec=1w

[Install]
WantedBy=multi-user.target
```

Enable the service

```bash
systemctl enable --now certbot-renew.timer
```

Cockpit now has a valid TLS certificate that auto-renews!