# Fedora Server Fedora server is an awesome container hosting OS. It has a lot built in, and setup is pretty quick. - [Fedora Server](#fedora-server) - [Initialization](#initialization) - [Disable swap](#disable-swap) - [Network](#network) - [Enable ISCSI](#enable-iscsi) - [Disable Firewalld](#disable-firewalld) - [Set SELinux to Permissive](#set-selinux-to-permissive) - [Install K3S](#install-k3s) - [Database Backups](#database-backups) - [Expanding Root Partition](#expanding-root-partition) - [Arping IP Address](#arping-ip-address) - [Optional Steps](#optional-steps) - [Certbot for Cockpit](#certbot-for-cockpit) ## Initialization 1. `dnf install vim pwgen wireguard-tools` 2. `hostnamectl hostname node1` 3. Set a static IP through the web interface 4. Allow wheel group members to sudo without password ## Disable swap ```bash swapoff -a dnf remove zram-generator-defaults ``` mask ## Network 1. Set MTU to 9000 If your network supports it, use 9000 as your mtu to allow more data per packet between servers. Note! For bridge interfaces you must set both the physical interface and bridge interface to 9000 - setting one but not the other can cause connectivity problems. ## Enable ISCSI ```bash # Install the following system packages dnf install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath # Enable multipathing mpathconf --enable --with_multipathd y # Ensure that iscsid and multipathd are running systemctl enable --now iscsid multipathd # Test that discovery works iscsiadm -m discovery -t st -p democratic-csi-server.reeselink.com # Remove them - democratic-csi will populate this rm -rf /var/lib/iscsi/nodes/ # Start and enable iscsi systemctl enable --now iscsi ``` ## Disable Firewalld Disable firewalld. You could add rules for each service but every time you open a port from a container you'd need to run a firewalld rule. You can disable firewalld from the web interface. ## Set SELinux to Permissive K3S is more than capable of running with SELinux set to enforcing. We won't be doing that, however. We'll set it to permissive and you can reenable it once you've added all the rules you need to keep your services running. Set SELinux to permissive by editing `/etc/selinux/config` SELINUX=permissive ## Install K3S We're going to be tweaking some installation parameters so if you already have k3s installed you can either uninstall it or skip these steps. This installation disables Traefik, local-storage, and Klipper. We'll replace them with our own components. 1. Generate a secure token for each node to use when connecting umask 077 echo -n $(pwgen 16 4) | sed 's/ /-/g' > token.txt 2. Create the cluster export SECRET=$(cat token.txt) curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - \ "--cluster-init" \ "--flannel-backend=wireguard-native" \ "--disable" \ "traefik" \ "--disable" \ "local-storage" \ "--disable" \ "servicelb" \ "--disable" \ "coredns" \ "--cluster-dns" \ "10.43.0.10" \ "--tls-san" \ "kube.reeselink.com" \ "--tls-san" \ "kube.reeseapps.com" \ 3. Join each server node export SECRET=$(cat token.txt) curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - server \ --server https://node1.reeselink.com:6443 \ --flannel-backend=wireguard-native \ "--disable" \ "traefik" \ "--disable" \ "local-storage" \ "--disable" \ "coredns" \ "--disable" \ "servicelb" \ "--cluster-dns" \ "10.43.0.10" \ "--tls-san" \ "kube.reeselink.com" \ Now you can change the ownership of (and copy) the k3s.yaml file: chown ducoterra /etc/rancher/k3s/k3s.yaml scp /etc/rancher/k3s/k3s.yaml ~/.kube/config Edit ~/.kube/config and change 127.0.0.1 to containers.reeselink.com ### Database Backups Note, you must backup `/var/lib/rancher/k3s/server/token` and use the contents as the token when restoring the backup as data is encrypted with that token. ## Expanding Root Partition ```bash dnf install cloud-utils-growpart growpart /dev/sda 3 lvextend -l +100%FREE fedora xfs_growfs /dev/mapper/fedora-root ``` ## Arping IP Address ```bash echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind arping -I bridge0 -U -s 10.1.2.102 10.1.0.1 ``` ## Optional Steps ### Certbot for Cockpit During this process you'll pick one node to act as your manager for your other nodes. You'll only need to cert a single node and then it will connect via ssh over your local network to the other nodes. Create an AWS user which will have route53 access. This is required for certbot's route53 validation. ```bash aws iam create-user --user-name replicator ``` You'll also need a policy which allows the user to modify the selected hosted zone: (list with `aws route53 list-hosted-zones`) ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "route53:ListHostedZones", "route53:GetChange" ], "Resource": [ "*" ] }, { "Effect" : "Allow", "Action" : [ "route53:ChangeResourceRecordSets" ], "Resource" : [ "arn:aws:route53:::hostedzone/Z012820733346FJ0U4FUF", "arn:aws:route53:::hostedzone/Z0092652G7L97DSINN18", "arn:aws:route53:::hostedzone/Z04612891U5Q2JRHUZ11T" ] } ] } ``` Attach the policy to the user: ```bash aws iam attach-user-policy \ --user-name replicator \ --policy-arn arn:aws:iam::892236928704:policy/certbot-route53-reeseapps ``` Generate credentials: ```bash aws iam create-access-key --user-name replicator ``` On the host machine: ```bash mkdir ~/.aws vim ~/.aws/config ``` ```conf [profile default] region=us-east-2 ``` ```bash vim ~/.aws/credentials ``` ```conf [default] aws_access_key_id= aws_secret_access_key= ``` Install the aws cli v2 on the manager node: ```bash curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" unzip awscliv2.zip sudo ./aws/install ``` Test your credentials with `aws route53 list-hosted-zones`. You should see as list of your hosted zones. Now install certbot and acquire a cert using those credentials: ```bash sudo dnf install certbot python3-certbot-dns-route53 sudo certbot certonly --dns-route53 -d containers.reeselink.com sudo cp /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert sudo cp /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key ``` Test the renewal process with: ```bash sudo certbot renew --cert-name containers.reeselink.com --dry-run ``` Create a renewal script in /usr/lib/scripts/certbot-renew.sh /usr/lib/scripts/certbot-renew.sh (chmod +x) ```bash #!/bin/bash /usr/bin/certbot renew --cert-name containers.reeselink.com /usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert /usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key ``` Now create a systemd oneshot service to run the script /etc/systemd/system/certbot-renew.service ```conf [Unit] Description=Certbot Renewal [Service] Type=oneshot ExecStart=/usr/lib/scripts/certbot-renew.sh ``` /etc/systemd/system/certbot-renew.timer ```conf [Unit] Description=Timer for Certbot Renewal [Timer] OnBootSec=300 OnUnitActiveSec=1w [Install] WantedBy=multi-user.target ``` Enable the service ```bash systemctl enable --now certbot-renew.timer ``` Cockpit now has a valid TLS certificate that auto-renews!