organize folders and MVP wireguard mesh
This commit is contained in:
328
k3s/FedoraServer.md
Normal file
328
k3s/FedoraServer.md
Normal file
@@ -0,0 +1,328 @@
|
||||
# Fedora Server
|
||||
|
||||
Fedora server is an awesome container hosting OS. It has a lot built in, and setup is pretty
|
||||
quick.
|
||||
|
||||
- [Fedora Server](#fedora-server)
|
||||
- [Initialization](#initialization)
|
||||
- [Disable swap](#disable-swap)
|
||||
- [Network](#network)
|
||||
- [Enable ISCSI](#enable-iscsi)
|
||||
- [Disable Firewalld](#disable-firewalld)
|
||||
- [Set SELinux to Permissive](#set-selinux-to-permissive)
|
||||
- [Install K3S](#install-k3s)
|
||||
- [Database Backups](#database-backups)
|
||||
- [Expanding Root Partition](#expanding-root-partition)
|
||||
- [Arping IP Address](#arping-ip-address)
|
||||
- [Optional Steps](#optional-steps)
|
||||
- [Certbot for Cockpit](#certbot-for-cockpit)
|
||||
|
||||
## Initialization
|
||||
|
||||
1. `dnf install vim pwgen wireguard-tools`
|
||||
2. `hostnamectl hostname node1`
|
||||
3. Set a static IP through the web interface
|
||||
4. Allow wheel group members to sudo without password
|
||||
|
||||
## Disable swap
|
||||
|
||||
```bash
|
||||
swapoff -a
|
||||
dnf remove zram-generator-defaults
|
||||
```
|
||||
|
||||
mask <systemd-zram-setup@zram0.service>
|
||||
|
||||
## Network
|
||||
|
||||
1. Set MTU to 9000
|
||||
|
||||
If your network supports it, use 9000 as your mtu to allow more data per packet between
|
||||
servers. Note! For bridge interfaces you must set both the physical interface and bridge
|
||||
interface to 9000 - setting one but not the other can cause connectivity problems.
|
||||
|
||||
## Enable ISCSI
|
||||
|
||||
```bash
|
||||
# Install the following system packages
|
||||
dnf install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath
|
||||
|
||||
# Enable multipathing
|
||||
mpathconf --enable --with_multipathd y
|
||||
|
||||
# Ensure that iscsid and multipathd are running
|
||||
systemctl enable --now iscsid multipathd
|
||||
|
||||
# Test that discovery works
|
||||
iscsiadm -m discovery -t st -p democratic-csi-server.reeselink.com
|
||||
# Remove them - democratic-csi will populate this
|
||||
rm -rf /var/lib/iscsi/nodes/
|
||||
|
||||
# Start and enable iscsi
|
||||
systemctl enable --now iscsi
|
||||
```
|
||||
|
||||
## Disable Firewalld
|
||||
|
||||
<https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos--fedora>
|
||||
|
||||
Disable firewalld. You could add rules for each service but every time you open a port
|
||||
from a container you'd need to run a firewalld rule.
|
||||
|
||||
You can disable firewalld from the web interface.
|
||||
|
||||
## Set SELinux to Permissive
|
||||
|
||||
K3S is more than capable of running with SELinux set to enforcing. We won't be doing
|
||||
that, however. We'll set it to permissive and you can reenable it once you've added all
|
||||
the rules you need to keep your services running.
|
||||
|
||||
Set SELinux to permissive by editing `/etc/selinux/config`
|
||||
|
||||
SELINUX=permissive
|
||||
|
||||
## Install K3S
|
||||
|
||||
<https://docs.k3s.io/installation/requirements>
|
||||
|
||||
We're going to be tweaking some installation parameters so if you already have k3s
|
||||
installed you can either uninstall it or skip these steps.
|
||||
|
||||
This installation disables Traefik, local-storage, and Klipper. We'll replace them with
|
||||
our own components.
|
||||
|
||||
1. Generate a secure token for each node to use when connecting
|
||||
|
||||
umask 077
|
||||
echo -n $(pwgen 16 4) | sed 's/ /-/g' > token.txt
|
||||
|
||||
2. Create the cluster
|
||||
|
||||
export SECRET=$(cat token.txt)
|
||||
|
||||
curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - \
|
||||
"--cluster-init" \
|
||||
"--flannel-backend=wireguard-native" \
|
||||
"--disable" \
|
||||
"traefik" \
|
||||
"--disable" \
|
||||
"local-storage" \
|
||||
"--disable" \
|
||||
"servicelb" \
|
||||
"--disable" \
|
||||
"coredns" \
|
||||
"--cluster-dns" \
|
||||
"10.43.0.10" \
|
||||
"--tls-san" \
|
||||
"kube.reeselink.com" \
|
||||
"--tls-san" \
|
||||
"kube.reeseapps.com" \
|
||||
|
||||
3. Join each server node
|
||||
|
||||
export SECRET=$(cat token.txt)
|
||||
|
||||
curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - server \
|
||||
--server https://node1.reeselink.com:6443 \
|
||||
--flannel-backend=wireguard-native \
|
||||
"--disable" \
|
||||
"traefik" \
|
||||
"--disable" \
|
||||
"local-storage" \
|
||||
"--disable" \
|
||||
"coredns" \
|
||||
"--disable" \
|
||||
"servicelb" \
|
||||
"--cluster-dns" \
|
||||
"10.43.0.10" \
|
||||
"--tls-san" \
|
||||
"kube.reeselink.com" \
|
||||
|
||||
Now you can change the ownership of (and copy) the k3s.yaml file:
|
||||
|
||||
chown ducoterra /etc/rancher/k3s/k3s.yaml
|
||||
|
||||
scp /etc/rancher/k3s/k3s.yaml ~/.kube/config
|
||||
|
||||
Edit ~/.kube/config and change 127.0.0.1 to containers.reeselink.com
|
||||
|
||||
### Database Backups
|
||||
|
||||
<https://docs.k3s.io/cli/etcd-snapshot>
|
||||
|
||||
Note, you must backup `/var/lib/rancher/k3s/server/token`
|
||||
and use the contents as the token when restoring the backup as data is encrypted with that token.
|
||||
|
||||
## Expanding Root Partition
|
||||
|
||||
```bash
|
||||
dnf install cloud-utils-growpart
|
||||
growpart /dev/sda 3
|
||||
lvextend -l +100%FREE fedora
|
||||
xfs_growfs /dev/mapper/fedora-root
|
||||
```
|
||||
|
||||
## Arping IP Address
|
||||
|
||||
```bash
|
||||
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
|
||||
arping -I bridge0 -U -s 10.1.2.102 10.1.0.1
|
||||
```
|
||||
|
||||
## Optional Steps
|
||||
|
||||
### Certbot for Cockpit
|
||||
|
||||
During this process you'll pick one node to act as your manager for your other nodes.
|
||||
You'll only need to cert a single node and then it will connect via ssh over your local
|
||||
network to the other nodes.
|
||||
|
||||
Create an AWS user which will have route53 access. This is required for certbot's route53
|
||||
validation.
|
||||
|
||||
```bash
|
||||
aws iam create-user --user-name replicator
|
||||
```
|
||||
|
||||
You'll also need a policy which allows the user to modify the selected hosted zone:
|
||||
|
||||
(list with `aws route53 list-hosted-zones`)
|
||||
|
||||
```json
|
||||
{
|
||||
"Version": "2012-10-17",
|
||||
"Statement": [
|
||||
{
|
||||
"Effect": "Allow",
|
||||
"Action": [
|
||||
"route53:ListHostedZones",
|
||||
"route53:GetChange"
|
||||
],
|
||||
"Resource": [
|
||||
"*"
|
||||
]
|
||||
},
|
||||
{
|
||||
"Effect" : "Allow",
|
||||
"Action" : [
|
||||
"route53:ChangeResourceRecordSets"
|
||||
],
|
||||
"Resource" : [
|
||||
"arn:aws:route53:::hostedzone/Z012820733346FJ0U4FUF",
|
||||
"arn:aws:route53:::hostedzone/Z0092652G7L97DSINN18",
|
||||
"arn:aws:route53:::hostedzone/Z04612891U5Q2JRHUZ11T"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Attach the policy to the user:
|
||||
|
||||
```bash
|
||||
aws iam attach-user-policy \
|
||||
--user-name replicator \
|
||||
--policy-arn arn:aws:iam::892236928704:policy/certbot-route53-reeseapps
|
||||
```
|
||||
|
||||
Generate credentials:
|
||||
|
||||
```bash
|
||||
aws iam create-access-key --user-name replicator
|
||||
```
|
||||
|
||||
On the host machine:
|
||||
|
||||
```bash
|
||||
mkdir ~/.aws
|
||||
vim ~/.aws/config
|
||||
```
|
||||
|
||||
```conf
|
||||
[profile default]
|
||||
region=us-east-2
|
||||
```
|
||||
|
||||
```bash
|
||||
vim ~/.aws/credentials
|
||||
```
|
||||
|
||||
```conf
|
||||
[default]
|
||||
aws_access_key_id=<key>
|
||||
aws_secret_access_key=<key>
|
||||
```
|
||||
|
||||
Install the aws cli v2 on the manager node:
|
||||
|
||||
```bash
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip awscliv2.zip
|
||||
sudo ./aws/install
|
||||
```
|
||||
|
||||
Test your credentials with `aws route53 list-hosted-zones`. You should see as list of your
|
||||
hosted zones.
|
||||
|
||||
Now install certbot and acquire a cert using those credentials:
|
||||
|
||||
```bash
|
||||
sudo dnf install certbot python3-certbot-dns-route53
|
||||
sudo certbot certonly --dns-route53 -d containers.reeselink.com
|
||||
|
||||
sudo cp /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
|
||||
sudo cp /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
|
||||
```
|
||||
|
||||
Test the renewal process with:
|
||||
|
||||
```bash
|
||||
sudo certbot renew --cert-name containers.reeselink.com --dry-run
|
||||
```
|
||||
|
||||
Create a renewal script in /usr/lib/scripts/certbot-renew.sh
|
||||
|
||||
/usr/lib/scripts/certbot-renew.sh (chmod +x)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
|
||||
/usr/bin/certbot renew --cert-name containers.reeselink.com
|
||||
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
|
||||
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
|
||||
```
|
||||
|
||||
Now create a systemd oneshot service to run the script
|
||||
|
||||
/etc/systemd/system/certbot-renew.service
|
||||
|
||||
```conf
|
||||
[Unit]
|
||||
Description=Certbot Renewal
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/lib/scripts/certbot-renew.sh
|
||||
```
|
||||
|
||||
/etc/systemd/system/certbot-renew.timer
|
||||
|
||||
```conf
|
||||
[Unit]
|
||||
Description=Timer for Certbot Renewal
|
||||
|
||||
[Timer]
|
||||
OnBootSec=300
|
||||
OnUnitActiveSec=1w
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
```
|
||||
|
||||
Enable the service
|
||||
|
||||
```bash
|
||||
systemctl enable --now certbot-renew.timer
|
||||
```
|
||||
|
||||
Cockpit now has a valid TLS certificate that auto-renews!
|
||||
Reference in New Issue
Block a user