services/homelab
1
1
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

organize folders and MVP wireguard mesh

Browse Source
This commit is contained in:
Reese Wells
2024-06-22 13:07:39 -04:00
parent 45c58aeba7
commit 44e484f22e
42 changed files with 243 additions and 708 deletions
Download Patch File Download Diff File Expand all files Collapse all files

328
k3s/FedoraServer.md Normal file
View File

@@ -0,0 +1,328 @@
# Fedora Server
Fedora server is an awesome container hosting OS. It has a lot built in, and setup is pretty
quick.
- [Fedora Server](#fedora-server)
- [Initialization](#initialization)
- [Disable swap](#disable-swap)
- [Network](#network)
- [Enable ISCSI](#enable-iscsi)
- [Disable Firewalld](#disable-firewalld)
- [Set SELinux to Permissive](#set-selinux-to-permissive)
- [Install K3S](#install-k3s)
- [Database Backups](#database-backups)
- [Expanding Root Partition](#expanding-root-partition)
- [Arping IP Address](#arping-ip-address)
- [Optional Steps](#optional-steps)
- [Certbot for Cockpit](#certbot-for-cockpit)
## Initialization
1. `dnf install vim pwgen wireguard-tools`
2. `hostnamectl hostname node1`
3. Set a static IP through the web interface
4. Allow wheel group members to sudo without password
## Disable swap
```bash
swapoff -a
dnf remove zram-generator-defaults
```
mask <systemd-zram-setup@zram0.service>
## Network
1. Set MTU to 9000
If your network supports it, use 9000 as your mtu to allow more data per packet between
servers. Note! For bridge interfaces you must set both the physical interface and bridge
interface to 9000 - setting one but not the other can cause connectivity problems.
## Enable ISCSI
```bash
# Install the following system packages
dnf install -y lsscsi iscsi-initiator-utils sg3_utils device-mapper-multipath
# Enable multipathing
mpathconf --enable --with_multipathd y
# Ensure that iscsid and multipathd are running
systemctl enable --now iscsid multipathd
# Test that discovery works
iscsiadm -m discovery -t st -p democratic-csi-server.reeselink.com
# Remove them - democratic-csi will populate this
rm -rf /var/lib/iscsi/nodes/
# Start and enable iscsi
systemctl enable --now iscsi
```
## Disable Firewalld
<https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos--fedora>
Disable firewalld. You could add rules for each service but every time you open a port
from a container you'd need to run a firewalld rule.
You can disable firewalld from the web interface.
## Set SELinux to Permissive
K3S is more than capable of running with SELinux set to enforcing. We won't be doing
that, however. We'll set it to permissive and you can reenable it once you've added all
the rules you need to keep your services running.
Set SELinux to permissive by editing `/etc/selinux/config`
SELINUX=permissive
## Install K3S
<https://docs.k3s.io/installation/requirements>
We're going to be tweaking some installation parameters so if you already have k3s
installed you can either uninstall it or skip these steps.
This installation disables Traefik, local-storage, and Klipper. We'll replace them with
our own components.
1. Generate a secure token for each node to use when connecting
umask 077
echo -n $(pwgen 16 4) | sed 's/ /-/g' > token.txt
2. Create the cluster
export SECRET=$(cat token.txt)
curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - \
"--cluster-init" \
"--flannel-backend=wireguard-native" \
"--disable" \
"traefik" \
"--disable" \
"local-storage" \
"--disable" \
"servicelb" \
"--disable" \
"coredns" \
"--cluster-dns" \
"10.43.0.10" \
"--tls-san" \
"kube.reeselink.com" \
"--tls-san" \
"kube.reeseapps.com" \
3. Join each server node
export SECRET=$(cat token.txt)
curl -sfL https://get.k3s.io | K3S_TOKEN=$SECRET sh -s - server \
--server https://node1.reeselink.com:6443 \
--flannel-backend=wireguard-native \
"--disable" \
"traefik" \
"--disable" \
"local-storage" \
"--disable" \
"coredns" \
"--disable" \
"servicelb" \
"--cluster-dns" \
"10.43.0.10" \
"--tls-san" \
"kube.reeselink.com" \
Now you can change the ownership of (and copy) the k3s.yaml file:
chown ducoterra /etc/rancher/k3s/k3s.yaml
scp /etc/rancher/k3s/k3s.yaml ~/.kube/config
Edit ~/.kube/config and change 127.0.0.1 to containers.reeselink.com
### Database Backups
<https://docs.k3s.io/cli/etcd-snapshot>
Note, you must backup `/var/lib/rancher/k3s/server/token`
and use the contents as the token when restoring the backup as data is encrypted with that token.
## Expanding Root Partition
```bash
dnf install cloud-utils-growpart
growpart /dev/sda 3
lvextend -l +100%FREE fedora
xfs_growfs /dev/mapper/fedora-root
```
## Arping IP Address
```bash
echo 1 > /proc/sys/net/ipv4/ip_nonlocal_bind
arping -I bridge0 -U -s 10.1.2.102 10.1.0.1
```
## Optional Steps
### Certbot for Cockpit
During this process you'll pick one node to act as your manager for your other nodes.
You'll only need to cert a single node and then it will connect via ssh over your local
network to the other nodes.
Create an AWS user which will have route53 access. This is required for certbot's route53
validation.
```bash
aws iam create-user --user-name replicator
```
You'll also need a policy which allows the user to modify the selected hosted zone:
(list with `aws route53 list-hosted-zones`)
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"route53:ListHostedZones",
"route53:GetChange"
],
"Resource": [
"*"
]
},
{
"Effect" : "Allow",
"Action" : [
"route53:ChangeResourceRecordSets"
],
"Resource" : [
"arn:aws:route53:::hostedzone/Z012820733346FJ0U4FUF",
"arn:aws:route53:::hostedzone/Z0092652G7L97DSINN18",
"arn:aws:route53:::hostedzone/Z04612891U5Q2JRHUZ11T"
]
}
]
}
```
Attach the policy to the user:
```bash
aws iam attach-user-policy \
--user-name replicator \
--policy-arn arn:aws:iam::892236928704:policy/certbot-route53-reeseapps
```
Generate credentials:
```bash
aws iam create-access-key --user-name replicator
```
On the host machine:
```bash
mkdir ~/.aws
vim ~/.aws/config
```
```conf
[profile default]
region=us-east-2
```
```bash
vim ~/.aws/credentials
```
```conf
[default]
aws_access_key_id=<key>
aws_secret_access_key=<key>
```
Install the aws cli v2 on the manager node:
```bash
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install
```
Test your credentials with `aws route53 list-hosted-zones`. You should see as list of your
hosted zones.
Now install certbot and acquire a cert using those credentials:
```bash
sudo dnf install certbot python3-certbot-dns-route53
sudo certbot certonly --dns-route53 -d containers.reeselink.com
sudo cp /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
sudo cp /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
```
Test the renewal process with:
```bash
sudo certbot renew --cert-name containers.reeselink.com --dry-run
```
Create a renewal script in /usr/lib/scripts/certbot-renew.sh
/usr/lib/scripts/certbot-renew.sh (chmod +x)
```bash
#!/bin/bash
/usr/bin/certbot renew --cert-name containers.reeselink.com
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/fullchain.pem /etc/cockpit/ws-certs.d/50-letsencrypt.cert
/usr/bin/cp -f /etc/letsencrypt/live/containers.reeselink.com/privkey.pem /etc/cockpit/ws-certs.d/50-letsencrypt.key
```
Now create a systemd oneshot service to run the script
/etc/systemd/system/certbot-renew.service
```conf
[Unit]
Description=Certbot Renewal
[Service]
Type=oneshot
ExecStart=/usr/lib/scripts/certbot-renew.sh
```
/etc/systemd/system/certbot-renew.timer
```conf
[Unit]
Description=Timer for Certbot Renewal
[Timer]
OnBootSec=300
OnUnitActiveSec=1w
[Install]
WantedBy=multi-user.target
```
Enable the service
```bash
systemctl enable --now certbot-renew.timer
```
Cockpit now has a valid TLS certificate that auto-renews!

30
k3s/cluster-readonly.yaml Executable file
View File

@@ -0,0 +1,30 @@
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-readonly
rules:
- apiGroups:
- ""
- rbac.authorization.k8s.io
- storage.k8s.io
- networking.k8s.io
- traefik.containo.us
resources:
- storageclasses
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-readonly
subjects:
- kind: Group
name: user
apiGroup: ""
roleRef:
kind: ClusterRole
name: cluster-readonly
apiGroup: ""

16
k3s/clusterrole.yaml Executable file
View File

@@ -0,0 +1,16 @@
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: user-readonly
rules:
- apiGroups:
- rbac.authorization.k8s.io
- storage.k8s.io
- networking.k8s.io
resources:
- clusterroles
- storageclasses
- ingressclasses
verbs:
- list
- watch

22
k3s/orange_yellow.md Normal file
View File

@@ -0,0 +1,22 @@
# Orange and Yellow Servers
## Services
1. Both servers run all quadlets in `podman/`
2. Both servers run the same nginx config in `nginx/`
## VMs
1. Orange runs unifi-external
2. Yellow runs nextcloud-aio
## Pihole
- Orange is ad-blocking
- Yellow is caching
## Load Balancing
- Orange handles all traffic from outside the network
- Yellow handles all internal traffic
- Both are capable of handling all traffic, port forwarding rule determines which is up

38
k3s/scripts/removeuser.sh Executable file
View File

@@ -0,0 +1,38 @@
#!/bin/bash
# Use
# ./removeuserspace <ssh_address> <server_fqdn (for kubectl)> <user>
export SERVER=$1
export FQDN=$2
export USER=$3
export CERT_DIR=$HOME/.kube/$FQDN/users/$USER
export CA_CERT_DIR=$HOME/.kube/$FQDN
export SERVER_USER_DIR="~/.kube/users/$USER"
export SERVER_NAME=$(echo "$FQDN" | sed 's/\./-/g')
export SERVER_USER="$USER-$SERVER_NAME"
export KUBECONFIG="$HOME/.kube/$USER-config"
echo "Checking if project namespace exists"
exists=$(ssh $SERVER "kubectl get namespace --output=jsonpath=\"{.items[?(@.metadata.name=='$USER')].metadata.name}\"")
if [ -z $exists ]; then
echo "Namespace not found, nothing to delete"
exit 1
else
echo "Namespace exists, deleting"
fi
echo "Deleting user namespace"
ssh $SERVER "kubectl delete -f $SERVER_USER_DIR/namespace.yaml"
echo "Deleting remote cert dir"
ssh $SERVER "rm -rf $SERVER_USER_DIR"
echo "Deleting local cert dir"
rm -rf $CERT_DIR
echo "Removing from kubeconfig"
rm $KUBECONFIG

12
k3s/scripts/setup.sh Executable file
View File

@@ -0,0 +1,12 @@
#!/bin/bash
# Use
# ./setup.sh <server_fqdn>
export SERVER=$1
ssh -t $SERVER sudo kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key
scp certsigner.yaml $SERVER:~/certsigner.yaml
ssh $SERVER kubectl apply -f certsigner.yaml
scp clusterrole.yaml $SERVER:~/clusterrole.yaml
ssh $SERVER kubectl apply -f clusterrole.yaml

140
k3s/scripts/upsertuser.sh Executable file
View File

@@ -0,0 +1,140 @@
#!/bin/bash
# Use
# ./upsert.sh <ssh_address> <server_fqdn (for kubectl)> <user>
# Note, do not specify https:// or :port for the fqdn, just give the domain
# Port is expected to be 6443. You can change this later in the generated conf
# ./upsert.sh node1 containers.reeseapps.com testuser
# ./upsert.sh 192.168.1.10 mydomain.ddns.net admin
export SERVER=$1
export FQDN=$2
export KUBE_USER=$3
export CERT_DIR=$HOME/.kube/$FQDN/users/$KUBE_USER
export CA_CERT_DIR=$HOME/.kube/$FQDN
export SERVER_USER_DIR="~/.kube/users/$KUBE_USER"
export SERVER_NAME=$(echo "$FQDN" | sed 's/\./-/g')
export SERVER_USER="$KUBE_USER-$SERVER_NAME"
export KUBECONFIG="$HOME/.kube/$KUBE_USER-config"
if [ -z $KUBE_USER ]; then
echo "No arguments supplied! Format is ./upsert.sh <SERVER_FQDN> <USER>"
exit 1
fi
if [ -z $SERVER ]; then
echo "No server supplied for user $KUBE_USER"
exit 1
fi
if [ $KUBE_USER = "admin" ]; then
echo "Creating admin user for server $SERVER"
fi
echo "Creating cert dir"
mkdir -p $CERT_DIR
if [ $? -ne 0 ]; then
echo "Couldn't create cert dir at $CERT_DIR"
exit 1
fi
echo "Generating openssl cert"
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl genrsa -out /$KUBE_USER/$KUBE_USER.key 2048
if [ $KUBE_USER = "admin" ]; then
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl req -new -key /$KUBE_USER/$KUBE_USER.key -out /$KUBE_USER/$KUBE_USER.csr -subj "/CN=$KUBE_USER/O=system:masters"
else
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl req -new -key /$KUBE_USER/$KUBE_USER.key -out /$KUBE_USER/$KUBE_USER.csr -subj "/CN=$KUBE_USER/O=user"
fi
# /CN=admin/O=manager
if [ $? -ne 0 ]; then
echo "Couldn't create cert with Podman. Are you sure it's running?"
exit 1
fi
echo "Creating namespace dir on server"
ssh $SERVER "mkdir -p $SERVER_USER_DIR"
echo "Copying client csr to server cert dir"
scp $CERT_DIR/$KUBE_USER.csr $SERVER:$SERVER_USER_DIR/$KUBE_USER.csr
if [ $? -ne 0 ]; then
echo "Failed to copy client csr to server cert dir"
exit 1
fi
echo "Getting cert signing pod"
export CERT_POD=$(ssh $SERVER "kubectl get pod -n kube-system --selector=app=certsigner --output=jsonpath={.items..metadata.name}")
if [ -z $CERT_POD ]; then
echo "Installing certsigner"
helm template certsigner ./certsigner | ssh $SERVER "sudo -t -E kubectl apply -f -"
fi
while [ -z $CERT_POD ]; do
echo "Getting cert signing pod"
export CERT_POD=$(ssh $SERVER "kubectl get pod -n kube-system --selector=app=certsigner --output=jsonpath={.items..metadata.name}")
sleep 2
done
if [ $? -ne 0 ]; then
echo "Failed to install certsigner."
exit 1
fi
echo "Signing cert with pod $CERT_POD"
ssh $SERVER "kubectl -n kube-system cp $SERVER_USER_DIR/$KUBE_USER.csr $CERT_POD:/certs/$KUBE_USER.csr"
ssh $SERVER "kubectl -n kube-system exec $CERT_POD -- openssl x509 -in /certs/$KUBE_USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -set_serial $(python -c 'import random; print(random.randint(1000000000, 9999999999))') -out /certs/$KUBE_USER.crt -days 5000"
ssh $SERVER "kubectl -n kube-system cp $CERT_POD:/certs/$KUBE_USER.crt ~/.kube/users/$KUBE_USER/$KUBE_USER.crt"
echo "retrieving signed cert"
scp $SERVER:$SERVER_USER_DIR/$KUBE_USER.crt $CERT_DIR/$KUBE_USER.crt
echo "retrieving server ca"
wget --no-check-certificate https://$FQDN:6443/cacerts -O $CA_CERT_DIR/server-ca.pem
echo "creating $FQDN-$KUBE_USER context"
kubectl config set-context $FQDN-$KUBE_USER
echo "setting $FQDN-$KUBE_USER as current context"
kubectl config set current-context $FQDN-$KUBE_USER
echo "adding server to config with new context $FQDN-$KUBE_USER"
kubectl config set-cluster $FQDN --server=https://$FQDN:6443 --certificate-authority=$CA_CERT_DIR/server-ca.pem
kubectl config set contexts.$(kubectl config current-context).cluster $FQDN
echo "adding user to config file"
kubectl config set-credentials $SERVER_USER --client-certificate=$CERT_DIR/$KUBE_USER.crt --client-key=$CERT_DIR/$KUBE_USER.key
echo "setting user context"
kubectl config set contexts.$(kubectl config current-context).user $SERVER_USER
if [ $KUBE_USER = "admin" ]; then
echo "Admin user created, skipping namespace"
echo "export KUBECONFIG=$KUBECONFIG"
exit 0
fi
echo "Templating namespace with helm and copying to server"
helm template $KUBE_USER --set user=$KUBE_USER ./helm/namespace | ssh $SERVER "cat - > $SERVER_USER_DIR/namespace.yaml"
if [ $? -ne 0 ]; then
echo "Failed to template namespace. Is helm installed?"
exit 1
fi
echo "Creating namespace from template"
ssh $SERVER "kubectl apply -f $SERVER_USER_DIR/namespace.yaml"
echo "Setting namespace context"
kubectl config set contexts.$(kubectl config current-context).namespace $KUBE_USER
if [ $? -ne 0 ]; then
echo "Failed to create namespace"
exit 1
fi
echo "export KUBECONFIG=$KUBECONFIG"

42
k3s/upgrade-plan.yaml Normal file
View File

@@ -0,0 +1,42 @@
# Server plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: server-plan
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: In
values:
- "true"
serviceAccountName: system-upgrade
upgrade:
image: rancher/k3s-upgrade
channel: https://update.k3s.io/v1-release/channels/latest
---
# Agent plan
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
name: agent-plan
namespace: system-upgrade
spec:
concurrency: 1
cordon: true
nodeSelector:
matchExpressions:
- key: node-role.kubernetes.io/control-plane
operator: DoesNotExist
prepare:
args:
- prepare
- server-plan
image: rancher/k3s-upgrade
serviceAccountName: system-upgrade
upgrade:
image: rancher/k3s-upgrade
channel: https://update.k3s.io/v1-release/channels/latest