K3S is more than capable of running with SELinux set to enforcing. We won't be doing that, however. We'll set it to permissive and you can reenable it once you've added all the rules you need to keep your services running.

Set SELinux to permissive by editing /etc/selinux/config

SELINUX=permissive

Install K3S (Single Node)

curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
    "--flannel-ipv6-masq" \
    "--disable" \
    "traefik" \
    "--disable" \
    "servicelb" \
    "--disable" \
    "coredns" \
    "--tls-san" \
    "kube.reeselink.com" \
    "--cluster-cidr" \
    "10.42.0.0/16,fd02:c91e:56f4::/56" \
    "--service-cidr" \
    "10.43.0.0/16,fd02:c91e:56f5::/112" \
    "--cluster-dns" \
    "fd02:c91e:56f5::10"

Kube Credentials

On the operator

# Copy the kube config down
scp kube:/etc/rancher/k3s/k3s.yaml ~/.kube/admin-kube-config

# Edit the server to match the remote address.

Storage

mkdir /var/lib/rancher/k3s/storage
Edit fstab to mount your drive to /var/lib/rancher/k3s/storage
systemctl daemon-reload
mount -a

Coredns

Edit coredns/values.yaml to ensure the forward nameserver is correct.

# Install CoreDNS
helm upgrade --install \
    --namespace=kube-system \
    --values coredns/values.yaml \
    coredns coredns/coredns

# Test DNS works
kubectl run -it --rm \
    --restart=Never \
    --image=infoblox/dnstools:latest \
    dnstools

Metal LB

We'll be swapping K3S's default load balancer with Metal LB for more flexibility. ServiceLB was struggling to allocate IP addresses for load balanced services. MetallLB does make things a little more complicated- you'll need special annotations (see below) but it's otherwise a well-tested, stable load balancing service with features to grow into.

Metallb is pretty cool. It works via l2 advertisement or BGP. We won't be using BGP, so let's focus on l2.

When we connect our nodes to a network we give them an IP address range: ex. 192.168.122.20/24. This range represents all the available addresses the node could be assigned. Usually we assign a single "static" IP address for our node and direct traffic to it by port forwarding from our router. This is fine for single nodes - but what if we have a cluster of nodes and we don't want our service to disappear just because one node is down for maintenance?

This is where l2 advertising comes in. Metallb will assign a static IP address from a given pool to any arbitrary node - then advertise that node's mac address as the location for the IP. When that node goes down metallb simply advertises a new mac address for the same IP address, effectively moving the IP to another node. This isn't really "load balancing" but "failover". Fortunately, that's exactly what we're looking for.

helm repo add metallb https://metallb.github.io/metallb
helm repo update
helm upgrade --install metallb \
    --namespace kube-system \
    metallb/metallb

MetalLB doesn't know what IP addresses are available for it to allocate so we'll have to provide it with a list. The metallb-addresspool.yaml has one IP address (we'll get to IP address sharing in a second) which is an unassigned IP address not allocated to any of our nodes. Note if you have many public IPs which all point to the same router or virtual network you can list them. We're only going to use one because we want to port forward from our router.

# create the metallb allocation pool
kubectl apply -f metallb/addresspool.yaml

You'll need to annotate your service as follows if you want an external IP:

metadata:
  annotations:
    metallb.universe.tf/address-pool: "external"
    # or
    metallb.universe.tf/address-pool: "internal"
spec:
  ipFamilyPolicy: SingleStack
  ipFamilies:
  - IPv6

External DNS

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md

Credentials

Generate credentials for the cluster

aws iam create-user --user-name "externaldns"
aws iam attach-user-policy --user-name "externaldns" --policy-arn arn:aws:iam::892236928704:policy/update-reeseapps

SECRET_ACCESS_KEY=$(aws iam create-access-key --user-name "externaldns")
ACCESS_KEY_ID=$(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')

cat <<-EOF > secrets/externaldns-credentials

[default]
aws_access_key_id = $(echo $ACCESS_KEY_ID)
aws_secret_access_key = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')
EOF

kubectl create secret generic external-dns \
  --namespace kube-system --from-file secrets/externaldns-credentials

kubectl apply -f external-dns/sa.yaml

kubectl apply -f external-dns/deploy.yaml

Annotation

metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.com

Nginx Ingress

Now we need an ingress solution (preferably with certs for https). We'll be using nginx since it's a little bit more configurable than traefik (though don't sell traefik short, it's really good. Just finnicky when you have use cases they haven't explicitly coded for).

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install \
    ingress-nginx \
    ingress-nginx/ingress-nginx \
    --values ingress-nginx/values.yaml \
    --namespace kube-system

Cert Manager

Install cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
    cert-manager jetstack/cert-manager \
    --namespace kube-system \
    --set crds.enabled=true

Create the let's encrypt issuer (Route53 DNS)

kubectl apply -f certmanager/letsencrypt-issuer.yaml

You can test if your ingress is working with:

# Navigate to demo.reeseapps.com
kubectl apply -f k3s/tests/ingress-nginx-test.yaml

# Cleanup
kubectl delete -f k3s/tests/ingress-nginx-test.yaml

Test Minecraft Server

helm upgrade --install minecraft ./helm/minecraft -n minecraft --create-namespace
helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespace

Automatic Updates

https://docs.k3s.io/upgrades/automated

kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
kubectl apply -f k3s/upgrade-plan.yaml

Manual Updates

https://docs.k3s.io/upgrades/manual#manually-upgrade-k3s-using-the-binary

sudo su -
wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s1/k3s
systemctl stop k3s
chmod +x k3s
mv k3s /usr/local/bin/k3s
systemctl start k3s

Create a Userspace

This creates a user, namespace, and permissions with a simple script.

Quickstart

# Create certsigner pod for all other operations
./setup.sh <server_fqdn>

# Create a user, use "admin" to create an admin user
./upsertuser.sh <ssh_address> <server_fqdn (for kubectl)> <user>

# Remove a user, their namespace, and their access
./removeuserspace <server_fqdn> <user>

Userspace

Namespace

apiVersion: v1
kind: Namespace
metadata:
  name: {{ .Release.Name }}

Roles

kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: namespace-manager
  namespace: {{ .Release.Name }}
rules:
- apiGroups:
    - ""
    - extensions
    - apps
    - batch
    - autoscaling
    - networking.k8s.io
    - traefik.containo.us
    - rbac.authorization.k8s.io
    - metrics.k8s.io
  resources: 
    - deployments
    - replicasets
    - pods
    - pods/exec
    - pods/log
    - pods/attach
    - daemonsets
    - statefulsets
    - replicationcontrollers
    - horizontalpodautoscalers
    - services
    - ingresses
    - persistentvolumeclaims
    - jobs
    - cronjobs
    - secrets
    - configmaps
    - serviceaccounts
    - rolebindings
    - ingressroutes
    - middlewares
    - endpoints
  verbs: 
    - "*"
- apiGroups:
    - ""
    - metrics.k8s.io
    - rbac.authorization.k8s.io
  resources:
    - resourcequotas
    - roles
  verbs:
    - list

Rolebinding

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  namespace: {{ .Release.Name }}
  name: namespace-manager
subjects:
- kind: User
  name: {{ .Release.Name }}
  apiGroup: ""
roleRef:
  kind: ClusterRole
  name: namespace-manager
  apiGroup: ""

Manual Steps

Create a kubernetes certsigner pod

This keeps the client-ca crt and key secret and allows the cert to be signed and stored on the pod

Create the certsigner secret

kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key

Set up the certsigner pod

scp certsigner.yaml <server>:~/certsigner.yaml
kubectl apply -f certsigner.yaml

Generate a cert

export USER=<user>
docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl genrsa -out /$USER/$USER.key 2048
docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl req -new -key /$USER/$USER.key -out /$USER/$USER.csr -subj "/CN=$USER/O=user"

Create a new Userspace

helm template $USER ./namespace | kubectl --context admin apply -f -

Sign the cert

export USER=<user>
kubectl --context admin cp $(pwd)/users/$USER/$USER.csr certsigner:/certs/$USER.csr
kubectl --context admin exec -it --context admin certsigner -- openssl x509 -in /certs/$USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -CAcreateserial -out /certs/$USER.crt -days 5000
kubectl --context admin cp certsigner:/certs/$USER.crt $(pwd)/users/$USER/$USER.crt

Add to the config

kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER

Delete

kubectl config delete-context $USER
helm template $USER ./namespace | kubectl --context admin delete -f -

Signing a user cert - detailed notes

NOTE: ca.crt and ca.key are in /var/lib/rancher/k3s/server/tls/client-ca.*

# First we create the credentials
# /CN=<username> - the user
# /O=<group> - the group

# Navigate to the user directory
export USER=<username>
cd $USER

# Generate a private key
openssl genrsa -out $USER.key 2048
# Check the key
# openssl pkey -in ca.key -noout -text
# Generate and send me the CSR
# The "user" group is my default group
openssl req -new -key $USER.key -out $USER.csr -subj "/CN=$USER/O=user"

# Check the CSR
# openssl req -in $USER.csr -noout -text
# If satisfactory, sign the CSR
# Copy from /var/lib/rancher/k3s/server/tls/client-ca.crt and client-ca.key
openssl x509 -req -in $USER.csr -CA ../client-ca.crt -CAkey ../client-ca.key -CAcreateserial -out $USER.crt -days 5000
# Review the certificate
# openssl x509 -in $USER.crt -text -noout

# Send back the crt
# cp $USER.crt $USER.key ../server-ca.crt ~/.kube/
kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER

# Now we create the namespace, rolebindings, and resource quotas
# kubectl apply -f k8s/

# Add the cluster
# CA file can be found at https://3.14.3.100:6443/cacerts
- cluster:
    certificate-authority: server-ca.crt
    server: https://3.14.3.100:6443
  name: mainframe

# Test if everything worked
kubectl --context=$USER-context get pods

Help

Troubleshooting

Deleting a stuck namespace

NAMESPACE=nginx
kubectl proxy &
kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize

Fixing a bad volume

xfs_repair -L /dev/sdg

Mounting an ix-application volume from truenas

# set the mountpoint
zfs set mountpoint=/ix_pvc enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b

#"unset" the mountpoint (back to legacy)
zfs set mountpoint=legacy enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b

Mounting a volume

# mount
mount -t xfs /dev/zvol/enc0/dcsi/apps/pvc-d5090258-cf20-4f2e-a5cf-330ac00d0049 /mnt/dcsi_pvc

# unmount
umount /mnt/dcsi_pvc

Database Backups

https://docs.k3s.io/cli/etcd-snapshot

Note, you must backup /var/lib/rancher/k3s/server/token and use the contents as the token when restoring the backup as data is encrypted with that token.

Uninstall

/usr/local/bin/k3s-uninstall.sh