services/homelab
1
1
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity
Files
7ccedb976824ecffd6a5aeb41148bd7e6d2831b1
homelab/active/software_k3s/k3s.md

10 KiB
Raw Blame History

K3S

Guide

  1. Configure Host
  2. Install CoreDNS for inter-container discovery
  3. Install Metal LB for load balancer IP address assignment
  4. install External DNS for laod balancer IP and ingress DNS records
  5. Install Nginx Ingress for http services
  6. Install Cert Manager for automatic Let's Encrypt certificates for Ingress nginx
  7. Install longhorn storage for automatic PVC creation and management
  8. Set up automatic database backups

Firewalld

firewall-cmd --permanent --zone=public --add-port=6443/tcp # apiserver
firewall-cmd --permanent --zone=trusted --add-source=10.42.0.0/16 # pods
firewall-cmd --permanent --zone=trusted --add-source=fd02:c91e:56f4::/56 # pods
firewall-cmd --permanent --zone=trusted --add-source=10.43.0.0/16 # services
firewall-cmd --permanent --zone=trusted --add-source=fd02:c91e:56f5::/112 # services

firewall-cmd --reload

Set SELinux to Permissive

Make sure to add --selinux to your install script.

Install K3S (Single Node)

Dual Stack IPv6 Support

curl -sfL https://get.k3s.io | sh -s - \
"--disable" \
"traefik" \
"--disable" \
"servicelb" \
"--tls-san" \
"k3s.reeselink.com" \
"--flannel-ipv6-masq" \
--kubelet-arg="node-ip=::" \
"--cluster-cidr" \
"10.42.0.0/16,fd02:c91e:56f4::/56" \
"--service-cidr" \
"10.43.0.0/16,fd02:c91e:56f5::/112" \
"--cluster-dns" \
"fd02:c91e:56f5::10" \
--selinux

Single Stack IPv4

curl -sfL https://get.k3s.io | sh -s - \
"--disable" \
"traefik" \
"--disable" \
"servicelb" \
"--tls-san" \
"k3s.reeselink.com" \
--selinux

Kube Credentials

On the operator

export KUBE_SERVER_ADDRESS="https://k3s.reeselink.com:6443"
# Copy the kube config down
ssh k3s cat /etc/rancher/k3s/k3s.yaml | \
yq -y ".clusters[0].cluster.server = \"${KUBE_SERVER_ADDRESS}\"" > \
~/.kube/admin-kube-config

Metal LB

VLAN Setup

I would remove firewalld to get this working. VLAN IPv6 traffic doesn't work for some reason and there aren't good docs yet. Your router firewall will suffice, just be sure to configure those rules correctly.

Before working with Metallb you'll need at least one available VLAN. On Unifi equipment this is accomplished by creating a new network. Don't assign it to anything.

On the linux machine you can use nmcli or cockpit to configure a new VLAN network interface. With cockpit:

  1. Add a new VLAN network
  2. The parent should be the physical adapter connected to your switch
  3. Set the VLAN ID to the VLAN number of your created unifi network
  4. Click create
  5. Click into the new network
  6. Turn off IPv4 and IPv6 DNS (it will overload the resolv.conf hosts limit)
  7. Turn on the network interface
  8. Attempt to ping the acquired address(es)

Installation

We'll be swapping K3S's default load balancer with Metal LB for more flexibility. ServiceLB was struggling to allocate IP addresses for load balanced services. MetallLB does make things a little more complicated- you'll need special annotations (see below) but it's otherwise a well-tested, stable load balancing service with features to grow into.

Metallb is pretty cool. It works via l2 advertisement or BGP. We won't be using BGP, so let's focus on l2.

When we connect our nodes to a network we give them an IP address range: ex. 192.168.122.20/24. This range represents all the available addresses the node could be assigned. Usually we assign a single "static" IP address for our node and direct traffic to it by port forwarding from our router. This is fine for single nodes - but what if we have a cluster of nodes and we don't want our service to disappear just because one node is down for maintenance?

This is where l2 advertising comes in. Metallb will assign a static IP address from a given pool to any arbitrary node - then advertise that node's mac address as the location for the IP. When that node goes down metallb simply advertises a new mac address for the same IP address, effectively moving the IP to another node. This isn't really "load balancing" but "failover". Fortunately, that's exactly what we're looking for.

helm repo add metallb https://metallb.github.io/metallb
helm repo update

# Install metallb
helm upgrade --install metallb \
--namespace kube-system \
metallb/metallb

MetalLB doesn't know what IP addresses are available for it to allocate so we'll have to provide it with a list. The metallb-addresspool.yaml has the configuration for our available pools. Note these should match the VLAN you created above.

# create the metallb allocation pool
kubectl apply -f active/kubernetes_metallb/addresspool.yaml

You'll need to annotate your service as follows if you want an external IP:

metadata:
  annotations:
    metallb.universe.tf/address-pool: "unifi-pool"
spec:
  ipFamilyPolicy: PreferDualStack
  ipFamilies:
  - IPv6
  - IPv4

Then test with

kubectl apply -f active/systemd_k3s/tests/metallb-test.yaml

External DNS

https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md

Credentials

  1. Generate credentials for the cluster
aws iam create-user --user-name "externaldns"
aws iam attach-user-policy --user-name "externaldns" --policy-arn arn:aws:iam::892236928704:policy/update-reeseapps
aws iam attach-user-policy --user-name "externaldns" --policy-arn arn:aws:iam::892236928704:policy/update-reeselink

GENERATED_ACCESS_KEY=$(aws iam create-access-key --user-name "externaldns")
ACCESS_KEY_ID=$(echo $GENERATED_ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')
SECRET_ACCESS_KEY=$(echo $GENERATED_ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')

cat <<-EOF > secrets/externaldns-credentials

[default]
aws_access_key_id = $ACCESS_KEY_ID
aws_secret_access_key = $SECRET_ACCESS_KEY
EOF

kubectl create secret generic external-dns \
  --namespace kube-system --from-file secrets/externaldns-credentials

helm repo add external-dns https://kubernetes-sigs.github.io/external-dns/
helm upgrade --install external-dns external-dns/external-dns \
--values active/kubernetes_external-dns/values.yaml \
--namespace kube-system

Annotation

metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: example.com

Nginx Ingress

Now we need an ingress solution (preferably with certs for https). We'll be using nginx since it's a little bit more configurable than traefik (though don't sell traefik short, it's really good. Just finnicky when you have use cases they haven't explicitly coded for).

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install \
    ingress-nginx \
    ingress-nginx/ingress-nginx \
    --values active/kubernetes_ingress-nginx/values.yaml \
    --namespace kube-system

Cert Manager

Install cert-manager

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
    cert-manager jetstack/cert-manager \
    --namespace kube-system \
    --set crds.enabled=true

Create the let's encrypt issuer (Route53 DNS)

export LE_ACCESS_KEY_ID=
export LE_SECRET_KEY=

cat <<EOF > secrets/cert-manager-secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: prod-route53-credentials-cert-manager
data:
  access-key-id: $(echo $LE_ACCESS_KEY_ID | base64)
  secret-access-key: $(echo $LE_SECRET_KEY | base64)
EOF

kubectl apply -f secrets/cert-manager-secret.yaml

cat <<EOF > secrets/route53-cluster-issuer.yaml
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: nginx@ducoterra.net
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - selector:
        dnsZones:
          - "reeseapps.com"
      dns01:
        route53:
          region: us-east-1
          hostedZoneID: Z012820733346FJ0U4FUF
          accessKeyID: ${LE_ACCESS_KEY_ID}
          secretAccessKeySecretRef:
            name: prod-route53-credentials-cert-manager
            key: secret-access-key
EOF

kubectl apply -f secrets/route53-cluster-issuer.yaml

You can test if your ingress is working with:

# Navigate to demo.reeseapps.com
kubectl apply -f active/infrastructure_k3s/tests/ingress-nginx-test.yaml

# Cleanup
kubectl delete -f active/infrastructure_k3s/tests/ingress-nginx-test.yaml

Test Minecraft Server

helm upgrade --install minecraft active/kubernetes_minecraft -n minecraft --create-namespace

Automatic Updates

https://docs.k3s.io/upgrades/automated

kubectl create namespace system-upgrade
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
kubectl apply -f active/infrastructure_k3s/upgrade-plan.yaml

# Check plan
kubectl get plan -n system-upgrade

Database Backups

https://docs.k3s.io/cli/etcd-snapshot

Note, you must backup /var/lib/rancher/k3s/server/token and use the contents as the toklisten when restoring the backup as data is encrypted with that token.

Backups are saved to /var/lib/rancher/k3s/server/db/snapshots/ by default.

k3s etcd-snapshot save
k3s etcd-snapshot list

k3s server \
  --cluster-reset \
  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/on-demand-kube-1720459685

Uninstall

/usr/local/bin/k3s-uninstall.sh
Reference in New Issue View Git Blame Copy Permalink