kube transfer to single-node host

2024-07-08 10:39:56 -04:00
parent d1afa569cc
commit 887df21477
69 changed files with 1675 additions and 2009 deletions
--- a/k3s/README.md
+++ b/k3s/README.md
@@ -0,0 +1,555 @@
+# K3S
+
+- [K3S](#k3s)
+  - [Guide](#guide)
+  - [Disable Firewalld](#disable-firewalld)
+  - [Set SELinux to Permissive](#set-selinux-to-permissive)
+  - [Install K3S (Single Node)](#install-k3s-single-node)
+  - [Kube Credentials](#kube-credentials)
+  - [Storage](#storage)
+  - [Coredns](#coredns)
+  - [Metal LB](#metal-lb)
+  - [External DNS](#external-dns)
+    - [Credentials](#credentials)
+    - [Annotation](#annotation)
+  - [Nginx Ingress](#nginx-ingress)
+  - [Cert Manager](#cert-manager)
+  - [Test Minecraft Server](#test-minecraft-server)
+  - [Automatic Updates](#automatic-updates)
+  - [Manual Updates](#manual-updates)
+  - [Create a Userspace](#create-a-userspace)
+    - [Quickstart](#quickstart)
+    - [Userspace](#userspace)
+      - [Namespace](#namespace)
+      - [Roles](#roles)
+      - [Rolebinding](#rolebinding)
+    - [Manual Steps](#manual-steps)
+      - [Create a kubernetes certsigner pod](#create-a-kubernetes-certsigner-pod)
+      - [Create the certsigner secret](#create-the-certsigner-secret)
+      - [Set up the certsigner pod](#set-up-the-certsigner-pod)
+      - [Generate a cert](#generate-a-cert)
+      - [Create a new Userspace](#create-a-new-userspace)
+      - [Sign the cert](#sign-the-cert)
+      - [Add to the config](#add-to-the-config)
+      - [Delete](#delete)
+    - [Signing a user cert - detailed notes](#signing-a-user-cert---detailed-notes)
+  - [Help](#help)
+    - [Troubleshooting](#troubleshooting)
+      - [Deleting a stuck namespace](#deleting-a-stuck-namespace)
+      - [Fixing a bad volume](#fixing-a-bad-volume)
+      - [Mounting an ix-application volume from truenas](#mounting-an-ix-application-volume-from-truenas)
+      - [Mounting a volume](#mounting-a-volume)
+  - [Database Backups](#database-backups)
+  - [Uninstall](#uninstall)
+
+## Guide
+
+1. Configure Host
+2. Install CoreDNS for inter-container discovery
+3. Install Metal LB for load balancer IP address assignment
+4. install External DNS for laod balancer IP and ingress DNS records
+5. Install Nginx Ingress for http services
+6. Install Cert Manager for automatic Let's Encrypt certificates for Ingress nginx
+7. Install longhorn storage for automatic PVC creation and management
+8. Set up automatic database backups
+
+## Disable Firewalld
+
+<https://docs.k3s.io/advanced#red-hat-enterprise-linux--centos--fedora>
+
+Disable firewalld. You could add rules for each service but every time you open a port
+from a container you'd need to run a firewalld rule.
+
+You can disable firewalld from the web interface.
+
+## Set SELinux to Permissive
+
+K3S is more than capable of running with SELinux set to enforcing. We won't be doing
+that, however. We'll set it to permissive and you can reenable it once you've added all
+the rules you need to keep your services running.
+
+Set SELinux to permissive by editing `/etc/selinux/config`
+
+    SELINUX=permissive
+
+## Install K3S (Single Node)
+
+```bash
+curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
+    "--flannel-ipv6-masq" \
+    "--disable" \
+    "traefik" \
+    "--disable" \
+    "servicelb" \
+    "--disable" \
+    "coredns" \
+    "--tls-san" \
+    "kube.reeselink.com" \
+    "--cluster-cidr" \
+    "10.42.0.0/16,fd02:c91e:56f4::/56" \
+    "--service-cidr" \
+    "10.43.0.0/16,fd02:c91e:56f5::/112" \
+    "--cluster-dns" \
+    "fd02:c91e:56f5::10"
+```
+
+## Kube Credentials
+
+On the operator
+
+```bash
+# Copy the kube config down
+scp kube:/etc/rancher/k3s/k3s.yaml ~/.kube/admin-kube-config
+
+# Edit the server to match the remote address.
+```
+
+## Storage
+
+1. `mkdir /var/lib/rancher/k3s/storage`
+2. Edit fstab to mount your drive to `/var/lib/rancher/k3s/storage`
+3. `systemctl daemon-reload`
+4. `mount -a`
+
+## Coredns
+
+1. Edit `coredns/values.yaml` to ensure the forward nameserver is correct.
+
+```bash
+# Install CoreDNS
+helm upgrade --install \
+    --namespace=kube-system \
+    --values coredns/values.yaml \
+    coredns coredns/coredns
+
+# Test DNS works
+kubectl run -it --rm \
+    --restart=Never \
+    --image=infoblox/dnstools:latest \
+    dnstools
+```
+
+## Metal LB
+
+We'll be swapping K3S's default load balancer with Metal LB for more flexibility. ServiceLB was
+struggling to allocate IP addresses for load balanced services. MetallLB does make things a little
+more complicated- you'll need special annotations (see below) but it's otherwise a well-tested,
+stable load balancing service with features to grow into.
+
+Metallb is pretty cool. It works via l2 advertisement or BGP. We won't be using BGP, so let's
+focus on l2.
+
+When we connect our nodes to a network we give them an IP address range: ex. `192.168.122.20/24`.
+This range represents all the available addresses the node could be assigned. Usually we assign
+a single "static" IP address for our node and direct traffic to it by port forwarding from our
+router. This is fine for single nodes - but what if we have a cluster of nodes and we don't want
+our service to disappear just because one node is down for maintenance?
+
+This is where l2 advertising comes in. Metallb will assign a static IP address from a given
+pool to any arbitrary node - then advertise that node's mac address as the location for the
+IP. When that node goes down metallb simply advertises a new mac address for the same IP
+address, effectively moving the IP to another node. This isn't really "load balancing" but
+"failover". Fortunately, that's exactly what we're looking for.
+
+```bash
+helm repo add metallb https://metallb.github.io/metallb
+helm repo update
+helm upgrade --install metallb \
+    --namespace kube-system \
+    metallb/metallb
+```
+
+MetalLB doesn't know what IP addresses are available for it to allocate so we'll have
+to provide it with a list. The `metallb-addresspool.yaml` has one IP address (we'll get to
+IP address sharing in a second) which is an unassigned IP address not allocated to any of our
+nodes. Note if you have many public IPs which all point to the same router or virtual network
+you can list them. We're only going to use one because we want to port forward from our router.
+
+```bash
+# create the metallb allocation pool
+kubectl apply -f metallb/addresspool.yaml
+```
+
+You'll need to annotate your service as follows if you want an external IP:
+
+```yaml
+metadata:
+  annotations:
+    metallb.universe.tf/address-pool: "external"
+    # or
+    metallb.universe.tf/address-pool: "internal"
+spec:
+  ipFamilyPolicy: SingleStack
+  ipFamilies:
+  - IPv6
+```
+
+## External DNS
+
+<https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/aws.md>
+
+### Credentials
+
+1. Generate credentials for the cluster
+
+```bash
+aws iam create-user --user-name "externaldns"
+aws iam attach-user-policy --user-name "externaldns" --policy-arn arn:aws:iam::892236928704:policy/update-reeseapps
+
+SECRET_ACCESS_KEY=$(aws iam create-access-key --user-name "externaldns")
+ACCESS_KEY_ID=$(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.AccessKeyId')
+
+cat <<-EOF > secrets/externaldns-credentials
+
+[default]
+aws_access_key_id = $(echo $ACCESS_KEY_ID)
+aws_secret_access_key = $(echo $SECRET_ACCESS_KEY | jq -r '.AccessKey.SecretAccessKey')
+EOF
+
+kubectl create secret generic external-dns \
+  --namespace kube-system --from-file secrets/externaldns-credentials
+
+kubectl apply -f external-dns/sa.yaml
+
+kubectl apply -f external-dns/deploy.yaml
+```
+
+### Annotation
+
+```yaml
+metadata:
+  annotations:
+    external-dns.alpha.kubernetes.io/hostname: example.com
+```
+
+## Nginx Ingress
+
+Now we need an ingress solution (preferably with certs for https). We'll be using nginx since
+it's a little bit more configurable than traefik (though don't sell traefik short, it's really
+good. Just finnicky when you have use cases they haven't explicitly coded for).
+
+```bash
+helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
+helm repo update
+helm upgrade --install \
+    ingress-nginx \
+    ingress-nginx/ingress-nginx \
+    --values ingress-nginx/values.yaml \
+    --namespace kube-system
+```
+
+## Cert Manager
+
+Install cert-manager
+
+```bash
+helm repo add jetstack https://charts.jetstack.io
+helm repo update
+helm upgrade --install \
+    cert-manager jetstack/cert-manager \
+    --namespace kube-system \
+    --set crds.enabled=true
+```
+
+Create the let's encrypt issuer (Route53 DNS)
+
+```bash
+kubectl apply -f certmanager/letsencrypt-issuer.yaml
+```
+
+You can test if your ingress is working with:
+
+```bash
+# Navigate to demo.reeseapps.com
+kubectl apply -f k3s/tests/ingress-nginx-test.yaml
+
+# Cleanup
+kubectl delete -f k3s/tests/ingress-nginx-test.yaml
+```
+
+## Test Minecraft Server
+
+```bash
+helm upgrade --install minecraft ./helm/minecraft -n minecraft --create-namespace
+helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespace
+```
+
+## Automatic Updates
+
+<https://docs.k3s.io/upgrades/automated>
+
+```bash
+kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
+kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
+kubectl apply -f k3s/upgrade-plan.yaml
+```
+
+## Manual Updates
+
+<https://docs.k3s.io/upgrades/manual#manually-upgrade-k3s-using-the-binary>
+
+```bash
+sudo su -
+wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s1/k3s
+systemctl stop k3s
+chmod +x k3s
+mv k3s /usr/local/bin/k3s
+systemctl start k3s
+```
+
+## Create a Userspace
+
+This creates a user, namespace, and permissions with a simple script.
+
+### Quickstart
+
+```bash
+# Create certsigner pod for all other operations
+./setup.sh <server_fqdn>
+
+# Create a user, use "admin" to create an admin user
+./upsertuser.sh <ssh_address> <server_fqdn (for kubectl)> <user>
+
+# Remove a user, their namespace, and their access
+./removeuserspace <server_fqdn> <user>
+```
+
+### Userspace
+
+#### Namespace
+
+```yaml
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: {{ .Release.Name }}
+```
+
+#### Roles
+
+```yaml
+kind: Role
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  name: namespace-manager
+  namespace: {{ .Release.Name }}
+rules:
+- apiGroups:
+    - ""
+    - extensions
+    - apps
+    - batch
+    - autoscaling
+    - networking.k8s.io
+    - traefik.containo.us
+    - rbac.authorization.k8s.io
+    - metrics.k8s.io
+  resources: 
+    - deployments
+    - replicasets
+    - pods
+    - pods/exec
+    - pods/log
+    - pods/attach
+    - daemonsets
+    - statefulsets
+    - replicationcontrollers
+    - horizontalpodautoscalers
+    - services
+    - ingresses
+    - persistentvolumeclaims
+    - jobs
+    - cronjobs
+    - secrets
+    - configmaps
+    - serviceaccounts
+    - rolebindings
+    - ingressroutes
+    - middlewares
+    - endpoints
+  verbs: 
+    - "*"
+- apiGroups:
+    - ""
+    - metrics.k8s.io
+    - rbac.authorization.k8s.io
+  resources:
+    - resourcequotas
+    - roles
+  verbs:
+    - list
+```
+
+#### Rolebinding
+
+```yaml
+kind: RoleBinding
+apiVersion: rbac.authorization.k8s.io/v1beta1
+metadata:
+  namespace: {{ .Release.Name }}
+  name: namespace-manager
+subjects:
+- kind: User
+  name: {{ .Release.Name }}
+  apiGroup: ""
+roleRef:
+  kind: ClusterRole
+  name: namespace-manager
+  apiGroup: ""
+```
+
+### Manual Steps
+
+#### Create a kubernetes certsigner pod
+
+This keeps the client-ca crt and key secret and allows the cert to be signed and stored on the pod
+
+#### Create the certsigner secret
+
+```bash
+kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key
+```
+
+#### Set up the certsigner pod
+
+```bash
+scp certsigner.yaml <server>:~/certsigner.yaml
+kubectl apply -f certsigner.yaml
+```
+
+#### Generate a cert
+
+```bash
+export USER=<user>
+docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl genrsa -out /$USER/$USER.key 2048
+docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl req -new -key /$USER/$USER.key -out /$USER/$USER.csr -subj "/CN=$USER/O=user"
+```
+
+#### Create a new Userspace
+
+```bash
+helm template $USER ./namespace | kubectl --context admin apply -f -
+```
+
+#### Sign the cert
+
+```bash
+export USER=<user>
+kubectl --context admin cp $(pwd)/users/$USER/$USER.csr certsigner:/certs/$USER.csr
+kubectl --context admin exec -it --context admin certsigner -- openssl x509 -in /certs/$USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -CAcreateserial -out /certs/$USER.crt -days 5000
+kubectl --context admin cp certsigner:/certs/$USER.crt $(pwd)/users/$USER/$USER.crt
+```
+
+#### Add to the config
+
+```bash
+kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
+kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
+```
+
+#### Delete
+
+```bash
+kubectl config delete-context $USER
+helm template $USER ./namespace | kubectl --context admin delete -f -
+```
+
+### Signing a user cert - detailed notes
+
+NOTE: ca.crt and ca.key are in /var/lib/rancher/k3s/server/tls/client-ca.*
+
+```bash
+# First we create the credentials
+# /CN=<username> - the user
+# /O=<group> - the group
+
+# Navigate to the user directory
+export USER=<username>
+cd $USER
+
+# Generate a private key
+openssl genrsa -out $USER.key 2048
+# Check the key
+# openssl pkey -in ca.key -noout -text
+# Generate and send me the CSR
+# The "user" group is my default group
+openssl req -new -key $USER.key -out $USER.csr -subj "/CN=$USER/O=user"
+
+# Check the CSR
+# openssl req -in $USER.csr -noout -text
+# If satisfactory, sign the CSR
+# Copy from /var/lib/rancher/k3s/server/tls/client-ca.crt and client-ca.key
+openssl x509 -req -in $USER.csr -CA ../client-ca.crt -CAkey ../client-ca.key -CAcreateserial -out $USER.crt -days 5000
+# Review the certificate
+# openssl x509 -in $USER.crt -text -noout
+
+# Send back the crt
+# cp $USER.crt $USER.key ../server-ca.crt ~/.kube/
+kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
+kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
+
+# Now we create the namespace, rolebindings, and resource quotas
+# kubectl apply -f k8s/
+
+# Add the cluster
+# CA file can be found at https://3.14.3.100:6443/cacerts
+- cluster:
+    certificate-authority: server-ca.crt
+    server: https://3.14.3.100:6443
+  name: mainframe
+
+# Test if everything worked
+kubectl --context=$USER-context get pods
+```
+
+## Help
+
+### Troubleshooting
+
+#### Deleting a stuck namespace
+
+```bash
+NAMESPACE=nginx
+kubectl proxy &
+kubectl get namespace $NAMESPACE -o json |jq '.spec = {"finalizers":[]}' >temp.json
+curl -k -H "Content-Type: application/json" -X PUT --data-binary @temp.json 127.0.0.1:8001/api/v1/namespaces/$NAMESPACE/finalize
+```
+
+#### Fixing a bad volume
+
+```bash
+xfs_repair -L /dev/sdg
+```
+
+#### Mounting an ix-application volume from truenas
+
+```bash
+# set the mountpoint
+zfs set mountpoint=/ix_pvc enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b
+
+#"unset" the mountpoint (back to legacy)
+zfs set mountpoint=legacy enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b
+```
+
+#### Mounting a volume
+
+```bash
+# mount
+mount -t xfs /dev/zvol/enc0/dcsi/apps/pvc-d5090258-cf20-4f2e-a5cf-330ac00d0049 /mnt/dcsi_pvc
+
+# unmount
+umount /mnt/dcsi_pvc
+```
+
+## Database Backups
+
+<https://docs.k3s.io/cli/etcd-snapshot>
+
+Note, you must backup `/var/lib/rancher/k3s/server/token`
+and use the contents as the token when restoring the backup as data is encrypted with that token.
+
+## Uninstall
+
+```bash
+/usr/local/bin/k3s-uninstall.sh
+```