kube local storage migration

2024-07-11 00:56:51 -04:00
parent 887df21477
commit f2f51a923a
90 changed files with 614 additions and 1087 deletions
--- a/k3s/README.md
+++ b/k3s/README.md
@@ -16,30 +16,14 @@
  - [Cert Manager](#cert-manager)
  - [Test Minecraft Server](#test-minecraft-server)
  - [Automatic Updates](#automatic-updates)
-  - [Manual Updates](#manual-updates)
-  - [Create a Userspace](#create-a-userspace)
+  - [Database Backups](#database-backups)
    - [Quickstart](#quickstart)
-    - [Userspace](#userspace)
-      - [Namespace](#namespace)
-      - [Roles](#roles)
-      - [Rolebinding](#rolebinding)
-    - [Manual Steps](#manual-steps)
-      - [Create a kubernetes certsigner pod](#create-a-kubernetes-certsigner-pod)
-      - [Create the certsigner secret](#create-the-certsigner-secret)
-      - [Set up the certsigner pod](#set-up-the-certsigner-pod)
-      - [Generate a cert](#generate-a-cert)
-      - [Create a new Userspace](#create-a-new-userspace)
-      - [Sign the cert](#sign-the-cert)
-      - [Add to the config](#add-to-the-config)
-      - [Delete](#delete)
-    - [Signing a user cert - detailed notes](#signing-a-user-cert---detailed-notes)
  - [Help](#help)
    - [Troubleshooting](#troubleshooting)
      - [Deleting a stuck namespace](#deleting-a-stuck-namespace)
      - [Fixing a bad volume](#fixing-a-bad-volume)
      - [Mounting an ix-application volume from truenas](#mounting-an-ix-application-volume-from-truenas)
      - [Mounting a volume](#mounting-a-volume)
-  - [Database Backups](#database-backups)
  - [Uninstall](#uninstall)

 ## Guide
@@ -76,6 +60,7 @@ Set SELinux to permissive by editing `/etc/selinux/config`

 ```bash
 curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
+    "--cluster-init" \
    "--flannel-ipv6-masq" \
    "--disable" \
    "traefik" \
@@ -83,6 +68,8 @@ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
    "servicelb" \
    "--disable" \
    "coredns" \
+    "--disable" \
+    "local-storage" \
    "--tls-san" \
    "kube.reeselink.com" \
    "--cluster-cidr" \
@@ -111,6 +98,19 @@ scp kube:/etc/rancher/k3s/k3s.yaml ~/.kube/admin-kube-config
 3. `systemctl daemon-reload`
 4. `mount -a`

+<https://github.com/rancher/local-path-provisioner/tree/master/deploy/chart/local-path-provisioner>
+
+```bash
+# Download the updated template from github
+kubectl kustomize "github.com/rancher/local-path-provisioner/deploy?ref=v0.0.28" > local-path-provisioner/local-path-storage.yaml
+
+# Apply customizations (ssd/hdd storage, read write many support)
+kubectl kustomize local-path-provisioner | kubectl apply -f -
+
+# Create test pod
+kubectl apply -f k3s/tests/local-storage-test.yaml
+```
+
 ## Coredns

 1. Edit `coredns/values.yaml` to ensure the forward nameserver is correct.
@@ -270,8 +270,7 @@ kubectl delete -f k3s/tests/ingress-nginx-test.yaml
 ## Test Minecraft Server

 ```bash
-helm upgrade --install minecraft ./helm/minecraft -n minecraft --create-namespace
-helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespace
+helm upgrade --install minecraft ./minecraft -n minecraft --create-namespace
 ```

 ## Automatic Updates
@@ -279,28 +278,33 @@ helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespa
 <https://docs.k3s.io/upgrades/automated>

 ```bash
+kubectl create namespace system-upgrade
 kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
 kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
 kubectl apply -f k3s/upgrade-plan.yaml
+
+# Check plan
+kubectl get plan -n system-upgrade
 ```

-## Manual Updates
+## Database Backups

-<https://docs.k3s.io/upgrades/manual#manually-upgrade-k3s-using-the-binary>
+<https://docs.k3s.io/cli/etcd-snapshot>
+
+Note, you must backup `/var/lib/rancher/k3s/server/token`
+and use the contents as the toklisten when restoring the backup as data is encrypted with that token.
+
+Backups are saved to `/var/lib/rancher/k3s/server/db/snapshots/` by default.

 ```bash
-sudo su -
-wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s1/k3s
-systemctl stop k3s
-chmod +x k3s
-mv k3s /usr/local/bin/k3s
-systemctl start k3s
+k3s etcd-snapshot save
+k3s etcd-snapshot list
+
+k3s server \
+  --cluster-reset \
+  --cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/on-demand-kube-1720459685
 ```

-## Create a Userspace
-
-This creates a user, namespace, and permissions with a simple script.
-
 ### Quickstart

 ```bash
@@ -314,194 +318,6 @@ This creates a user, namespace, and permissions with a simple script.
 ./removeuserspace <server_fqdn> <user>
 ```

-### Userspace
-
-#### Namespace
-
-```yaml
-apiVersion: v1
-kind: Namespace
-metadata:
-  name: {{ .Release.Name }}
-```
-
-#### Roles
-
-```yaml
-kind: Role
-apiVersion: rbac.authorization.k8s.io/v1beta1
-metadata:
-  name: namespace-manager
-  namespace: {{ .Release.Name }}
-rules:
- apiGroups:
-    - ""
-    - extensions
-    - apps
-    - batch
-    - autoscaling
-    - networking.k8s.io
-    - traefik.containo.us
-    - rbac.authorization.k8s.io
-    - metrics.k8s.io
-  resources: 
-    - deployments
-    - replicasets
-    - pods
-    - pods/exec
-    - pods/log
-    - pods/attach
-    - daemonsets
-    - statefulsets
-    - replicationcontrollers
-    - horizontalpodautoscalers
-    - services
-    - ingresses
-    - persistentvolumeclaims
-    - jobs
-    - cronjobs
-    - secrets
-    - configmaps
-    - serviceaccounts
-    - rolebindings
-    - ingressroutes
-    - middlewares
-    - endpoints
-  verbs: 
-    - "*"
- apiGroups:
-    - ""
-    - metrics.k8s.io
-    - rbac.authorization.k8s.io
-  resources:
-    - resourcequotas
-    - roles
-  verbs:
-    - list
-```
-
-#### Rolebinding
-
-```yaml
-kind: RoleBinding
-apiVersion: rbac.authorization.k8s.io/v1beta1
-metadata:
-  namespace: {{ .Release.Name }}
-  name: namespace-manager
-subjects:
- kind: User
-  name: {{ .Release.Name }}
-  apiGroup: ""
-roleRef:
-  kind: ClusterRole
-  name: namespace-manager
-  apiGroup: ""
-```
-
-### Manual Steps
-
-#### Create a kubernetes certsigner pod
-
-This keeps the client-ca crt and key secret and allows the cert to be signed and stored on the pod
-
-#### Create the certsigner secret
-
-```bash
-kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key
-```
-
-#### Set up the certsigner pod
-
-```bash
-scp certsigner.yaml <server>:~/certsigner.yaml
-kubectl apply -f certsigner.yaml
-```
-
-#### Generate a cert
-
-```bash
-export USER=<user>
-docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl genrsa -out /$USER/$USER.key 2048
-docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl req -new -key /$USER/$USER.key -out /$USER/$USER.csr -subj "/CN=$USER/O=user"
-```
-
-#### Create a new Userspace
-
-```bash
-helm template $USER ./namespace | kubectl --context admin apply -f -
-```
-
-#### Sign the cert
-
-```bash
-export USER=<user>
-kubectl --context admin cp $(pwd)/users/$USER/$USER.csr certsigner:/certs/$USER.csr
-kubectl --context admin exec -it --context admin certsigner -- openssl x509 -in /certs/$USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -CAcreateserial -out /certs/$USER.crt -days 5000
-kubectl --context admin cp certsigner:/certs/$USER.crt $(pwd)/users/$USER/$USER.crt
-```
-
-#### Add to the config
-
-```bash
-kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
-kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
-```
-
-#### Delete
-
-```bash
-kubectl config delete-context $USER
-helm template $USER ./namespace | kubectl --context admin delete -f -
-```
-
-### Signing a user cert - detailed notes
-
-NOTE: ca.crt and ca.key are in /var/lib/rancher/k3s/server/tls/client-ca.*
-
-```bash
-# First we create the credentials
-# /CN=<username> - the user
-# /O=<group> - the group
-
-# Navigate to the user directory
-export USER=<username>
-cd $USER
-
-# Generate a private key
-openssl genrsa -out $USER.key 2048
-# Check the key
-# openssl pkey -in ca.key -noout -text
-# Generate and send me the CSR
-# The "user" group is my default group
-openssl req -new -key $USER.key -out $USER.csr -subj "/CN=$USER/O=user"
-
-# Check the CSR
-# openssl req -in $USER.csr -noout -text
-# If satisfactory, sign the CSR
-# Copy from /var/lib/rancher/k3s/server/tls/client-ca.crt and client-ca.key
-openssl x509 -req -in $USER.csr -CA ../client-ca.crt -CAkey ../client-ca.key -CAcreateserial -out $USER.crt -days 5000
-# Review the certificate
-# openssl x509 -in $USER.crt -text -noout
-
-# Send back the crt
-# cp $USER.crt $USER.key ../server-ca.crt ~/.kube/
-kubectl config set-credentials $USER --client-certificate=$USER.crt  --client-key=$USER.key
-kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
-
-# Now we create the namespace, rolebindings, and resource quotas
-# kubectl apply -f k8s/
-
-# Add the cluster
-# CA file can be found at https://3.14.3.100:6443/cacerts
- cluster:
-    certificate-authority: server-ca.crt
-    server: https://3.14.3.100:6443
-  name: mainframe
-
-# Test if everything worked
-kubectl --context=$USER-context get pods
-```
-
 ## Help

 ### Troubleshooting
@@ -541,13 +357,6 @@ mount -t xfs /dev/zvol/enc0/dcsi/apps/pvc-d5090258-cf20-4f2e-a5cf-330ac00d0049 /
 umount /mnt/dcsi_pvc
 ```

-## Database Backups
-
-<https://docs.k3s.io/cli/etcd-snapshot>
-
-Note, you must backup `/var/lib/rancher/k3s/server/token`
-and use the contents as the token when restoring the backup as data is encrypted with that token.
-
 ## Uninstall

 ```bash