services/homelab
1
1
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

kube local storage migration

Browse Source
This commit is contained in:
Reese Wells
2024-07-11 00:56:51 -04:00
parent 887df21477
commit f2f51a923a
90 changed files with 614 additions and 1087 deletions
Download Patch File Download Diff File Expand all files Collapse all files

261
k3s/README.md
View File

@@ -16,30 +16,14 @@
- [Cert Manager](#cert-manager)
- [Test Minecraft Server](#test-minecraft-server)
- [Automatic Updates](#automatic-updates)
- [Manual Updates](#manual-updates)
- [Create a Userspace](#create-a-userspace)
- [Database Backups](#database-backups)
- [Quickstart](#quickstart)
- [Userspace](#userspace)
- [Namespace](#namespace)
- [Roles](#roles)
- [Rolebinding](#rolebinding)
- [Manual Steps](#manual-steps)
- [Create a kubernetes certsigner pod](#create-a-kubernetes-certsigner-pod)
- [Create the certsigner secret](#create-the-certsigner-secret)
- [Set up the certsigner pod](#set-up-the-certsigner-pod)
- [Generate a cert](#generate-a-cert)
- [Create a new Userspace](#create-a-new-userspace)
- [Sign the cert](#sign-the-cert)
- [Add to the config](#add-to-the-config)
- [Delete](#delete)
- [Signing a user cert - detailed notes](#signing-a-user-cert---detailed-notes)
- [Help](#help)
- [Troubleshooting](#troubleshooting)
- [Deleting a stuck namespace](#deleting-a-stuck-namespace)
- [Fixing a bad volume](#fixing-a-bad-volume)
- [Mounting an ix-application volume from truenas](#mounting-an-ix-application-volume-from-truenas)
- [Mounting a volume](#mounting-a-volume)
- [Database Backups](#database-backups)
- [Uninstall](#uninstall)
## Guide
@@ -76,6 +60,7 @@ Set SELinux to permissive by editing `/etc/selinux/config`
```bash
curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
"--cluster-init" \
"--flannel-ipv6-masq" \
"--disable" \
"traefik" \
@@ -83,6 +68,8 @@ curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION=v1.30.2+k3s2 sh -s - \
"servicelb" \
"--disable" \
"coredns" \
"--disable" \
"local-storage" \
"--tls-san" \
"kube.reeselink.com" \
"--cluster-cidr" \
@@ -111,6 +98,19 @@ scp kube:/etc/rancher/k3s/k3s.yaml ~/.kube/admin-kube-config
3. `systemctl daemon-reload`
4. `mount -a`
<https://github.com/rancher/local-path-provisioner/tree/master/deploy/chart/local-path-provisioner>
```bash
# Download the updated template from github
kubectl kustomize "github.com/rancher/local-path-provisioner/deploy?ref=v0.0.28" > local-path-provisioner/local-path-storage.yaml
# Apply customizations (ssd/hdd storage, read write many support)
kubectl kustomize local-path-provisioner | kubectl apply -f -
# Create test pod
kubectl apply -f k3s/tests/local-storage-test.yaml
```
## Coredns
1. Edit `coredns/values.yaml` to ensure the forward nameserver is correct.
@@ -270,8 +270,7 @@ kubectl delete -f k3s/tests/ingress-nginx-test.yaml
## Test Minecraft Server
```bash
helm upgrade --install minecraft ./helm/minecraft -n minecraft --create-namespace
helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespace
helm upgrade --install minecraft ./minecraft -n minecraft --create-namespace
```
## Automatic Updates
@@ -279,28 +278,33 @@ helm upgrade --install minecraft1 ./helm/minecraft -n minecraft --create-namespa
<https://docs.k3s.io/upgrades/automated>
```bash
kubectl create namespace system-upgrade
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml
kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/crd.yaml
kubectl apply -f k3s/upgrade-plan.yaml
# Check plan
kubectl get plan -n system-upgrade
```
## Manual Updates
## Database Backups
<https://docs.k3s.io/upgrades/manual#manually-upgrade-k3s-using-the-binary>
<https://docs.k3s.io/cli/etcd-snapshot>
Note, you must backup `/var/lib/rancher/k3s/server/token`
and use the contents as the toklisten when restoring the backup as data is encrypted with that token.
Backups are saved to `/var/lib/rancher/k3s/server/db/snapshots/` by default.
```bash
sudo su -
wget https://github.com/k3s-io/k3s/releases/download/v1.28.3%2Bk3s1/k3s
systemctl stop k3s
chmod +x k3s
mv k3s /usr/local/bin/k3s
systemctl start k3s
k3s etcd-snapshot save
k3s etcd-snapshot list
k3s server \
--cluster-reset \
--cluster-reset-restore-path=/var/lib/rancher/k3s/server/db/snapshots/on-demand-kube-1720459685
```
## Create a Userspace
This creates a user, namespace, and permissions with a simple script.
### Quickstart
```bash
@@ -314,194 +318,6 @@ This creates a user, namespace, and permissions with a simple script.
./removeuserspace <server_fqdn> <user>
```
### Userspace
#### Namespace
```yaml
apiVersion: v1
kind: Namespace
metadata:
name: {{ .Release.Name }}
```
#### Roles
```yaml
kind: Role
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: namespace-manager
namespace: {{ .Release.Name }}
rules:
- apiGroups:
- ""
- extensions
- apps
- batch
- autoscaling
- networking.k8s.io
- traefik.containo.us
- rbac.authorization.k8s.io
- metrics.k8s.io
resources:
- deployments
- replicasets
- pods
- pods/exec
- pods/log
- pods/attach
- daemonsets
- statefulsets
- replicationcontrollers
- horizontalpodautoscalers
- services
- ingresses
- persistentvolumeclaims
- jobs
- cronjobs
- secrets
- configmaps
- serviceaccounts
- rolebindings
- ingressroutes
- middlewares
- endpoints
verbs:
- "*"
- apiGroups:
- ""
- metrics.k8s.io
- rbac.authorization.k8s.io
resources:
- resourcequotas
- roles
verbs:
- list
```
#### Rolebinding
```yaml
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
namespace: {{ .Release.Name }}
name: namespace-manager
subjects:
- kind: User
name: {{ .Release.Name }}
apiGroup: ""
roleRef:
kind: ClusterRole
name: namespace-manager
apiGroup: ""
```
### Manual Steps
#### Create a kubernetes certsigner pod
This keeps the client-ca crt and key secret and allows the cert to be signed and stored on the pod
#### Create the certsigner secret
```bash
kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key
```
#### Set up the certsigner pod
```bash
scp certsigner.yaml <server>:~/certsigner.yaml
kubectl apply -f certsigner.yaml
```
#### Generate a cert
```bash
export USER=<user>
docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl genrsa -out /$USER/$USER.key 2048
docker run -it -v $(pwd)/users/$USER:/$USER python:latest openssl req -new -key /$USER/$USER.key -out /$USER/$USER.csr -subj "/CN=$USER/O=user"
```
#### Create a new Userspace
```bash
helm template $USER ./namespace | kubectl --context admin apply -f -
```
#### Sign the cert
```bash
export USER=<user>
kubectl --context admin cp $(pwd)/users/$USER/$USER.csr certsigner:/certs/$USER.csr
kubectl --context admin exec -it --context admin certsigner -- openssl x509 -in /certs/$USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -CAcreateserial -out /certs/$USER.crt -days 5000
kubectl --context admin cp certsigner:/certs/$USER.crt $(pwd)/users/$USER/$USER.crt
```
#### Add to the config
```bash
kubectl config set-credentials $USER --client-certificate=$USER.crt --client-key=$USER.key
kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
```
#### Delete
```bash
kubectl config delete-context $USER
helm template $USER ./namespace | kubectl --context admin delete -f -
```
### Signing a user cert - detailed notes
NOTE: ca.crt and ca.key are in /var/lib/rancher/k3s/server/tls/client-ca.*
```bash
# First we create the credentials
# /CN=<username> - the user
# /O=<group> - the group
# Navigate to the user directory
export USER=<username>
cd $USER
# Generate a private key
openssl genrsa -out $USER.key 2048
# Check the key
# openssl pkey -in ca.key -noout -text
# Generate and send me the CSR
# The "user" group is my default group
openssl req -new -key $USER.key -out $USER.csr -subj "/CN=$USER/O=user"
# Check the CSR
# openssl req -in $USER.csr -noout -text
# If satisfactory, sign the CSR
# Copy from /var/lib/rancher/k3s/server/tls/client-ca.crt and client-ca.key
openssl x509 -req -in $USER.csr -CA ../client-ca.crt -CAkey ../client-ca.key -CAcreateserial -out $USER.crt -days 5000
# Review the certificate
# openssl x509 -in $USER.crt -text -noout
# Send back the crt
# cp $USER.crt $USER.key ../server-ca.crt ~/.kube/
kubectl config set-credentials $USER --client-certificate=$USER.crt --client-key=$USER.key
kubectl config set-context $USER --cluster=mainframe --namespace=$USER --user=$USER
# Now we create the namespace, rolebindings, and resource quotas
# kubectl apply -f k8s/
# Add the cluster
# CA file can be found at https://3.14.3.100:6443/cacerts
- cluster:
certificate-authority: server-ca.crt
server: https://3.14.3.100:6443
name: mainframe
# Test if everything worked
kubectl --context=$USER-context get pods
```
## Help
### Troubleshooting
@@ -541,13 +357,6 @@ mount -t xfs /dev/zvol/enc0/dcsi/apps/pvc-d5090258-cf20-4f2e-a5cf-330ac00d0049 /
umount /mnt/dcsi_pvc
```
## Database Backups
<https://docs.k3s.io/cli/etcd-snapshot>
Note, you must backup `/var/lib/rancher/k3s/server/token`
and use the contents as the token when restoring the backup as data is encrypted with that token.
## Uninstall
```bash

30
k3s/cluster-readonly.yaml
View File

@@ -1,30 +0,0 @@
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cluster-readonly
rules:
- apiGroups:
- ""
- rbac.authorization.k8s.io
- storage.k8s.io
- networking.k8s.io
- traefik.containo.us
resources:
- storageclasses
verbs:
- list
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: cluster-readonly
subjects:
- kind: Group
name: user
apiGroup: ""
roleRef:
kind: ClusterRole
name: cluster-readonly
apiGroup: ""

16
k3s/clusterrole.yaml
View File

@@ -1,16 +0,0 @@
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: user-readonly
rules:
- apiGroups:
- rbac.authorization.k8s.io
- storage.k8s.io
- networking.k8s.io
resources:
- clusterroles
- storageclasses
- ingressclasses
verbs:
- list
- watch

38
k3s/scripts/removeuser.sh
View File

@@ -1,38 +0,0 @@
#!/bin/bash
# Use
# ./removeuserspace <ssh_address> <server_fqdn (for kubectl)> <user>
export SERVER=$1
export FQDN=$2
export USER=$3
export CERT_DIR=$HOME/.kube/$FQDN/users/$USER
export CA_CERT_DIR=$HOME/.kube/$FQDN
export SERVER_USER_DIR="~/.kube/users/$USER"
export SERVER_NAME=$(echo "$FQDN" | sed 's/\./-/g')
export SERVER_USER="$USER-$SERVER_NAME"
export KUBECONFIG="$HOME/.kube/$USER-config"
echo "Checking if project namespace exists"
exists=$(ssh $SERVER "kubectl get namespace --output=jsonpath=\"{.items[?(@.metadata.name=='$USER')].metadata.name}\"")
if [ -z $exists ]; then
echo "Namespace not found, nothing to delete"
exit 1
else
echo "Namespace exists, deleting"
fi
echo "Deleting user namespace"
ssh $SERVER "kubectl delete -f $SERVER_USER_DIR/namespace.yaml"
echo "Deleting remote cert dir"
ssh $SERVER "rm -rf $SERVER_USER_DIR"
echo "Deleting local cert dir"
rm -rf $CERT_DIR
echo "Removing from kubeconfig"
rm $KUBECONFIG

12
k3s/scripts/setup.sh
View File

@@ -1,12 +0,0 @@
#!/bin/bash
# Use
# ./setup.sh <server_fqdn>
export SERVER=$1
ssh -t $SERVER sudo kubectl -n kube-system create secret generic certsigner --from-file /var/lib/rancher/k3s/server/tls/client-ca.crt --from-file /var/lib/rancher/k3s/server/tls/client-ca.key
scp certsigner.yaml $SERVER:~/certsigner.yaml
ssh $SERVER kubectl apply -f certsigner.yaml
scp clusterrole.yaml $SERVER:~/clusterrole.yaml
ssh $SERVER kubectl apply -f clusterrole.yaml

140
k3s/scripts/upsertuser.sh
View File

@@ -1,140 +0,0 @@
#!/bin/bash
# Use
# ./upsert.sh <ssh_address> <server_fqdn (for kubectl)> <user>
# Note, do not specify https:// or :port for the fqdn, just give the domain
# Port is expected to be 6443. You can change this later in the generated conf
# ./upsert.sh node1 containers.reeseapps.com testuser
# ./upsert.sh 192.168.1.10 mydomain.ddns.net admin
export SERVER=$1
export FQDN=$2
export KUBE_USER=$3
export CERT_DIR=$HOME/.kube/$FQDN/users/$KUBE_USER
export CA_CERT_DIR=$HOME/.kube/$FQDN
export SERVER_USER_DIR="~/.kube/users/$KUBE_USER"
export SERVER_NAME=$(echo "$FQDN" | sed 's/\./-/g')
export SERVER_USER="$KUBE_USER-$SERVER_NAME"
export KUBECONFIG="$HOME/.kube/$KUBE_USER-config"
if [ -z $KUBE_USER ]; then
echo "No arguments supplied! Format is ./upsert.sh <SERVER_FQDN> <USER>"
exit 1
fi
if [ -z $SERVER ]; then
echo "No server supplied for user $KUBE_USER"
exit 1
fi
if [ $KUBE_USER = "admin" ]; then
echo "Creating admin user for server $SERVER"
fi
echo "Creating cert dir"
mkdir -p $CERT_DIR
if [ $? -ne 0 ]; then
echo "Couldn't create cert dir at $CERT_DIR"
exit 1
fi
echo "Generating openssl cert"
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl genrsa -out /$KUBE_USER/$KUBE_USER.key 2048
if [ $KUBE_USER = "admin" ]; then
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl req -new -key /$KUBE_USER/$KUBE_USER.key -out /$KUBE_USER/$KUBE_USER.csr -subj "/CN=$KUBE_USER/O=system:masters"
else
podman run -it -v $CERT_DIR:/$KUBE_USER python:latest openssl req -new -key /$KUBE_USER/$KUBE_USER.key -out /$KUBE_USER/$KUBE_USER.csr -subj "/CN=$KUBE_USER/O=user"
fi
# /CN=admin/O=manager
if [ $? -ne 0 ]; then
echo "Couldn't create cert with Podman. Are you sure it's running?"
exit 1
fi
echo "Creating namespace dir on server"
ssh $SERVER "mkdir -p $SERVER_USER_DIR"
echo "Copying client csr to server cert dir"
scp $CERT_DIR/$KUBE_USER.csr $SERVER:$SERVER_USER_DIR/$KUBE_USER.csr
if [ $? -ne 0 ]; then
echo "Failed to copy client csr to server cert dir"
exit 1
fi
echo "Getting cert signing pod"
export CERT_POD=$(ssh $SERVER "kubectl get pod -n kube-system --selector=app=certsigner --output=jsonpath={.items..metadata.name}")
if [ -z $CERT_POD ]; then
echo "Installing certsigner"
helm template certsigner ./certsigner | ssh $SERVER "sudo -t -E kubectl apply -f -"
fi
while [ -z $CERT_POD ]; do
echo "Getting cert signing pod"
export CERT_POD=$(ssh $SERVER "kubectl get pod -n kube-system --selector=app=certsigner --output=jsonpath={.items..metadata.name}")
sleep 2
done
if [ $? -ne 0 ]; then
echo "Failed to install certsigner."
exit 1
fi
echo "Signing cert with pod $CERT_POD"
ssh $SERVER "kubectl -n kube-system cp $SERVER_USER_DIR/$KUBE_USER.csr $CERT_POD:/certs/$KUBE_USER.csr"
ssh $SERVER "kubectl -n kube-system exec $CERT_POD -- openssl x509 -in /certs/$KUBE_USER.csr -req -CA /keys/client-ca.crt -CAkey /keys/client-ca.key -set_serial $(python -c 'import random; print(random.randint(1000000000, 9999999999))') -out /certs/$KUBE_USER.crt -days 5000"
ssh $SERVER "kubectl -n kube-system cp $CERT_POD:/certs/$KUBE_USER.crt ~/.kube/users/$KUBE_USER/$KUBE_USER.crt"
echo "retrieving signed cert"
scp $SERVER:$SERVER_USER_DIR/$KUBE_USER.crt $CERT_DIR/$KUBE_USER.crt
echo "retrieving server ca"
wget --no-check-certificate https://$FQDN:6443/cacerts -O $CA_CERT_DIR/server-ca.pem
echo "creating $FQDN-$KUBE_USER context"
kubectl config set-context $FQDN-$KUBE_USER
echo "setting $FQDN-$KUBE_USER as current context"
kubectl config set current-context $FQDN-$KUBE_USER
echo "adding server to config with new context $FQDN-$KUBE_USER"
kubectl config set-cluster $FQDN --server=https://$FQDN:6443 --certificate-authority=$CA_CERT_DIR/server-ca.pem
kubectl config set contexts.$(kubectl config current-context).cluster $FQDN
echo "adding user to config file"
kubectl config set-credentials $SERVER_USER --client-certificate=$CERT_DIR/$KUBE_USER.crt --client-key=$CERT_DIR/$KUBE_USER.key
echo "setting user context"
kubectl config set contexts.$(kubectl config current-context).user $SERVER_USER
if [ $KUBE_USER = "admin" ]; then
echo "Admin user created, skipping namespace"
echo "export KUBECONFIG=$KUBECONFIG"
exit 0
fi
echo "Templating namespace with helm and copying to server"
helm template $KUBE_USER --set user=$KUBE_USER ./helm/namespace | ssh $SERVER "cat - > $SERVER_USER_DIR/namespace.yaml"
if [ $? -ne 0 ]; then
echo "Failed to template namespace. Is helm installed?"
exit 1
fi
echo "Creating namespace from template"
ssh $SERVER "kubectl apply -f $SERVER_USER_DIR/namespace.yaml"
echo "Setting namespace context"
kubectl config set contexts.$(kubectl config current-context).namespace $KUBE_USER
if [ $? -ne 0 ]; then
echo "Failed to create namespace"
exit 1
fi
echo "export KUBECONFIG=$KUBECONFIG"

71
k3s/tests/local-storage-test.yaml Normal file
View File

@@ -0,0 +1,71 @@
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: ssd-test
namespace: default
spec:
storageClassName: ssd
accessModes:
- ReadWriteMany
resources:
requests:
storage: 8Gi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: hdd-test
namespace: default
spec:
storageClassName: hdd
accessModes:
- ReadWriteMany
resources:
requests:
storage: 8Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: local-storage-test
namespace: default
spec:
selector:
matchLabels:
app: local-storage-test
template:
metadata:
labels:
app: local-storage-test
spec:
containers:
- image: debian
command:
- bash
- -c
- 'sleep infinity'
name: local-storage-test
volumeMounts:
- mountPath: /ssd
name: ssd
- mountPath: /hdd
name: hdd
resources:
limits:
memory: "4Gi"
cpu: "2"
requests:
memory: "1Mi"
cpu: "1m"
restartPolicy: Always
volumes:
- name: hdd
persistentVolumeClaim:
claimName: hdd-test
- name: ssd
persistentVolumeClaim:
claimName: ssd-test

2
k3s/upgrade-plan.yaml
View File

@@ -16,4 +16,4 @@ spec:
serviceAccountName: system-upgrade
upgrade:
image: rancher/k3s-upgrade
channel: https://update.k3s.io/v1-release/channels/latest
channel: https://update.k3s.io/v1-release/channels/stable