K3S

Raspberry Pi

Set static DNS

For the love of god do this before anything else. Your pods depend on it.

echo 'static domain_name_servers=1.1.1.1 1.0.0.1' >> /etc/dhcpcd.conf
service dhcpcd restart

Wait for network at boot

Because so many services rely on proper mounting of nfs you should really wait for network at boot:

raspi-config
> boot
> Wait for network

Install K3s

Enable cgroups and legacy iptables

echo "$(cat /boot/cmdline.txt) cgroup_memory=1 cgroup_enable=memory" > /boot/cmdline.txt
iptables -F
update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
reboot

Install k3s on our lead server (assuming you have etcd configured) to generate token and certs

curl -sfL https://get.k3s.io | sh -s -
systemctl stop k3s
export token=$(cat /var/lib/rancher/k3s/server/node-token)
curl -sfL https://get.k3s.io | sh -s - \
    --datastore-endpoint=https://3.14.3.102:2379,https://3.14.3.103:2379,https://3.14.3.107:2379 \
    --datastore-cafile=/certs/ca.pem \
    --datastore-certfile=/certs/client.pem \
    --datastore-keyfile=/certs/client-key.pem \
    --token $token
journalctl -u k3s -f

Join worker node (token located at /var/lib/rancher/k3s/server/node-token)

export token=<token>
curl -sfL https://get.k3s.io | sh -s - \
    --datastore-endpoint=https://3.14.3.102:2379,https://3.14.3.103:2379,https://3.14.3.107:2379 \
    --datastore-cafile=/certs/ca.pem \
    --datastore-certfile=/certs/client.pem \
    --datastore-keyfile=/certs/client-key.pem \
    --token $token

Generate certs for a new user

export USER=

mkdir $USER
cd $USER
openssl genrsa -out $USER.key 2048
openssl req -new -key $USER.key -out $USER.csr -subj "/CN=$USER/O=user"
sudo openssl x509 -req -in $USER.csr -CA /var/lib/rancher/k3s/server/tls/client-ca.crt -CAkey /var/lib/rancher/k3s/server/tls/client-ca.key -CAcreateserial -out $USER.crt -days 5000
sudo chown pi:pi $USER.crt
sudo kubectl create namespace $USER
sudo kubectl -n $USER create role $USER --verb=get,list,create,update,patch,watch,delete,deletecollection --resource=deployments,daemonsets,pods,pods/exec,pods/log,pods/attach,services,secrets,configmaps,persistentvolumeclaims,endpoints,ingresses.extensions,ingresses.networking.k8s.io
sudo kubectl -n $USER create rolebinding -n $USER $USER --role=$USER --user=$USER
sudo kubectl create clusterrole volume --verb=get,list,create,update,patch,watch,delete,deletecollection --resource=persistentvolumes
sudo kubectl create clusterrolebinding --role=volume --user=$USER

Copy certs

export SERVER=
export USER=
scp $SERVER:~/$USER/$USER.crt  $SERVER:~/$USER/$USER.csr  $SERVER:~/$USER/$USER.key certs/

Test a bunch of deploys

for i in {1..10}; do kubectl create deploy test$i --image=nginx; done;
for i in {1..10}; do kubectl delete deploy test$i; done;

To uninstall

/usr/local/bin/k3s-uninstall.sh

To drain a node (for maintenance)

export NODE=
kubectl drain $NODE --ignore-daemonsets --delete-local-data

Create an NFS pv and pvc

pv.yaml

apiVersion: v1
kind: PersistentVolume
metadata:
  name: {{ .Release.Name }}
  annotations:
    "helm.sh/resource-policy": keep
spec:
  storageClassName: {{ .Release.Name }}
  accessModes:
    - ReadWriteOnce
  capacity:
    storage: 100Gi
  nfs:
    server: freenas
    path: "/mnt/enc0/pi/pihole"

pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: {{ .Release.Name }}
  annotations:
    "helm.sh/resource-policy": keep
spec:
  storageClassName: {{ .Release.Name }}
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

Disable local-storage

'--disable=local-storage' \
systemctl daemon-reload
service k3s restart

Install Docker

curl -sSL https://get.docker.com | sh
sudo usermod -aG docker pi
docker run hello-world
sudo apt-get install -y libffi-dev libssl-dev
sudo apt-get install -y python3 python3-pip
sudo apt-get remove python-configparser
sudo pip3 -v install docker-compose

Pihole password

echo $(kubectl get secret pihole --output=jsonpath='{.data.WEBPASSWORD}' | base64 --decode)