homelab/active/os_arch/kubernetes.md

Kubernetes

• Kubernetes
  • Setup
  • MetalLB
  • Ingress Nginx
  • Cert Manager
  • Storage

https://wiki.archlinux.org/title/Kubernetes

Setup

pacman -S kubeadm kubelet containerd cni-plugins cilium-cli helm kubectl

/etc/modules-load.d/k8s.conf

overlay
br_netfilter

/etc/sysctl.d/k8s.conf

net.bridge.bridge-nf-call-iptables  = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.ipv4.ip_forward                 = 1

/etc/containerd/config.toml

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
  ...
  [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
    SystemdCgroup = true

reboot
mkdir /etc/containerd
containerd config default > /etc/containerd/config.toml
systemctl enable --now containerd
systemctl enable --now kubelet
kubeadm init --pod-network-cidr='10.244.0.0/16'

mkdir -p $HOME/.kube
cp /etc/kubernetes/admin.conf $HOME/.kube/config
chown $(id -u):$(id -g) $HOME/.kube/config

cilium-cli install

# Note the "-" at the end, this removes the taint
kubectl taint node kube node-role.kubernetes.io/control-plane:NoSchedule-

MetalLB

Install with helm:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb -n kube-system

You must create a production pool if IP Addresses. Apply the following config (substituting your public IP address space)

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: production
  namespace: kube-system
spec:
  # Production services will go here. Public IPs are expensive, so we leased
  # just 4 of them.
  addresses:
  - 192.168.122.206/32

Here is an example service which allows IP sharing and uses the "production" address pool.

apiVersion: v1
kind: Service
metadata:
  name: ingress-nginx-demo
  annotations:
    metallb.universe.tf/address-pool: production
    metallb.universe.tf/allow-shared-ip: "nginx"
spec:
  type: LoadBalancer
  externalTrafficPolicy: Cluster
  selector:
    app.kubernetes.io/name: ingress-nginx-demo
  ports:
  - name: ingress-nginx-demo
    protocol: TCP
    port: 8000
    targetPort: http

Ingress Nginx

Now we need an ingress solution (preferably with certs for https). We'll be using nginx since it's a little bit more configurable than traefik (though don't sell traefik short, it's really good. Just finnicky when you have use cases they haven't explicitly coded for).

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update
helm upgrade --install \
    ingress-nginx \
    ingress-nginx/ingress-nginx \
    --values ingress-nginx-values.yaml \
    --namespace ingress-nginx \
    --create-namespace

Cert Manager

Cert manager handles automatic TLS for our ingress with Let's Encrypt.

Install with helm:

helm repo add jetstack https://charts.jetstack.io
helm repo update
helm upgrade --install \
    cert-manager jetstack/cert-manager \
    --namespace cert-manager \
    --create-namespace \
    --version v1.12.4 \
    --set installCRDs=true

Now we need to create an issuer. Apply the following config:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    # The ACME server URL
    server: https://acme-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: nginx@ducoterra.net
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt
    # Enable the HTTP-01 challenge provider
    solvers:
    - http01:
        ingress:
          class: nginx

Here's an example ingress definition:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress-nginx-demo
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.org/client-max-body-size: "0"
spec:
  rules:
  - host: ingress-nginx-demo.reeseapps.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: ingress-nginx-demo
            port:
              number: 80
  tls:
  - hosts:
    - ingress-nginx-demo.reeseapps.com
    secretName: ingress-nginx-demo-tls-cert

Storage

We can use host-path storage immediately like so:

apiVersion: v1
kind: Pod
metadata:
  name: test-webserver
spec:
  containers:
  - name: test-webserver
    image: registry.k8s.io/test-webserver:latest
    volumeMounts:
    - mountPath: /var/local/aaa
      name: mydir
    - mountPath: /var/local/aaa/1.txt
      name: myfile
  volumes:
  - name: mydir
    hostPath:
      # Ensure the file directory is created.
      path: /var/local/aaa
      type: DirectoryOrCreate
  - name: myfile
    hostPath:
      path: /var/local/aaa/1.txt
      type: FileOrCreate