overhauls of most service docs

templates/os/foobar.md

@@ -0,0 +1,411 @@
+# Foobar
+
+These docs frequently reference an "operator" and a "client". The operator is
+your laptop, computer, pipeline, or any device will be connecting to, or
+configuring, the OS during/after install. The client is the device you're
+installing the OS onto.
+
+- [Foobar](#foobar)
+  - [Installation](#installation)
+  - [Network](#network)
+    - [Firewalld Configuration](#firewalld-configuration)
+    - [Setup SSH](#setup-ssh)
+    - [Fail2Ban](#fail2ban)
+    - [Set eui64 on network interface](#set-eui64-on-network-interface)
+    - [Set up Network Bridge](#set-up-network-bridge)
+  - [Storage](#storage)
+    - [BTRFS Parent Volumes](#btrfs-parent-volumes)
+    - [BTRFS Snapshots](#btrfs-snapshots)
+    - [BTRFS Maintenance](#btrfs-maintenance)
+    - [TPM2 Luks Decryption](#tpm2-luks-decryption)
+  - [Users](#users)
+    - [Change your password](#change-your-password)
+  - [Updates](#updates)
+    - [Automatic Updates](#automatic-updates)
+  - [Monitoring](#monitoring)
+    - [Disk Usage](#disk-usage)
+    - [Disk Wear](#disk-wear)
+  - [Backups](#backups)
+    - [Downgrading Kernel](#downgrading-kernel)
+  - [Apps](#apps)
+    - [Package Manager](#package-manager)
+    - [Install and Enable Cockpit](#install-and-enable-cockpit)
+  - [Install and Enable Virtualization](#install-and-enable-virtualization)
+  - [Install and Enable Containers](#install-and-enable-containers)
+  - [Troubleshooting](#troubleshooting)
+    - [Disable Swap](#disable-swap)
+    - [Disable Selinux](#disable-selinux)
+
+## Installation
+
+1. Configure network first
+   1. Set a hostname
+   2. Disable ipv6 privacy extensions
+2. Software Selection
+   1. Headless Management
+3. User Creation
+   1. Set a simple password, we'll change it later
+4. Disk partitioning
+   1. Select manual (blivet) partitioning
+   2. Create a 1GB EFI system partition and mount it at `/boot/efi`
+   3. Create a 1GB ext4 partition and mount it at `/boot`
+   4. Create a btrfs volume with the remaining data and name it something unqiue, do not mount it
+   5. Create a btrfs subvolume called "root" and mount it at `/`
+   6. Create any other btrfs subvolumes you might need
+5. Take note of the ipv4 and ipv6 address. Update any DNS records at this time.
+6. Install and reboot
+
+## Network
+
+### Firewalld Configuration
+
+Set the default firewalld zone to `public`
+
+```bash
+# Note, you probably don't have to do this. Check Cockpit Network -> Firewall
+# firewall-cmd --set-default-zone=public
+```
+
+Firewalld will be on and blocking by default. You can check the zone and allowed ports with:
+
+```bash
+firewall-cmd --zone=public --list-ports
+firewall-cmd --zone=public --list-services
+```
+
+Allow Cockpit with
+
+```bash
+firewall-cmd --permanent --zone=public --add-port=9090/tcp
+firewall-cmd --reload
+```
+
+### Setup SSH
+
+See [README](/README.md#ssh-key-generation)
+
+### Fail2Ban
+
+On the server:
+
+```bash
+# Run tmux session
+tmux
+
+dnf install -y fail2ban
+
+# Setup initial rules
+cat <<EOF > /etc/fail2ban/jail.local
+# Jail configuration additions for local installation
+
+# Adjust the default configuration's default values
+[DEFAULT]
+# Optional enter an trusted IP never to ban
+# ignoreip = 2600:1700:1e6c:a81f::0/64
+bantime  = 6600
+backend = auto
+
+# The main configuration file defines all services but
+# deactivates them by default. We have to activate those neeeded
+[sshd]
+enabled = true
+EOF
+
+systemctl enable fail2ban --now
+
+# OPTIONAL: follow logs
+tail -f /var/log/fail2ban.log
+```
+
+Checking, banning, unbanning
+
+```bash
+# See banned clients
+fail2ban-client banned
+# See jails (sshd should be one of them)
+fail2ban-client status
+# Unban a client from the sshd jail
+fail2ban-client set sshd unbanip <IP address>
+```
+
+### Set eui64 on network interface
+
+Ensures consistent mac-based IPv6 address.
+
+```bash
+nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64
+```
+
+### Set up Network Bridge
+
+Networking -> Add bridge -> add network interface and save
+
+```bash
+nmcli connection modify bridge0 ipv6.addr-gen-mode eui64
+```
+
+## Storage
+
+### BTRFS Parent Volumes
+
+In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see
+all subvolumes.
+
+```conf
+UUID=64beedac-c0c9-48bf-a3ae-7707df6ebc97 /btrfs/3dserver-root    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
+UUID=3c76b83f-7547-4c18-b08f-9e7902022b8d /btrfs/3dserver-data    btrfs   subvolid=5,compress=zstd:1,x-systemd.device-timeout=0 0 0
+```
+
+```bash
+systemctl daemon-reload
+mount -a --mkdir
+```
+
+### BTRFS Snapshots
+
+<https://en.opensuse.org/openSUSE:Snapper_Tutorial>
+
+<http://snapper.io/manpages/snapper-configs.html>
+
+We'll be using snapper, a tool for automating and controlling snapshot behavior.
+
+```bash
+dnf install snapper dnf-plugin-snapper
+
+# Allow selinux management
+semanage permissive -a snapperd_t
+
+# Note, if you mess something up you can run snapper -c root delete-config to delete
+# System configs are stored in /etc/sysconfig/snapper as well as /etc/snapper
+snapper -c root create-config /
+snapper -c data create-config /path/to/other/data
+
+# Enable automatic snapshots
+systemctl enable --now snapper-timeline.timer
+# Enable automatic cleanup
+systemctl enable --now snapper-cleanup.timer
+# Enable snapshots on boot
+systemctl enable --now snapper-boot.timer
+
+# List snapshots
+snapper -c root list
+# Create snapshot manually
+snapper -c root create --description "test snapshot"
+# Delete first snapshot
+snapper -c root delete 1
+```
+
+Note - you probably don't want to keep yearly snapshots.
+Edit `/etc/snapper/configs/root` and change `TIMELINE_LIMIT_YEARLY=` to `0`.
+
+### BTRFS Maintenance
+
+```bash
+# Start a scrub in the foreground (-B) at /
+btrfs scrub start -B /
+```
+
+### TPM2 Luks Decryption
+
+Mostly taken from here:
+<https://gist.github.com/jdoss/777e8b52c8d88eb87467935769c98a95>
+
+PCR reference for `--tpm2-pcrs` args
+
+```text
+0: System firmware executable
+2: Kernel
+4: Bootloader
+7: Secure boot state
+8: Cmdline
+9: Initrd
+```
+
+Note, if your threat vector is people trying to get data off your old disks after throwing them
+away, you can set `--tpm2-pcrs=""`. Someone could gain access to your encrypted partition if they
+can access your machine physically by manipulating the boot parameters but you're guaranteed to
+unlock despite updates and upgrades.
+
+Basic commands:
+
+```bash
+# Run tmux session
+tmux
+
+# Show tpm2 devices
+systemd-cryptenroll --tpm2-device=list
+# Show crypto luks block devices
+blkid -t TYPE=crypto_LUKS
+
+# Enroll the tpm2 device with systemd-cryptenroll
+systemd-cryptenroll /dev/nvme0n1p3 --tpm2-device=auto --tpm2-pcrs=""
+
+####################
+##### OPTIONAL #####
+####################
+# If you have lots of devices to decrypt (like a btrfs raid array), use these commands.
+# Get all crypto luks partitions
+blkid | grep crypto_LUKS
+# List them all space-separated and drop the '/dev'
+LUKS_DEVS="nvme0n1p4 nvme1n1p1 nvme2n1p1 nvme3n1p1 nvme5n1p1 nvme4n1p1 nvme6n1p1"
+
+# Check that your list is good
+for dev in $LUKS_DEVS; do echo will enroll /dev/$dev; done
+
+# Enroll
+for dev in $LUKS_DEVS; do \
+echo "Enrolling /dev/$dev"; \
+systemd-cryptenroll /dev/$dev --tpm2-device=auto --tpm2-pcrs=""; \
+done
+########################
+##### END OPTIONAL #####
+########################
+
+# Append to command line args
+echo "add_dracutmodules+=\" tpm2-tss \"" | tee /etc/dracut.conf.d/tpm2.conf
+dracut -f
+```
+
+Finally, `vim /etc/default/grub` and add `rd.luks.options=tpm2-device=auto` to GRUB_CMDLINE_LINUX
+
+```bash
+# Update Grub
+grub2-mkconfig -o /boot/grub2/grub.cfg
+reboot
+# Cross your fingers that you don't have to go type in the password manually.
+# Yes, 60 full seconds is too long. Go type your password in.
+```
+
+If you need to reenroll for some reason:
+
+```bash
+# Reenroll
+systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=""
+```
+
+## Users
+
+### Change your password
+
+In Cockpit navigate to Accounts -> user -> Set password
+
+## Updates
+
+### Automatic Updates
+
+In Cockpit navigate to software updates -> automatic updates -> install -> security updates only
+
+## Monitoring
+
+In Cockpit: Overview -> View metrics and history -> Install PCP Support -> Metrics settings -> Turn on Collect Metrics
+
+### Disk Usage
+
+TODO
+
+### Disk Wear
+
+TODO
+
+## Backups
+
+See [borg.md](/active/systemd_borg/borg.md)
+
+### Downgrading Kernel
+
+```bash
+dnf install koji
+
+# Note: format is kernel-version.fedora-version
+cd $(mktemp -d) && koji download-build --arch=x86_64 --arch=noarch kernel-6.11.3-300.fc41 && dnf install ./*
+
+reboot
+```
+
+## Apps
+
+### Package Manager
+
+Configure dnf to use the fastest mirror:
+
+```bash
+echo 'fastestmirror=1' >> /etc/dnf/dnf.conf
+dnf clean all
+dnf update --refresh -y
+# libdnf5 is required for ansible to work
+dnf install -y glances tmux vim python3-libdnf5
+```
+
+### Install and Enable Cockpit
+
+<https://cockpit-project.org/running>
+
+```bash
+dnf install cockpit
+systemctl enable --now cockpit.socket
+firewall-cmd --add-service=cockpit
+firewall-cmd --add-service=cockpit --permanent
+```
+
+## Install and Enable Virtualization
+
+Don't forget to add a btrfs subvolume for `/var/lib/libvirt`
+
+```bash
+# Since we already created our /btrfs mountpoint, this volume will show up automatically
+# at /btrfs/libvirt
+btrfs sub create /btrfs/libvirt
+```
+
+Now create an fstab entry that mounts the volume at /var/lib/libvirt
+
+```bash
+UUID=... /var/lib/libvirt    btrfs   subvol=libvirt,compress=zstd:1,x-systemd.device-timeout=0 0 0
+```
+
+Mount the libvirt volume:
+
+```bash
+systemctl daemon-reload
+mount -a --mkdir
+# Check that the mount was successful. This will print something if our mount worked.
+mount | grep -i /var/lib/libvirt
+```
+
+Create a snapshot schedule for libvirt.
+
+```bash
+snapper -c libvirt create-config /var/lib/libvirt
+# Don't forget to edit "YEARLY" at /etc/snapper/configs/libvirt
+```
+
+Install and enable the virtualization service.
+
+```bash
+dnf group install --with-optional virtualization
+systemctl enable --now libvirtd
+```
+
+Install the cockpit machines application.
+
+## Install and Enable Containers
+
+## Troubleshooting
+
+### Disable Swap
+
+```bash
+swapoff -a
+zramctl --reset /dev/zram0
+dnf -y remove zram-generator-defaults
+```
+
+### Disable Selinux
+
+By default selinux will be enforcing. You can set it to permissive with
+
+```bash
+setenforce 0
+```
+
+And then make it permanent by editing `/etc/selinux/config` and inserting `SELINUX=permissive`.

templates/podman/compose/README.md

@@ -0,0 +1,3 @@
+# Compose
+
+Put your compose.yaml here.

templates/podman/compose/compose.yaml

@@ -0,0 +1,12 @@
+services:
+  foobar:
+    container_name: foobar
+    restart: always
+    image: localhost/foobar:latest
+    ports:
+      - "8080:8080"
+    volumes:
+      - /home/foobar/data:/var/app/data
+    security_opt:
+      - label=disable
+    userns_mode: keep-id

templates/podman/foobar.md

@@ -0,0 +1,131 @@
+# Podman foobar
+
+- [Podman foobar](#podman-foobar)
+  - [Setup foobar Project](#setup-foobar-project)
+  - [Install foobar](#install-foobar)
+    - [Create the foobar user](#create-the-foobar-user)
+    - [Write the foobar compose spec](#write-the-foobar-compose-spec)
+    - [Convert foobar compose spec to quadlets](#convert-foobar-compose-spec-to-quadlets)
+    - [Expose foobar](#expose-foobar)
+      - [firewalld](#firewalld)
+  - [Backup foobar](#backup-foobar)
+  - [Upgrade foobar](#upgrade-foobar)
+    - [Upgrade Quadlets](#upgrade-quadlets)
+  - [Notes](#notes)
+    - [SELinux](#selinux)
+
+## Setup foobar Project
+
+1. Copy and rename this folder to active/podman_foobar
+2. Find and replace foobar with the name of the service.
+3. Create the rootless user to run the podman containers
+4. Write the compose.yaml spec for your service
+5. Convert the compose.yaml spec to a quadlet
+6. Install the quadlet on the podman server
+7. Expose the quadlet service
+8. Install a backup service and timer
+
+## Install foobar
+
+### Create the foobar user
+
+```bash
+useradd foobar
+
+su - foobar
+ssh-keygen
+exit
+cp ~/.ssh/authorized_keys /home/foobar/.ssh/authorized_keys
+chown foobar:foobar /home/foobar/.ssh/authorized_keys
+loginctl enable-linger $(id -u foobar)
+```
+
+SSH into the server as foobar
+
+```bash
+systemctl --user enable podman-restart
+systemctl --user enable --now podman.socket
+mkdir -p ~/.config/containers/systemd
+```
+
+### Write the foobar compose spec
+
+Edit the compose.yaml at active/foobar/compose/compose.yaml
+
+### Convert foobar compose spec to quadlets
+
+On your local machine:
+
+```bash
+# Generate the systemd service
+podman run \
+--security-opt label=disable \
+--rm \
+-v $(pwd)/active/foobar/:/compose \
+-v $(pwd)/active/foobar/quadlets:/quadlets \
+quay.io/k9withabone/podlet \
+-f /quadlets \
+-i \
+--overwrite \
+compose /compose/compose.yaml
+
+# Copy the files to the server
+scp -r active/foobar/quadlets/. foobar:~/.config/containers/systemd/
+```
+
+```bash
+ssh foobar systemctl --user daemon-reload
+ssh foobar systemctl --user restart foobar
+# Enables auto-update service which will pull new container images automatically every day
+ssh foobar systemctl --user enable --now podman-auto-update.timer
+```
+
+### Expose foobar
+
+1. If you need a domain, follow the [DDNS instructions](/active/podman_ddns/ddns.md#install-a-new-ddns-service)
+2. For a web service, follow the [Caddy instructions](/active/podman_caddy/caddy.md#adding-a-new-caddy-record)
+3. Finally, follow your OS's guide for opening ports via its firewall service.
+
+#### firewalld
+
+```bash
+# command to get current active zone and default zone
+firewall-cmd --get-active-zones
+firewall-cmd --get-default-zone
+
+# command to open 443 on tcp
+firewall-cmd --permanent --zone=<zone> --add-port=443/tcp
+
+# command to open 80 and 443 on tcp and udp
+firewall-cmd --permanent --zone=<zone> --add-port={80,443}/{tcp,udp}
+
+# command to list available services and then open http and https
+firewall-cmd --get-services
+firewall-cmd --permanent --zone=<zone> --add-service={http,https}
+```
+
+## Backup foobar
+
+Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
+
+## Upgrade foobar
+
+### Upgrade Quadlets
+
+Upgrades should be a repeat of [writing the compose spec](#convert-compose-to-quadlet) and [installing the quadlets](#convert-compose-to-quadlet)
+
+```bash
+scp -r quadlets/. foobar:~/.config/containers/systemd/
+ssh foobar systemctl --user daemon-reload
+ssh foobar systemctl --user restart foobar
+```
+
+## Notes
+
+### SELinux
+
+<https://blog.christophersmart.com/2021/01/31/podman-volumes-and-selinux/>
+
+:z allows a container to share a mounted volume with all other containers.
+
+:Z allows a container to reserve a mounted volume and prevents any other container from accessing.

templates/podman/quadlets/README.md

@@ -0,0 +1,3 @@
+# Quadlets
+
+Put your quadlets here.

templates/systemd/foobar.md

@@ -0,0 +1,49 @@
+# foobar
+
+- [foobar](#foobar)
+  - [Setup foobar Project](#setup-foobar-project)
+  - [Service Variables](#service-variables)
+  - [Install foobar](#install-foobar)
+  - [Upgrade foobar](#upgrade-foobar)
+  - [Backup foobar](#backup-foobar)
+  - [Restore foobar](#restore-foobar)
+  - [Uninstall foobar](#uninstall-foobar)
+
+## Setup foobar Project
+
+1. Copy and rename this folder to active/systemd_foobar
+2. Find and replace foobar with the name of the service
+3. Write the foobar.service spec
+4. (OPTIONAL) Write the foobar.timer spec
+5. (OPTIONAL) Write the foobar.sh.j2 template
+6. Write the install_foobar.yaml ansible template
+7. Install the service via ansible
+8. Expose the service
+9. Install a backup service and timer
+
+## Service Variables
+
+1. For most vars,populate `vars.yaml`
+2. For secret vars, create a new folder called `secrets/` and put a `vars.yaml` there.
+
+## Install foobar
+
+```bash
+# Run the playbook
+ansible-playbook \
+-i ansible/inventory.yaml \
+-l podman \
+active/systemd_foobar/install_foobar.yaml \
+-e "@active/systemd_foobar/vars.yaml" \
+-e "@active/systemd_foobar/secrets/vars.yaml"
+```
+
+## Upgrade foobar
+
+## Backup foobar
+
+Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
+
+## Restore foobar
+
+## Uninstall foobar

templates/systemd/foobar.service

@@ -0,0 +1,12 @@
+[Unit]
+Description=Runs foobar
+After=network.target
+Wants=network-online.target
+
+[Service]
+Restart=no
+Type=oneshot
+ExecStart=/usr/local/script/foobar.sh
+
+[Install]
+WantedBy=multi-user.target

templates/systemd/foobar.sh.j2

@@ -0,0 +1,3 @@
+#!/bin/sh
+
+echo "hello foobar!"

templates/systemd/foobar.timer

@@ -0,0 +1,11 @@
+[Unit]
+Description=Run Foobar every day
+
+[Timer]
+OnCalendar=*-*-* 2:00:00
+AccuracySec=10min
+Persistent=true
+Unit=foobar.service
+
+[Install]
+WantedBy=timers.target

templates/systemd/install_foobar.yaml

@@ -0,0 +1,39 @@
+- name: Create Backup Service
+  hosts: all
+  vars_files:
+    - secrets/vars.yaml
+  tasks:
+  - name: Create /usr/local/script dir
+    ansible.builtin.file:
+      path: /usr/local/script
+      state: directory
+      mode: '0755'
+  - name: Copy foobar.service
+    template:
+      src: backup.service
+      dest: /etc/systemd/system/foobar.service
+      owner: root
+      group: root
+      mode: '0644'
+  - name: Copy foobar.timer
+    template:
+      src: backup.timer
+      dest: /etc/systemd/system/foobar.timer
+      owner: root
+      group: root
+      mode: '0644'
+  - name: Template foobar.sh
+    template:
+      src: backup.sh.j2
+      dest: /usr/local/script/foobar.sh
+      owner: root
+      group: root
+      mode: '0744'
+  - name: Something foobar related that might fail
+    script: echo 'hello foobar!'
+    ignore_errors: yes
+  - name: Reload foobar.timer
+    ansible.builtin.systemd_service:
+      name: foobar.timer
+      enabled: true
+      daemon_reload: true

templates/systemd/vars.yaml

@@ -0,0 +1 @@
+name: foobar