moving closer to podman systemd services for everything

This commit is contained in:
2025-04-19 12:52:56 -04:00
parent 9acff25d43
commit 6e393d90ee
47 changed files with 1455 additions and 433 deletions

View File

@@ -2,7 +2,7 @@
- [Framework 16 Fixes](#framework-16-fixes)
- [Wake from Sleep](#wake-from-sleep)
- [Wrong keys pressed in the browser, keyboard occasionally freezes after clicking a field.](#wrong-keys-pressed-in-the-browser-keyboard-occasionally-freezes-after-clicking-a-field)
- [Wrong keys pressed in the browser, keyboard occasionally freezes after clicking a field](#wrong-keys-pressed-in-the-browser-keyboard-occasionally-freezes-after-clicking-a-field)
## Wake from Sleep
@@ -28,6 +28,6 @@ If you want to specifically disable the framework keyboard you can use this:
ACTION=="add", SUBSYSTEM=="usb", ATTRS{idVendor}=="32ac", ATTRS{idProduct}=="0012", RUN+="/bin/sh -c 'echo disabled > /sys$env{DEVPATH}/power/wakeup'"
```
## Wrong keys pressed in the browser, keyboard occasionally freezes after clicking a field.
## Wrong keys pressed in the browser, keyboard occasionally freezes after clicking a field
This was caused by a Keychron M3 mini mouse.

View File

@@ -20,8 +20,13 @@
- [QEMU Images](#qemu-images)
- [Firewalld](#firewalld)
- [Backups](#backups)
- [Connect to the ISCSI Backup Target](#connect-to-the-iscsi-backup-target)
- [Connect to Backup Target with Cockpit](#connect-to-backup-target-with-cockpit)
- [Connect to Backup Target with iscsiadm](#connect-to-backup-target-with-iscsiadm)
- [Format backup disk](#format-backup-disk)
- [Troubleshooting Backup ISCSI Connection](#troubleshooting-backup-iscsi-connection)
- [Quick Backup](#quick-backup)
- [Regular Backup to an NFS Share](#regular-backup-to-an-nfs-share)
- [Regular Backups with Borg](#regular-backups-with-borg)
- [Optional Steps](#optional-steps)
- [Docker with Podman as Runtime](#docker-with-podman-as-runtime)
- [Vanilla Docker](#vanilla-docker)
@@ -156,6 +161,17 @@ systemctl enable fail2ban --now
tail -f /var/log/fail2ban.log
```
Checking, banning, unbanning
```bash
# See banned clients
fail2ban-client banned
# See jails (sshd should be one of them)
fail2ban-client status
# Unban a client from the sshd jail
fail2ban-client set sshd unbanip <IP address>
```
## BTRFS Parent Volumes
In `/etc/fstab`, add the parent volumes for your disks mounted with subvolid=5 at `/btrfs` so you can see
@@ -414,44 +430,94 @@ firewall-cmd --reload
## Backups
Note: this assumes you've set up [an iscsi backup disk](/infrastructure/graduated/truenas/truenas.md#iscsi-backup-volumes)
### Connect to the ISCSI Backup Target
#### Connect to Backup Target with Cockpit
1. Storage -> Hamburger menu -> Add iSCSI portal
2. Type your portal address, username, and password
#### Connect to Backup Target with iscsiadm
<https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/5/html/online_storage_reconfiguration_guide/iscsi-replacements_timeout#iscsi-replacements_timeout>
```bash
# Set username and password for discovered nodes
# Optionally you can add "-T <target name>" to specify which target has the username/password
iscsiadm -m node \
-o update \
-n node.session.auth.username -v username \
-n node.session.auth.password -v password
# Set replacement_timeout to 10 minutes in case server reboots
iscsiadm -m node \
-o update \
-n node.session.timeo.replacement_timeout -v 600
systemctl restart iscsid
# Discover targets
iscsiadm -m discovery -t st -p driveripper.reeselink.com
# Login to all nodes
iscsiadm -m node -l
```
#### Format backup disk
```bash
# list disks
lsblk
# Create partition
fdisk /dev/sdx
# Format partition with btrfs
mkfs.btrfs /dev/sdx1
# Get the UUID
blkid /dev/sdx1
```
Update /etc/fstab with the iscsi disk details.
Note:
- `x-systemd.automount` which only mounts the device when it's accessed.
- `x-systemd.mount-timeout=30` allows a 30 second timeout
- `_netdev` ensures the device won't be mounted until after the network is available
```conf
UUID=... /btrfs/some-name btrfs subvolid=5,compress=zstd:1,x-systemd.automount,x-systemd.mount-timeout=30,_netdev 0 0
```
#### Troubleshooting Backup ISCSI Connection
```bash
# List targets
iscsiadm -m node
# Delete node
iscsiadm -m node -o delete -T iqn.2022-01.com.reeselink:driveripper:iqn.2022-01.com.reeselink:driveripper
# List discovered targets
iscsiadm -m discovery
# Delete from discovery db
iscsiadm -m discoverydb -t sendtargets -p driveripper.reeselink.com -o delete
```
### Quick Backup
```bash
rsync -av / \
--exclude={"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"} \
/mnt/root/
rsync -av --progress --exclude '.snapshots' /btrfs/yellow/root /btrfs/backup-yellow --dry-run
```
### Regular Backup to an NFS Share
### Regular Backups with Borg
1. Create a new dataset called "<server>_backup"
2. Remove "other" read/exec permissions from the dataset
3. Create a new NFS share for that dataset with maproot user and group set to root
4. Mount the NFS share to your server at `/backup`
5. Copy the following script into /root/backup.sh
```bash
#!/bin/bash
BACKUP_PATH="/backup"
EXCLUDE_DIR='{"/dev/*","/proc/*","/sys/*","/tmp/*","/run/*","/mnt/*","/media/*","/lost+found"}'
SOURCE_DIR="/"
rsync -aAXv ${SOURCE_DIR} --exclude=${EXCLUDE_DIR} ${BACKUP_PATH}
if [ $? -eq 0 ]; then
echo "Backup completed successfully"
else
echo "Some error occurred during backup"
fi
```
6. `chmod +x /root/backup.sh`
7. `crontab -e`
```cron
0 2 * * * bash /root/backup.sh >> /root/backup.log
```
See [borg.md](/systemd/graduated/borg/borg.md)
## Optional Steps

View File

@@ -4,6 +4,7 @@
- [Framework 16 Fixes](#framework-16-fixes)
- [Wake from Sleep](#wake-from-sleep)
- [Wrong keys pressed in the browser](#wrong-keys-pressed-in-the-browser)
- [Wifi Powersave](#wifi-powersave)
- [Podman](#podman)
- [Autostarting services with quadlets](#autostarting-services-with-quadlets)
- [Network](#network)
@@ -38,6 +39,30 @@ sudo udevadm control --reload-rules && sudo udevadm trigger
Sometimes keys will stop working when using search bars or do strange things like move the page around. This seems to be caused by some "alt" keypress combination. Pressing "alt" twice fixes it.
### Wifi Powersave
**NOTE: THIS DOESN'T WORK. IT CAUSES WIFI DISCONNECT AND RECONNECT ISSUES.**
<https://www.networkmanager.dev/docs/api/latest/settings-802-11-wireless.html>
<https://gist.github.com/jcberthon/ea8cfe278998968ba7c5a95344bc8b55>
<https://askubuntu.com/questions/1230525/ubuntu-20-04-network-performance-extremely-slow>
```bash
vim /etc/NetworkManager/conf.d/wifi-powersave-off.conf
```
```conf
[connection]
# Values are 0 (use default), 1 (ignore/don't touch), 2 (disable) or 3 (enable).
wifi.powersave = 2
```
```bash
systemctl restart NetworkManager
```
## Podman
Since you'll be using podman for most container-based services, you'll want to set the

View File

@@ -2,19 +2,24 @@
- [Truenas](#truenas)
- [Bios settings](#bios-settings)
- [Archiving](#archiving)
- [Deleting snapshots](#deleting-snapshots)
- [But First, ZFS on RPi](#but-first-zfs-on-rpi)
- [Pi Setup](#pi-setup)
- [Datasets, Snapshots, and Encryption](#datasets-snapshots-and-encryption)
- [Periodic Snapshot Recommendations](#periodic-snapshot-recommendations)
- [Hourly Snapshots](#hourly-snapshots)
- [Daily Snapshots](#daily-snapshots)
- [Replication Tasks](#replication-tasks)
- [Source](#source)
- [Destination](#destination)
- [Manually Create Named Snapshots](#manually-create-named-snapshots)
- [Migrating encrypted pools](#migrating-encrypted-pools)
- [Migrating Properties](#migrating-properties)
- [Backup Task Settings](#backup-task-settings)
- [Create and Destroy zfs Datasets](#create-and-destroy-zfs-datasets)
- [Create and send snapshots](#create-and-send-snapshots)
- [Cleaning up old snapshots](#cleaning-up-old-snapshots)
- [Creating and restoring snapshots](#creating-and-restoring-snapshots)
- [Filesystem ACLs](#filesystem-acls)
- [ISCSI Backup Volumes](#iscsi-backup-volumes)
- [Create Backup ZVOL](#create-backup-zvol)
- [Create Backup ISCSI Target](#create-backup-iscsi-target)
- [VMs](#vms)
- [Converting zvol to qcow2](#converting-zvol-to-qcow2)
- [Converting qcow2 to zvol](#converting-qcow2-to-zvol)
@@ -37,7 +42,11 @@
- [ZFS Size Data](#zfs-size-data)
- [ZFS Rename](#zfs-rename)
- [ISCSI](#iscsi)
- [Base Name](#base-name)
- [ISCSI Base Name](#iscsi-base-name)
- [Archiving](#archiving)
- [Deleting snapshots](#deleting-snapshots)
- [But First, ZFS on RPi](#but-first-zfs-on-rpi)
- [Pi Setup](#pi-setup)
## Bios settings
@@ -48,120 +57,65 @@ You can check the bios version with `dmidecode -t bios -q`
2. Turn off boosting
3. Enable XMP
## Archiving
1. Create a recursive snapshot called "archive_pool_year_month_day"
2. Create a replication task called "archive_pool_year_month_day"
- select all datasets you want to backup
- fill in enc0/archives/archive-year-month-day_hour-minute
- full filesystem replication
- select "Matching naming schema"
- Use `archive-%Y-%m-%d_%H-%M`
- Deselect run automatically
- Save and run
## Deleting snapshots
Sometimes you need to delete many snapshots from a certain dataset. The UI is terrible for this, so
we need to use `zfs destroy`. xargs is the best way to do this since it allows parallel processing.
```bash
# zfs list snapshots with:
# -o name: only print the name
# -S creation: sort by creation time
# -H: don't display headers
# -r: recurse through every child dataset
zfs list -t snapshot enc0/archives -o name -S creation -H -r
# pipe it through xargs with:
# -n 1: take only 1 argument from the pipe per command
# -P 8: eight parallel processes
# Also pass to zfs destroy:
# -v: verbose
# -n: dryrun
zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v -n
# if that looks good you can remove the "-n"
zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v
```
## But First, ZFS on RPi
A really good backup server is an RPi running openzfs. See [the openzfs docs](https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS%20for%20Raspberry%20Pi.html#step-2-setup-zfs) for more info.
### Pi Setup
Add the vault ssh CA key to your pi.
```bash
curl -o /etc/ssh/trusted-user-ca-keys.pem https://vault.ducoterra.net/v1/ssh-client-signer/public_key
echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" >> /etc/ssh/sshd_config
service ssh restart
```
Create a pi user.
```bash
adduser pi
usermod -a -G sudo pi
```
SSH to the pi as the "pi" user. Delete the ubuntu user.
```bash
killall -u ubuntu
userdel -r ubuntu
```
Disable SSH password authentication
```bash
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
service ssh restart
```
Change the hostname.
```bash
echo pi-nas > /etc/hostname
```
Upgrade and restart the pi.
```bash
apt update && apt upgrade -y && apt autoremove -y
reboot
```
Install ZFS.
```bash
apt install -y pv zfs-initramfs
```
Find the disks you want to use to create your pool
```bash
fdisk -l
```
Create a pool.
```bash
mkdir -p /mnt/backup
zpool create \
-o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/mnt/backup \
backup ${DISK}
```
## Datasets, Snapshots, and Encryption
### Periodic Snapshot Recommendations
#### Hourly Snapshots
- Lifetime: `1 day`
- Naming Schema: `hourly-%Y-%m-%d_%H-%M`
- Schedule: `Hourly`
- Begin: `00:00:00`
- End: `23:59:00`
- Disallow taking empty snapshots
- Enabled
- Recursive
Assuming 100 datasets: 100 datasets x 24 hours = 2400 snapshots
Disallowing empty snapshots will help keep that number down.
#### Daily Snapshots
- Lifetime: `1 week`
- Naming Schema: `daily-%Y-%m-%d_%H-%M`
- Schedule: `Daily`
- Allow taking empty snapshots
- Enabled
- Recursive
Assuming 100 datasets: 100 datasets x 7 days = 700 snapshots
### Replication Tasks
Before configuring, create a dataset that you'll be replicating to.
Use advanced settings.
- Transport `LOCAL`
#### Source
- Recursive
- Include Dataset Properties
- Periodic Snapshot Tasks: Select your `daily` task
- Run automatically
#### Destination
- Read-only Policy: `SET`
- Snapshot Retention Policy: `Custom`
- Lifetime: `1 month`
- Naming Schema: `daily-%Y-%m-%d_%H-%M`
Assuming 100 datasets: 100 datasets x 30 days = 3000 snapshots
#### Manually Create Named Snapshots
1. Datasets -> Select dataset -> Create Snapshot -> Naming Schema (daily)
2. Start replication from Data Protection
### Migrating encrypted pools
Since you can't use `-R` to send encrypted datasets recursively you'll need to use more creative tactics. Here's my recommendation:
@@ -201,16 +155,6 @@ If you need to migrate your dataset comments you can use the following bash to a
for i in $(zfs list -H -d 1 -o name backup/nvme/k3os-private); do read -r name desc < <(zfs list -H -o name,org.freenas:description $i) && pvc=$(echo "$name" | awk -F "/" '{print $NF}') && zfs set org.freenas:description=$desc enc1/k3os-private/$pvc; done
```
### Backup Task Settings
| Key | Value |
| ------------------------------------ | --------------------- |
| Destination Dataset Read-only Policy | SET |
| Recursive | true |
| Snapshot Retention Policy | Same as Source |
| Include Dataset Properties | true |
| Periodic Snapshot Tasks | <daily-snapshot-task> |
### Create and Destroy zfs Datasets
```bash
@@ -269,7 +213,7 @@ zfs list -t snap -r $POOL | wc -l
zfs list -t snap -r -H -o name $POOL > $SNAPSHOTS_FILE
# Check the file
cat $SNAPSHOTS_FILE | less
cat $SNAPSHOTS_FILE | less
# Dry run
for SNAPSHOT in $(cat $SNAPSHOTS_FILE); do echo "zfs destroy -v $SNAPSHOT"; done | less
@@ -303,6 +247,35 @@ Dataset -> Dataset details (edit) -> Advanced Options -> ACL Type (inherit)
setfacl -b -R /mnt/enc0/smb/media
```
## ISCSI Backup Volumes
### Create Backup ZVOL
1. Create a new dataset called "iscsi" and then a dataset under that called "backups"
1. Set sync to always
2. Disable compression
3. Enable Sparse
2. Create a new dataset under backups with the same name as your server hostname
3. Set the size to something reasonable (Note you may need to "force size")
### Create Backup ISCSI Target
1. In System -> Services -> ISCSI set the Base Name following [these rules](#iscsi-base-name)
2. In Shared -> ISCSI -> Authorized Access -> Create a new authorized access
1. Group ID arbitrary - just pick a number you haven't used
2. User: The connecting machine's ISCSI Base Name
3. Secret: A 16 character password with no special characters
3. Wizard -> Create New
1. Extent Name: `backup-<hostname>`
2. Extent Type: `Device`
3. Extent Device: The ZVOL you just created
4. Extent Sharing Platform: `Modern OS`
5. Protocol Options Portal: Either create new (0.0.0.0 and ::) or select your existing portal
6. Protocol Options Initiators: The base name of the connecting machine following [these rules](#iscsi-base-name)
4. Targets -> Select the backup-<hostname> target -> Edit
1. Authentication Method: `CHAP`
2. Authentication Group Number: The group number you created above
## VMs
1. Force UEFI installation
@@ -568,10 +541,123 @@ zfs rename enc0/something enc0/something_else
## ISCSI
### Base Name
### ISCSI Base Name
<https://datatracker.ietf.org/doc/html/rfc3721.html#section-1.1>
| iqn | . | year-month of domain registration | . | reversed domain | : | unique string
iqn.2022-01.com.reeselink:driveripper
## Archiving
1. Create a recursive snapshot called "archive_pool_year_month_day"
2. Create a replication task called "archive_pool_year_month_day"
- select all datasets you want to backup
- fill in enc0/archives/archive-year-month-day_hour-minute
- full filesystem replication
- select "Matching naming schema"
- Use `archive-%Y-%m-%d_%H-%M`
- Deselect run automatically
- Save and run
## Deleting snapshots
Sometimes you need to delete many snapshots from a certain dataset. The UI is terrible for this, so
we need to use `zfs destroy`. xargs is the best way to do this since it allows parallel processing.
```bash
# zfs list snapshots with:
# -o name: only print the name
# -S creation: sort by creation time
# -H: don't display headers
# -r: recurse through every child dataset
zfs list -t snapshot enc0/archives -o name -S creation -H -r
# pipe it through xargs with:
# -n 1: take only 1 argument from the pipe per command
# -P 8: eight parallel processes
# Also pass to zfs destroy:
# -v: verbose
# -n: dryrun
zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v -n
# if that looks good you can remove the "-n"
zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v
```
## But First, ZFS on RPi
A really good backup server is an RPi running openzfs. See [the openzfs docs](https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS%20for%20Raspberry%20Pi.html#step-2-setup-zfs) for more info.
### Pi Setup
Add the vault ssh CA key to your pi.
```bash
curl -o /etc/ssh/trusted-user-ca-keys.pem https://vault.ducoterra.net/v1/ssh-client-signer/public_key
echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" >> /etc/ssh/sshd_config
service ssh restart
```
Create a pi user.
```bash
adduser pi
usermod -a -G sudo pi
```
SSH to the pi as the "pi" user. Delete the ubuntu user.
```bash
killall -u ubuntu
userdel -r ubuntu
```
Disable SSH password authentication
```bash
sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config
service ssh restart
```
Change the hostname.
```bash
echo pi-nas > /etc/hostname
```
Upgrade and restart the pi.
```bash
apt update && apt upgrade -y && apt autoremove -y
reboot
```
Install ZFS.
```bash
apt install -y pv zfs-initramfs
```
Find the disks you want to use to create your pool
```bash
fdisk -l
```
Create a pool.
```bash
mkdir -p /mnt/backup
zpool create \
-o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/mnt/backup \
backup ${DISK}
```

View File

@@ -117,31 +117,7 @@ On the server:
```bash
# Install glances for system monitoring
apt install -y glances net-tools vim
# Install zsh with autocomplete and suggestions
apt install -y zsh zsh-autosuggestions zsh-syntax-highlighting
cat <<EOF > ~/.zshrc
# Basic settings
autoload bashcompinit && bashcompinit
autoload -U compinit; compinit
zstyle ':completion:*' menu select
# Prompt settings
autoload -Uz promptinit
promptinit
prompt redhat
PROMPT_EOL_MARK=
# Syntax Highlighting
source /usr/share/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source /usr/share/zsh-autosuggestions/zsh-autosuggestions.zsh
### Custom Commands and Aliases ###
EOF
chsh -s $(which zsh) && chsh -s $(which zsh) ducoterra
apt install -y glances net-tools vim tmux
# Cockpit
apt install -y cockpit
@@ -163,3 +139,7 @@ To disable this, You can create a blank file with the same name in /etc:
sudo touch /etc/NetworkManager/conf.d/10-globally-managed-devices.conf
sudo systemctl restart NetworkManager
Then `nmcli device set <device-name> managed yes`
Then turn on "Connect Automatically" in Cockpit.