homelab/infrastructure/graduated/arch

Arch Base

This is the base configuration from which you can build a variety of systems. Right now I have instructions for building a:

1. Workstation
2. Gaming PC
3. Kubernetes Server

Table of Contents

• Arch Base
  • Table of Contents
  • Installation
    • Preparation
    • Installation
    • Gnome Keyring
    • Base Tools
    • ZSH
    • Prompt Themes
    • AUR
    • Security
      • Secure Boot
      • TPM2 LUKS Decryption with Secure Boot
        • Re-enroll
      • FIDO2 LUKS Decryption
      • Firewall
      • ClamAV
      • btrbk
        • fstab
        • Snapshots
        • Backups
        • Backing up a snapshot
        • Restoring a snapshot
      • Chroots
  • Hardware Management
    • Power Profiles
    • Color Management
    • Washed out colors with power-profiles-daemon
    • Hardware Acceleration
    • Don't sleep while plugged in
    • Bluetooth
    • Audio
  • Software Stores
    • Flatpak
    • AppImage
    • Overriding .desktop files
      • Flatpak
  • Networking
    • IPv6

Installation

Preparation

Follow most of the instructions here: https://wiki.archlinux.org/title/Installation_guide

1. Download Arch

2. Verify the image

   gpg --auto-key-locate clear,wkd -v --locate-external-key pierre@archlinux.org
   gpg --keyserver-options auto-key-retrieve --verify archlinux-...
   

3. Create a bootable ISO https://wiki.archlinux.org/title/USB_flash_installation_medium

   1. If you are booting into a VM, create an ISO with installation files so you don't have to copy-paste:

      sudo pacman -S cdrtools
      mkisofs -r -iso-level 4 -l -o /tmp/arch-files.iso ./arch
      

   2. If you are booting from a live usb, copy the files in ./arch to the usb drive

4. Disable secureboot (reenable later)

Installation

You'll want two usb drives while following this guide. One will be the Arch boot drive. The other will be a support drive with critical files and passwords which we will need to access after we finish the install.

1. Boot into the live image

2. If you only have wifi, use iwctl to connect https://wiki.archlinux.org/title/Iwd#iwctl

   1. iwctl
   2. device list
   3. adapter wlan0 set-property Powered on <- Note: replace "wlan0" with the name of your device
   4. station wlan0 scan
   5. station wlan0 get-networks
   6. station wlan0 connect SSID

3. Check for network connectivity

   # Check for internet
   ip a
   ping archlinux.org
   

4. timedatectl to update system clock

5. Sync the pacman database with pacman -Sy

6. Install pwgen for password generation pacman -S pwgen

7. If using a VM, mount the iso with arch conf files

   mount --mkdir /dev/sr1 /media
   

8. If using a physical computer, mount your support drive

   mount --mkdir /dev/sdb1 /media
   

9. Create disk partitions. Use gdisk or beware "bootctl install is not on a gpt partition table"

   fdisk -l
   gdisk /dev/vda
   

   1. Delete all existing partitions with d
   2. Create a new partition (partition 1) with n
   3. When prompted for last sector type +1G
   4. When prompted for partition structure, type L and search for EFI SYSTEM, then use that hex code
   5. Create a second new partition (partition 2) with n
   6. Press enter through the remaining options (the defaults are good)

10. mkfs.fat -F 32 /dev/vda1 (/mnt/boot partition)

11. This next step involves generating a secure, random password. We're going to save this to our support drive.

    echo -n $(pwgen 8 5) | sed 's/ /-/g' > /media/root-key.txt

12. cryptsetup luksFormat /dev/vda2 --key-file /path/to/root-key.txt

13. cryptsetup luksOpen /dev/vda2 root --key-file /path/to/root-key.txt

14. mkfs.btrfs /dev/mapper/root (root partition)

15. At this point you can choose how to subvolume your root partition

    mount --mkdir -o subvolid=5 /btr_pool
    btrfs sub create root /btr_pool
    btrfs sub create home /btr_pool
    

16. Mount the root partition with mount -o subvol=root /dev/mapper/root /mnt

17. Mount the home partition with mount -o subvol=home /dev/mapper/root /mnt/home

18. Mount the boot partition with mount --mkdir /dev/vda1 /mnt/boot

19. pacstrap -K /mnt base linux linux-firmware

    This command might show an error. This is ok, we'll fix it later.

20. genfstab -U /mnt >> /mnt/etc/fstab

21. If on VM: Mount the conf files with mount --mkdir /dev/sr1 /mnt/media

22. If on a physical computer: mount the support parition with mount --mkdir /dev/sdb1 /mnt/media

23. arch-chroot /mnt

24. ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime

25. hwclock --systohc

26. echo 'en_US.UTF-8 UTF-8' > /etc/locale.gen

27. echo 'KEYMAP=us' > /etc/vconsole.conf

28. echo 'hostname' > /etc/hostname

29. pacman -S sudo vim dhclient dhcpcd bash-completion btrfs-progs plymouth fwupd

    • dhclient/dhcpcd provides dhcp for network
    • bash-completion provides tab complete
    • btrfs-progs provides fsck for btrfs
    • plymouth gives a nice bootloader screen

30. Edit /etc/mkinitcpio.conf and uncomment the line for systemd-boot with an encrypted drive.

31. mkinitcpio -P

32. Install systemd-boot

    https://wiki.archlinux.org/title/systemd-boot

    bootctl install
    

    If this raises an error like "efi partition not found" you probably forgot to format /mnt/boot as an EFI partition. Edit this by reformatting it with gdisk (ef00 is the hex code).

33. edit your loader.conf with some defaults

    /boot/loader/loader.conf

    default  arch.conf
    timeout  4
    console-mode max
    editor   no
    

34. Create a loader (/usr/share/systemd/bootctl/arch.conf for example)

    /boot/loader/entries/arch.conf

    title   Arch Linux
    linux   /vmlinuz-linux
    initrd  /initramfs-linux.img
    options ... rd.luks.name=d9828faa-2b8c-4184-9e74-9054ae328c6d=root root=/dev/mapper/root rootflags=subvol=root ...
    

    You can get the UUID of the disk into arch.conf with some grepping. Use vim to cut the excess and copy it into the correct location.

    blkid | grep /dev/vda2 >> /boot/loader/entries/arch.conf
    

35. useradd ducoterra

36. passwd ducoterra

37. groupadd sudo

38. Edit /etc/sudoers and uncomment the section allowing sudo and wheel group privilege

39. usermod -aG sudo ducoterra

40. usermod -aG wheel ducoterra

41. mkdir /home/ducoterra

42. chown ducoterra:ducoterra /home/ducoterra

43. locale-gen

44. systemctl enable dhcpcd

45. If on VM install guest drivers: pacman -S qemu-guest-agent spice-vdagent

46. If you need ssh: pacman -S openssh; systemctl enable sshd

47. Add a pacman hook for systemd-boot updates

    /etc/pacman.d/hooks/95-systemd-boot.hook

    [Trigger]
    Type = Package
    Operation = Upgrade
    Target = systemd
    
    [Action]
    Description = Gracefully upgrading systemd-boot...
    When = PostTransaction
    Exec = /usr/bin/systemctl restart systemd-boot-update.service
    

48. Install gnome: pacman -S gdm gnome

    • choose pipewire-jack
    • choose wireplumber
    • choose noto-fonts-emoji

49. systemctl enable gdm

50. Install NetworkManager pacman -S networkmanager

51. systemctl enable NetworkManager

52. Install gnome nice-to-haves pacman -S gnome-tweaks dconf-editor seahorse

53. Install tpm2-tss for tpm2 disk decryption pacman -S tpm2-tss

54. Setup tpm2 disk decryption

    systemd-cryptenroll /dev/vda2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs="" --unlock-key-file=/media/root-key.txt
    

55. exit

56. reboot

Gnome Keyring

Don't set a password for single-user systems. We're using full-disk encryption. This will let you login with just a fingerprint.

1. Install seahorse if you haven't already
2. Open the Passwords and Keys apps
3. Create a new Password keyring called "Login"
4. Do not enter a password
5. Set it as default

Base Tools

# gvfs and gvfs-dnssd are for webdav support
pacman -S rsync which git iperf3 pwgen dosfstools exfatprogs gvfs gvfs-dnssd wget man-db net-tools

ZSH

pacman -S zsh grml-zsh-config zsh-syntax-highlighting zsh-autosuggestions pkgfile
chsh -s $(which zsh)

cat <<EOF > ~/.zshrc
# Basic settings
autoload bashcompinit && bashcompinit
autoload -U compinit; compinit
zstyle ':completion:*' menu select

# Prompt settings
autoload -Uz promptinit
promptinit
PROMPT_EOL_MARK=

# Syntax Highlighting
source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh

# Command Not Found Autocomplete
source /usr/share/doc/pkgfile/command-not-found.zsh

### Custom Commands and Aliases ###
EOF

Prompt Themes

See: https://wiki.archlinux.org/title/Zsh#Prompt_themes

Use prompt -l to list prompts

Use prompt -p to see previews

In your .zshrc set the following:

autoload -Uz promptinit
promptinit
prompt grml

AUR

The AUR lets you install community-created and maintained packages. Here are the basics:

pacman -S --needed git base-devel
mkdir ~/AUR

# When you find a project, the basic installation looks like this:
git clone <git repo from aur>
cd <folder name>

PAUSE!

Read through the PKGBUILD. Specifically check:

1. depends=: Do these make sense? Is there anything you wouldn't install?
2. source=: This should be a link to an official repository. If it links to https://joes-back-door.git don't install it.
3. sha256sums=: Make sure this matches the package! If you have to - wget https://link-to-deb and sha256sum something.deb
4. package() {: This is the script for installing the package. Does everything look right? Is there something going on you don't understand? Are we curling or wgeting things we shouldn't be?
5. Also look through the remaining fields and check for oddities. Remember, anyone can slip anything into these PKGBUILD files, but if the info looks kosher you're good to go. Basically there won't be any surprises since the PKGBUILD defines everything related to the install/build process.

If you've already installed a few packages and keep them in a folder, you can run a quick and dirty check with:

# Find all PKGBUILD files in the current tree and grep for `https` so we can find potentially unwanted web downloads.
find . -name "PKGBUILD" -exec grep --color -Hni "https://" {} \;

makepkg -si

We can update our AUR packages with a script. As long as you clone your AUR packages into ~/AUR this will work:

~./local/scripts/update-aur.sh

#!/bin/bash

for file in $(ls /home/ducoterra/AUR);
do
    cd /home/ducoterra/AUR/$file
    git pull
    makepkg -si
done

Security

https://wiki.archlinux.org/title/security

Every machine, regardless of use-case, should perform some basic hardening. You don't need to follow every instruction in the above wiki, but you should at least enable secure boot, tpm2 disk decryption, firewall, apparmor, clamav, btrfs snapshots, and btrfs backups.

Security Philosophy

1. Secure Boot

   Protection from pre-boot malware that might hijack your EFI binary.

   https://www.rodsbooks.com/efi-bootloaders/secureboot.html

2. TPM2 Decryption

   Since we have secure boot enabled we can safely auto-decrypt our hard drive with a tpm2 device. This is purely a convenience.

3. Firewall

   This should be self-explanatory, but I'll explain anyway. Don't allow any arbitrary network traffic into your device. Block those ports. Only open what you need. Firewalls drastically reduce the risk of remote exploits by stopping them before they can even establish a connection. Firewalls can also be used to limit an attacker's ability to even discover you on a network with icmp blocking.

4. ClamAV

   Much like Windows has Windows Defender, Linux has ClamAV. Running an antivirus scanner certainly isn't the end-all-be-all of security, and it definitely isn't good enough on its own to keep your system safe, but in combination with apparmor and a firewall you can identify and quarantine malware before it has a chance to compromise your system. That being said, finding any malware on a system is reason enough to nuke it from orbit and restore from a known good backup.

5. BTRFS Snapshots

   This is not a backup, this is a snapshot. It serves an equally important function, however, in that it protects you from accidental deletion and corruption. Let's imagine you perform an update, reboot, and your computer crashes mid-startup. You could easily restore root from a btrfs snapshot on your system and go on with your day like nothing happened.

6. BTRFS Backups

   This is a backup. Unlike snapshots, which live on the same drive your system exists on, backups are physically separate copies of your computer stored (hopefully) in a physically separate location. In the event your computer is lost or stolen these backups give you a way to perfectly restore your system to its former glory.

Secure Boot

1. Put your machine in setup mode

   On framework this is done in the UEFI setup page for Security, sub-page Secure Boot, choose “Erase all Secure Boot Settings.”

   On my Gigabyte motherboard this is done in the BIOS under security. Set secure boot to custom.

2. pacman -S efitools sbctl

3. cd /btr_pools/root/support/

4. for var in PK KEK db dbx ; do efi-readvar -v $var -o old_${var}.esl ; done

5. sbctl create-keys

6. sbctl enroll-keys -m

7. sbctl status

8. sbctl verify

9. sbctl sign -s /boot/vmlinuz-linux

10. sbctl sign -s /boot/EFI/BOOT/BOOTX64.EFI

11. sbctl sign -s /boot/EFI/systemd/systemd-bootx64.efi

12. sbctl verify

13. reboot

14. Enable secure boot

15. sbctl status to check secure boot

16. bootctl to check boot loader status

There is a pacman hook which will automatically sign new binaries on update.

TPM2 LUKS Decryption with Secure Boot

You can optionally allow tpm2 decryption only while secure boot is active.

Using --tpm2-pcrs=7 enforces secure boot and will require password if secure boot is disabled.

1. pacman -S tpm2-tss
2. systemd-cryptenroll /dev/vda2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt

Re-enroll

systemd-cryptenroll /dev/nvme0n1p2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt

FIDO2 LUKS Decryption

1. pacman -S libfido2

Firewall

pacman -S ufw
systemctl enable --now ufw

ClamAV

1. pacman -S clamav
2. clamscan --recursive --infected /path/to/dir

• OR -

1. touch /var/log/clamav/freshclam.log
2. systemctl enable --now clamav-freshclam.service
3. systemctl enable --now clamav-daemon.service
4. clamdscan --multiscan --fdpass /home/ducoterra

btrbk

sudo pacman -S mbuffer # for progress monitoring
cd Downloads
wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
clamdscan .
chmod +x btrbk
sudo mv btrbk /usr/bin/

fstab

You'll need to mount your btrfs volumes in a location which exposes their subvolumes.

mkdir -p /btr_pools/root

/etc/fstab

# btr_pools
UUID=84153269-f194-43f7-a4fe-e72aaffdb97a       /btr_pools/root               btrfs           rw,relatime,ssd,space_cache=v2,subvolid=5  0 0

systemctl daemon-reload
mount -a
btrfs sub create /btr_pools/root/.snapshots

Snapshots

mkdir /etc/btrbk

Create a snapshot config

/etc/btrbk/snapshots.conf

snapshot_preserve_min   24h
snapshot_preserve       24h

# root
volume /btr_pools/root
    subvolume           root
    snapshot_dir        .snapshots

# home
volume /btr_pools/root
    subvolume           home
    snapshot_dir        .snapshots

Then create a snapshot service

/etc/systemd/system/btrbk_snapshots.service

[Unit]
Description=Runs btrbk with config file at /etc/btrbk/snapshots.conf

[Service]
ExecStart=/usr/bin/btrbk -c /etc/btrbk/snapshots.conf --progress run

Then create a timer for the service

/etc/systemd/system/btrbk_snapshots.timer

[Unit]
Description=Run snapshots every hour

[Timer]
OnCalendar=hourly

AccuracySec=10min
Persistent=true
Unit=btrbk_snapshots.service

[Install]
WantedBy=timers.target

Then enable the service

systemctl enable --now btrbk_snapshots.timer

Backups

Before you begin, go through the usual process of setting up an encrypted drive. If you're using Gnome I recommend using the GUI since it handles encrypted USB drives really nicely.

First, I'd recommend manually creating the mountpoint and setting it as a read-only. This prevents backups from being written to the root device when the backup disk isn't mounted.

btrfs sub create /btr_pools/backup
btrfs property set /btr_pools/backup ro true

Second, I'd recommend creating subvolumes within your existing volumes for things you don't want backed up. These include:

1. /var/lib/libvirt
2. Nextcloud

Third, I'd recommend iterating dot directories you'd need to restore and writing them down somewhere:

1. .aws
2. .cache
3. .config
4. .gitconfig
5. .icons
6. .kube
7. .local
8. .minecraft
9. .mozilla
10. .ssh
11. .steam
12. .vimrc
13. .wireguard
14. .zshrc

Now set up the backup:

1. Create a backup config

   /etc/btrbk/backups.conf

   snapshot_create         no
   target_preserve_min     no
   target_preserve         24h
   
   # root
   volume /btr_pools/root
       target /btr_pools/backup
       subvolume           root
       snapshot_dir        .snapshots
   
   # home
   volume /btr_pools/root
       target /btr_pools/backup
       subvolume           home
       snapshot_dir        .snapshots
   

2. Create a backup service

   /etc/systemd/system/btrbk_backups.service

   [Unit]
   Description=Runs btrbk with config file at /etc/btrbk/backups.conf
   
   [Service]
   ExecStart=/usr/bin/btrbk -c /etc/btrbk/backups.conf --progress run
   

3. Create a timer to activate the service

   /etc/systemd/system/btrbk_backups.timer

   [Unit]
   Description=Run btrbk backups every hour
   
   [Timer]
   OnCalendar=hourly
   AccuracySec=10min
   Persistent=true
   Unit=btrbk_backups.service
   
   [Install]
   WantedBy=timers.target
   

4. Enable the timer

   systemctl enable --now btrbk_backup.conf
   

Backing up a snapshot

pacman -S pv

btrfs send /mnt/btr_backup/root.20230727T1000 | pv | btrfs receive /mnt/btr_iscsi

Restoring a snapshot

export ROOT_SNAPSHOT_NAME=<name>
export HOME_SNAPSHOT_NAME=<name>
rsync -av --delete /btr_pools/root/.snapshots/${ROOT_SNAPSHOT_NAME}/ /btr_pools/root/root/
rsync -av --delete /btr_pools/root/.snapshots/${HOME_SNAPSHOT_NAME}/ /btr_pools/root/home/
pacman -Syu # This is only required if you're restoring a snapshot from an old kernel
reboot

Chroots

You can create chroot environments to run firejails or just use for testing purposes.

1. cd /btr_pools/root
2. btrfs sub create chroots
3. mkdir /btr_pools/root/chroots/testing
4. pacman -S arch-install-scripts
5. pacstrap -K /btr_pools/root/chroots/testing base base-devel
6. arch-chroot /btr_pools/root/chroots/testing

Hardware Management

Power Profiles

https://wiki.archlinux.org/title/CPU_frequency_scaling#power-profiles-daemon

pacman -S power-profiles-daemon
systemctl enable --now power-profiles-daemon

Color Management

https://wiki.archlinux.org/title/Framework_Laptop_13#Display

https://wiki.archlinux.org/title/ICC_profiles#Wayland

cp /home/ducoterra/Downloads/BOE_CQ... /usr/share/color/icc/colord/
colormgr get-profiles
colormgr get-devices
colormgr device-add-profile xrandr-BOE-0x095f-0x00000000 icc-eca2e6d155d550a5e78c97a34ac3fcae

Washed out colors with power-profiles-daemon

https://wiki.archlinux.org/title/Framework_Laptop_13#(AMD)_Washed-out_colors_when_using_power-profiles-daemon_in_power-saver_or_balanced_mode

systemctl edit power-profiles-daemon.service --drop-in=disable_panel_powersavings

[Service]
ExecStart=
ExecStart=/usr/lib/power-profiles-daemon --block-action=amdgpu_panel_power

Hardware Acceleration

(This helps enable hardware encoding/decoding for steam streaming)

Intel

pacman -S libva-utils intel-media-driver
vainfo

AMD

pacman -S vulkan-radeon libva-utils libva-mesa-driver xf86-video-amdgpu
vainfo

Don't sleep while plugged in

This is needed for the Framework 13 (11th gen) since sleeping while plugged in to a dock will prevent it from waking up.

vim /etc/systemd/logind.conf

Bluetooth

1. pacman -S bluez bluez-utils
2. systemctl enable --now bluetooth

Audio

Without pipewire-pulse the audio level/device will reset every reboot.

1. pacman -S pipewire-pulse (remove conflicting packages)

Software Stores

Flatpak

pacman -S flatpak

AppImage

Install fuse for appimage support.

sudo pacman -S fuse

Make sure to chmod +x the .appimage file before running.

1. cp ~/Downloads/xxxxxxx.appimage ~/Applications

2. Find an icon online and save it to ~/.icons

3. Write a .desktop entry at ~/.local/share/applications/

   [Desktop Entry]
   Name=
   Exec=/home/ducoterra/Applications/
   Icon=/home/ducoterra/.icons/
   Type=Application
   

4. desktop-file-validate ~/.local/share/applications/*.desktop

5. update-desktop-database

Overriding .desktop files

Sometimes you need to make changes to the default .desktop files. You can do this by copying the .desktop file to ~/.local/share/applications/ and editing its settings.

Finding the .desktop file can be tricky. I usually use the following find command:

# 2>/dev/null pipes all the "permission denied" errors to /dev/null
# Searching in the btr_pools/root/root will skip .snapshots, /home, and /proc, /sys, etc.
find /btr_pools/root/root/ -name '*.desktop' 2>/dev/null

Flatpak

cp /var/lib/flatpak/exports/share/applications/... ~/.local/share/applications/

Networking

IPv6

Set address generation type to eui64 with nmcli:

nmcli connection modify Wired\ connection\ 1 ipv6.ip6-privacy 0
nmcli connection modify Wired\ connection\ 1 ipv6.addr-gen-mode eui64
systemctl restart NetworkManager