homelab/infrastructure/graduated/arch

Arch Base

This is the base configuration from which you can build a variety of systems. Right now I have instructions for building a:

1. Workstation
2. Gaming PC
3. Kubernetes Server

Table of Contents

• Arch Base
  • Table of Contents
  • Installation
    • Preparation
    • Installation
    • Gnome Keyring
    • Base Tools
    • ZSH
    • Prompt Themes
    • AUR
    • Security
      • Secure Boot
      • TPM2 LUKS Decryption with Secure Boot
        • Re-enroll
      • FIDO2 LUKS Decryption
      • Firewall
      • ClamAV
      • btrbk
        • fstab
        • Snapshots
        • Backups
        • Backing up a snapshot
      • Chroots
      • Fingerprint Reader Support
        • Setup
        • Turn Off Fingerprint When Laptop Lid Closed
  • Hardware Management
    • Power Profiles
    • Color Management
    • Washed out colors with power-profiles-daemon
    • Hardware Acceleration
    • Don't sleep while plugged in
    • Bluetooth
    • Audio
  • Software Stores
    • AppImage Support
      • Troubleshooting
    • Flatpak
  • Apps
    • Firefox
      • Gnome Extensions
    • Avahi (Bonjour)
    • CUPS Printing

Installation

Preparation

Follow most of the instructions here: https://wiki.archlinux.org/title/Installation_guide

1. Download Arch

2. Verify the image

   gpg --auto-key-locate clear,wkd -v --locate-external-key pierre@archlinux.org
   gpg --keyserver-options auto-key-retrieve --verify archlinux-...
   

3. Create a bootable ISO

   1. If you are booting into a VM, create an ISO with installation files so you don't have to copy-paste:

      sudo pacman -S cdrtools
      mkisofs -r -iso-level 4 -l -o /tmp/arch-files.iso ./arch
      

   2. If you are booting from a live usb, copy the files in ./arch to the usb drive

4. Disable secureboot (reenable later)

Installation

You'll want two usb drives while following this guide. One will be the Arch boot drive. The other will be a support drive with critical files and passwords which we will need to access after we finish the install.

1. Boot into the live image

2. Check for network connectivity

   # Check for internet
   ip a
   ping archlinux.org
   

3. timedatectl to update system clock

4. Install pwgen for password generation pacman -S pwgen

5. If using a VM, mount the iso with arch conf files

   mount --mkdir /dev/sr1 /media
   

6. If using a physical computer, mount your support drive

   mount --mkdir /dev/sdb1 /media
   

7. Create disk partitions. Use gdisk or beware "bootctl install is not on a gpt partition table"

   fdisk -l
   gdisk /dev/vda
   

   • +1G for /boot
   • t EFI SYSTEM for /boot
   • remaining for /

8. mkfs.fat -F 32 /dev/vda1 (/mnt/boot partition)

9. This next step involves generating a secure, random password. Make sure to save this somewhere. I recommend having an encrypted partition on your installation drive to which you can write a few bytes of text.

   echo -n $(pwgen 8 5) | sed 's/ /-/g' > /media/root-key.txt

10. cryptsetup luksFormat /dev/vda2 --key-file /path/to/root-key.txt

11. cryptsetup luksOpen /dev/vda2 root --key-file /path/to/root-key.txt

12. mkfs.btrfs /dev/mapper/root (root partition)

13. At this point you can choose how to subvolume your root partition

    mount --mkdir -o subvolid=5 /btr_pool
    btrfs sub create root /btr_pool
    btrfs sub create home /btr_pool
    

14. Mount the root partition with mount -o subvol=root /dev/mapper/root /mnt

15. Mount the home partition with mount -o subvol=home /dev/mapper/root /mnt/home

16. Mount the boot partition with mount --mkdir /dev/vda1 /mnt/boot

17. pacstrap -K /mnt base linux linux-firmware

    This command might show an error. This is ok, we'll fix it later.

18. genfstab -U /mnt >> /mnt/etc/fstab

19. If on VM: Mount the conf files with mount --mkdir /dev/sr1 /mnt/media

20. If on a physical computer: mount the support parition with mount --mkdir /dev/sdb1 /mnt/media

21. arch-chroot /mnt

22. ln -sf /usr/share/zoneinfo/America/New_York /etc/localtime

23. hwclock --systohc

24. echo 'en_US.UTF-8 UTF-8' > /etc/locale.gen

25. echo 'KEYMAP=us' > /etc/vconsole.conf

26. echo 'hostname' > /etc/hostname

27. pacman -S sudo vim dhclient dhcpcd bash-completion btrfs-progs plymouth

    • dhclient/dhcpcd provides dhcp for network
    • bash-completion provides tab complete
    • btrfs-progs provides fsck for btrfs
    • plymouth gives a nice bootloader screen

28. Edit /etc/mkinitcpio.conf and uncomment the line for systemd-boot with an encrypted drive.

29. mkinitcpio -P

30. Install systemd-boot

    https://wiki.archlinux.org/title/systemd-boot

    bootctl install
    

    If this raises an error like "efi partition not found" you probably forgot to format /mnt/boot as an EFI partition. Edit this by reformatting it with gdisk (ef00 is the hex code).

31. edit your loader.conf with some defaults

    /boot/loader/loader.conf

    default  arch.conf
    timeout  4
    console-mode max
    editor   no
    

32. Create a loader (/usr/share/systemd/bootctl/arch.conf for example)

    /boot/loader/entries/arch.conf

    title   Arch Linux
    linux   /vmlinuz-linux
    initrd  /initramfs-linux.img
    options ... rd.luks.name=d9828faa-2b8c-4184-9e74-9054ae328c6d=root root=/dev/mapper/root rootflags=subvol=root ...
    

    You can get the UUID of the disk into arch.conf with some grepping. Use vim to cut the excess and copy it into the correct location.

    blkid | grep /dev/vda2 >> /boot/loader/entries/arch.conf
    

33. useradd ducoterra

34. passwd ducoterra

35. groupadd sudo

36. Edit /etc/sudoers and uncomment the section allowing sudo and wheel group privilege

37. usermod -aG sudo ducoterra

38. usermod -aG wheel ducoterra

39. mkdir /home/ducoterra

40. chown ducoterra:ducoterra /home/ducoterra

41. locale-gen

42. systemctl enable dhcpcd

43. If on VM install guest drivers: pacman -S qemu-guest-agent spice-vdagent

44. If you need ssh: pacman -S openssh; systemctl enable sshd

45. Add a pacman hook for systemd-boot updates

    /etc/pacman.d/hooks/95-systemd-boot.hook

    [Trigger]
    Type = Package
    Operation = Upgrade
    Target = systemd
    
    [Action]
    Description = Gracefully upgrading systemd-boot...
    When = PostTransaction
    Exec = /usr/bin/systemctl restart systemd-boot-update.service
    

46. Install gnome: pacman -S gdm gnome

    • choose pipewire-jack
    • choose wireplumber
    • choose noto-fonts-emoji

47. systemctl enable gdm

48. Install NetworkManager pacman -S networkmanager

49. systemctl enable NetworkManager

50. Install gnome nice-to-haves pacman -S gnome-tweaks dconf-editor seahorse

51. Install tpm2-tss for tpm2 disk decryption pacman -S tpm2-tss

52. Setup tpm2 disk decryption

    systemd-cryptenroll /dev/vda2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs="" --unlock-key-file=/media/root-key.txt
    

53. exit

54. reboot

Gnome Keyring

Don't set a password for single-user systems. We're using full-disk encryption. This will let you login with just a fingerprint.

1. Install seahorse if you haven't already
2. Open the Passwords and Keys apps
3. Create a new Password keyring called "Login"
4. Do not enter a password
5. Set it as default

Base Tools

# gvfs and gvfs-dnssd are for webdav support
pacman -S rsync which git iperf3 pwgen dosfstools exfatprogs gvfs gvfs-dnssd wget man-db net-tools

ZSH

pacman -S zsh grml-zsh-config zsh-syntax-highlighting zsh-autosuggestions pkgfile
chsh -s $(which zsh)

cat <<EOF > ~/.zshrc
# Basic settings
autoload bashcompinit && bashcompinit
autoload -U compinit; compinit
zstyle ':completion:*' menu select

# Prompt settings
autoload -Uz promptinit
promptinit
PROMPT_EOL_MARK=

# Syntax Highlighting
source /usr/share/zsh/plugins/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh
source /usr/share/zsh/plugins/zsh-autosuggestions/zsh-autosuggestions.zsh

# Command Not Found Autocomplete
source /usr/share/doc/pkgfile/command-not-found.zsh

### Custom Commands and Aliases ###
EOF

Prompt Themes

See: https://wiki.archlinux.org/title/Zsh#Prompt_themes

Use prompt -l to list prompts

Use prompt -p to see previews

In your .zshrc set the following:

autoload -Uz promptinit
promptinit
prompt grml

AUR

The AUR lets you install community-created and maintained packages. Here are the basics:

pacman -S --needed git base-devel
mkdir ~/AUR

# When you find a project, the basic installation looks like this:
git clone <git repo from aur>
cd <folder name>
makepkg -si

Security

https://wiki.archlinux.org/title/security

Every machine, regardless of use-case, should perform some basic hardening. You don't need to follow every instruction in the above wiki, but you should at least enable secure boot, tpm2 disk decryption, firewall, apparmor, clamav, btrfs snapshots, and btrfs backups.

Security Philosophy

1. Secure Boot

   Protection from pre-boot malware that might hijack your EFI binary.

   https://www.rodsbooks.com/efi-bootloaders/secureboot.html

2. TPM2 Decryption

   Since we have secure boot enabled we can safely auto-decrypt our hard drive with a tpm2 device. This is purely a convenience.

3. Firewall

   This should be self-explanatory, but I'll explain anyway. Don't allow any arbitrary network traffic into your device. Block those ports. Only open what you need. Firewalls drastically reduce the risk of remote exploits by stopping them before they can even establish a connection. Firewalls can also be used to limit an attacker's ability to even discover you on a network with icmp blocking.

4. ClamAV

   Much like Windows has Windows Defender, Linux has ClamAV. Running an antivirus scanner certainly isn't the end-all-be-all of security, and it definitely isn't good enough on its own to keep your system safe, but in combination with apparmor and a firewall you can identify and quarantine malware before it has a chance to compromise your system. That being said, finding any malware on a system is reason enough to nuke it from orbit and restore from a known good backup.

5. BTRFS Snapshots

   This is not a backup, this is a snapshot. It serves an equally important function, however, in that it protects you from accidental deletion and corruption. Let's imagine you perform an update, reboot, and your computer crashes mid-startup. You could easily restore root from a btrfs snapshot on your system and go on with your day like nothing happened.

6. BTRFS Backups

   This is a backup. Unlike snapshots, which live on the same drive your system exists on, backups are physically separate copies of your computer stored (hopefully) in a physically separate location. In the event your computer is lost or stolen these backups give you a way to perfectly restore your system to its former glory.

Secure Boot

1. Put your machine in setup mode

   On framework this is done in the UEFI setup page for Security, sub-page Secure Boot, choose “Erase all Secure Boot Settings.”

   On my Gigabyte motherboard this is done in the BIOS under security. Set secure boot to custom.

2. pacman -S efitools sbctl

3. cd /btr_pools/root/support/

4. for var in PK KEK db dbx ; do efi-readvar -v $var -o old_${var}.esl ; done

5. sbctl create-keys

6. sbctl enroll-keys -m

7. sbctl status

8. sbctl verify

9. sbctl sign -s /boot/vmlinuz-linux

10. sbctl sign -s /boot/EFI/BOOT/BOOTX64.EFI

11. sbctl sign -s /boot/EFI/systemd/systemd-bootx64.efi

12. sbctl verify

13. reboot

14. Enable secure boot

15. sbctl status to check secure boot

16. bootctl to check boot loader status

There is a pacman hook which will automatically sign new binaries on update.

TPM2 LUKS Decryption with Secure Boot

You can optionally allow tpm2 decryption only while secure boot is active.

Using --tpm2-pcrs=7 enforces secure boot and will require password if secure boot is disabled.

1. pacman -S tpm2-tss
2. systemd-cryptenroll /dev/vda2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt

Re-enroll

systemd-cryptenroll /dev/nvme0n1p2 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt
systemd-cryptenroll /dev/nvme0n1p3 --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --unlock-key-file=/btr_pools/root/support/root-key.txt

FIDO2 LUKS Decryption

1. pacman -S libfido2

Firewall

pacman -S ufw
systemctl enable --now ufw

ClamAV

1. pacman -S clamav
2. clamscan --recursive --infected /path/to/dir

• OR -

1. touch /var/log/clamav/freshclam.log
2. systemctl enable --now clamav-freshclam.service
3. systemctl enable --now clamav-daemon.service
4. clamdscan --multiscan --fdpass /home/ducoterra

btrbk

cd Downloads
wget https://raw.githubusercontent.com/digint/btrbk/master/btrbk
clamdscan .
chmod +x btrbk
sudo mv btrbk /usr/bin/

fstab

You'll need to mount your btrfs volumes in a location which exposes their subvolumes.

mkdir -p /btr_pools/root

/etc/fstab

# btr_pools
UUID=84153269-f194-43f7-a4fe-e72aaffdb97a       /btr_pools/root               btrfs           rw,relatime,ssd,space_cache=v2,subvolid=5  0 0

systemctl daemon-reload
mount -a
btrfs sub create /btr_pools/root/.snapshots

Snapshots

mkdir /etc/btrbk

Create a snapshot config

/etc/btrbk/snapshots.conf

snapshot_preserve_min   24h
snapshot_preserve       24h

# root
volume /btr_pools/root
    subvolume           root
    snapshot_dir        .snapshots

# home
volume /btr_pools/root
    subvolume           home
    snapshot_dir        .snapshots

Then create a snapshot service

/etc/systemd/system/btrbk_snapshots.service

[Unit]
Description=Runs btrbk with config file at /etc/btrbk/snapshots.conf

[Service]
ExecStart=/usr/bin/btrbk -c /etc/btrbk/snapshots.conf -v run

Then create a timer for the service

/etc/systemd/system/btrbk_snapshots.timer

[Unit]
Description=Run snapshots every hour

[Timer]
OnCalendar=hourly

AccuracySec=10min
Persistent=true
Unit=btrbk_snapshots.service

[Install]
WantedBy=timers.target

Then enable the service

systemctl enable --now btrbk_snapshots.timer

Backups

Before you begin, go through the usual process of setting up an encrypted drive. If you're using Gnome I recommend using the GUI since it handles encrypted USB drives really nicely.

First, I'd recommend manually creating the mountpoint and setting it as a read-only. This prevents backups from being written to the root device when the backup disk isn't mounted.

btrfs sub create /btr_pools/backup
btrfs property set /btr_pools/backup ro true

Second, I'd recommend creating subvolumes within your existing volumes for things you don't want backed up. These include:

1. /var/lib/libvirt
2. Nextcloud

Third, I'd recommend iterating dot directories you'd need to restore and writing them down somewhere:

1. .aws
2. .cache
3. .config
4. .gitconfig
5. .icons
6. .kube
7. .local
8. .minecraft
9. .mozilla
10. .ssh
11. .steam
12. .vimrc
13. .wireguard
14. .zshrc

Now set up the backup:

1. Create a backup config

   /etc/btrbk/backups.conf

   snapshot_create         no
   target_preserve_min     no
   target_preserve         24h
   
   # root
   volume /btr_pools/root
       target /btr_pools/backup
       subvolume           root
       snapshot_dir        .snapshots
   
   # home
   volume /btr_pools/root
       target /btr_pools/backup
       subvolume           home
       snapshot_dir        .snapshots
   

2. Create a backup service

   /etc/systemd/system/btrbk_backups.service

   [Unit]
   Description=Runs btrbk with config file at /etc/btrbk/backups.conf
   
   [Service]
   ExecStart=/usr/bin/btrbk -c /etc/btrbk/backups.conf -v run
   

3. Create a timer to activate the service

   /etc/systemd/system/btrbk_backups.timer

   [Unit]
   Description=Run btrbk backups every hour
   
   [Timer]
   OnCalendar=hourly
   AccuracySec=10min
   Persistent=true
   Unit=btrbk_backups.service
   
   [Install]
   WantedBy=timers.target
   

4. Enable the timer

   systemctl enable --now btrbk_backup.conf
   

Backing up a snapshot

pacman -S pv

btrfs send /mnt/btr_backup/root.20230727T1000 | pv | btrfs receive /mnt/btr_iscsi

Chroots

You can create chroot environments to run firejails or just use for testing purposes.

1. cd /btr_pools/root
2. btrfs sub create chroots
3. mkdir /btr_pools/root/chroots/testing
4. pacman -S arch-install-scripts
5. pacstrap -K /btr_pools/root/chroots/testing base base-devel
6. arch-chroot /btr_pools/root/chroots/testing

Fingerprint Reader Support

Setup

1. pacman -S fprintd
2. systemctl enable --now fprintd
3. fprintd-enroll ducoterra
4. Install https://aur.archlinux.org/pam-fprint-grosshack.git to use fingerprint with gnome

In order to use fingerprint auth with gnome for privileged system stuff with gdm, edit /etc/pam.d/system-auth to include auth sufficient pam_fprintd_grosshack.so.

#%PAM-1.0

auth       required                    pam_shells.so # User must have shell in /etc/shells
auth       requisite                   pam_nologin.so # Prevents users from loging in if /etc/nologin exists
auth       required                    pam_faillock.so      preauth # Timeout after certain number of fails
# Optionally use requisite above if you do not want to prompt for the password
# on locked accounts.
auth       sufficient                  pam_fprintd_grosshack.so
-auth      [success=2 default=ignore]  pam_systemd_home.so
auth       [success=1 default=bad]     pam_unix.so          try_first_pass nullok
auth       [default=die]               pam_faillock.so      authfail
auth       optional                    pam_permit.so
auth       required                    pam_env.so
auth       required                    pam_faillock.so      authsucc
# If you drop the above call to pam_faillock.so the lock will be done also
# on non-consecutive authentication failures.

-account   [success=1 default=ignore]  pam_systemd_home.so
account    required                    pam_unix.so
account    optional                    pam_permit.so
account    required                    pam_time.so

-password  [success=1 default=ignore]  pam_systemd_home.so
password   required                    pam_unix.so          try_first_pass nullok shadow
password   optional                    pam_permit.so

-session   optional                    pam_systemd_home.so
session    required                    pam_limits.so
session    required                    pam_unix.so
session    optional                    pam_permit.so

Turn Off Fingerprint When Laptop Lid Closed

NOTE: This may break fingerprint unlock. Testing in progress.

To disable fingerprint authentication when the laptop lid is closed, and re-enable when it is reopened, we will use acpid to bind to the button/lid.* event to a custom script that will comment out fprintd auth in /etc/pam.d/sudo.

Usually we'd just systemctl mask fprintd but this breaks gdm (as of 08/06/23). See https://gitlab.gnome.org/GNOME/gnome-shell/-/merge_requests/2267 and https://gitlab.gnome.org/GNOME/gnome-shell/-/issues/6585.

1. pacman -S acpid and then systemctl enable --now acpid

2. Create file /etc/acpi/laptop-lid.sh with the following contents:

   #!/bin/bash
   
   if grep -Fq closed /proc/acpi/button/lid/LID0/state # &&
       # This is used to detect if a display is connected.
       # For USB C displayport use: 
       # grep -Fxq connected /sys/class/drm/card1-DP-2/status
       # For hdmi use:
       # grep -Fxq connected /sys/class/drm/card0-HDMI-A-1/status
   then
       # comment out fprintd
       sed -i -E 's/^([^#].*pam_fprintd.so)/#\1/g' /etc/pam.d/sudo
   else
       # uncomment fprintd
       sed -i -E 's/#(.*pam_fprintd.so)/\1/g' /etc/pam.d/sudo
   
   fi
   

3. Make the file executable with

   chmod +x /etc/acpi/laptop-lid.sh

4. Create file /etc/acpi/events/laptop-lid with the following contents:

   event=button/lid.*
   action=/etc/acpi/laptop-lid.sh
   

5. Restart the acpid service with:

   systemctl restart acpid

Now the fingerprint will be used only when the lid is open.

In order to ensure the correct state after suspend we need a service file which runs our script on wake.

1. Create a file named /etc/systemd/system/laptop-lid.service with the following contents:

   [Unit]
   Description=Laptop Lid
   After=suspend.target
   
   [Service]
   ExecStart=/etc/acpi/laptop-lid.sh
   
   [Install]
   WantedBy=multi-user.target
   WantedBy=suspend.target
   

2. Reload the systemd config files with

   sudo systemctl daemon-reload

3. Start and enable the service with

   sudo systemctl enable --now laptop-lid.service

Now the status should be correct even after connecting/disconnecting when the computer is off.

Hardware Management

Power Profiles

https://wiki.archlinux.org/title/CPU_frequency_scaling#power-profiles-daemon

pacman -S power-profiles-daemon
systemctl enable --now power-profiles-daemon

Color Management

https://wiki.archlinux.org/title/Framework_Laptop_13#Display

https://wiki.archlinux.org/title/ICC_profiles#Wayland

cp /home/ducoterra/Downloads/BOE_CQ... /usr/share/color/icc/colord/
colormgr get-profiles
colormgr get-devices
colormgr device-add-profile xrandr-BOE-0x095f-0x00000000 icc-eca2e6d155d550a5e78c97a34ac3fcae

Washed out colors with power-profiles-daemon

https://wiki.archlinux.org/title/Framework_Laptop_13#(AMD)_Washed-out_colors_when_using_power-profiles-daemon_in_power-saver_or_balanced_mode

systemctl edit power-profiles-daemon.service --drop-in=disable_panel_powersavings

[Service]
ExecStart=
ExecStart=/usr/lib/power-profiles-daemon --block-action=amdgpu_panel_power

Hardware Acceleration

(This helps enable hardware encoding/decoding for steam streaming)

Intel

pacman -S libva-utils intel-media-driver
vainfo

AMD

pacman -S vulkan-radeon libva-utils libva-mesa-driver xf86-video-amdgpu
vainfo

Don't sleep while plugged in

This is needed for the Framework 13 (11th gen) since sleeping while plugged in to a dock will prevent it from waking up.

vim /etc/systemd/logind.conf

Bluetooth

1. pacman -S bluez bluez-utils
2. systemctl enable --now bluetooth

Audio

Without pipewire-pulse the audio level/device will reset every reboot.

1. pacman -S pipewire-pulse (remove conflicting packages)

Software Stores

AppImage Support

Also chmod +x before running.

1. cp ~/Downloads/xxxxxxx.appimage ~/Applications

2. Find an icon online and save it to ~/.icons

3. Write a .desktop entry at ~/.local/share/applications/

   [Desktop Entry]
   Name=
   Exec=/home/ducoterra/Applications/
   Icon=/home/ducoterra/.icons/
   Type=Application
   

4. desktop-file-validate ~/.local/share/applications/*.desktop

5. update-desktop-database

Troubleshooting

fuse may be required to run an appimage.

sudo pacman -S fuse

Flatpak

pacman -S flatpak

Apps

Firefox

You'll want firefox and gnome-browser-connector (for gnome extension management).

pacman -S firefox gnome-browser-connector

Choose noto-fonts

Gnome Extensions

1. AlphabeticalAppGrid@stuarthayhurst
2. Vitals@CoreCoding.com
3. dash-to-dock@micxgx.gmail.com
4. tactile@lundal.io

Avahi (Bonjour)

1. pacman -S avahi

2. vim /etc/nsswitch.conf

   hosts: mymachines mdns [NOTFOUND=return] resolve [!UNAVAIL=return] files myhostname dns
   

3. vim /etc/mdns.allow

.local.
.local

CUPS Printing

Note: you need avahi for auto-discovery.

1. pacman -S cups cups-pdf system-config-printer gutenprint foomatic-db-gutenprint-ppds
2. cups-genppdupdate
3. usermod -aG lp ducoterra
4. systemctl enable --now cups
5. In gnome settings:
   1. Add printer
   2. Enter the IP address
   3. Wait...
   4. Select "JetDirect"
   5. Select Generic
   6. Select IPP Printer
   7. Print