homelab/active/software_btrfs/btrfs.md

# BTRFS

- [BTRFS](#btrfs)
  - [Creating an Array](#creating-an-array)
  - [Mounting the Array](#mounting-the-array)
  - [Adding Disks](#adding-disks)
  - [Replacing a Disk](#replacing-a-disk)
  - [Scrubbing the Array](#scrubbing-the-array)
  - [Creating Subvolumes](#creating-subvolumes)
  - [Monitoring Usage](#monitoring-usage)
  - [Encrypting BTRFS with LUKS](#encrypting-btrfs-with-luks)
  - [Monitoring Disk Health](#monitoring-disk-health)
  - [Defragmenting and Compressing](#defragmenting-and-compressing)

Oracle [has decent docs here](https://docs.oracle.com/en/operating-systems/oracle-linux/8/btrfs/btrfs-ResizingaBtrfsFileSystem.html)

You'll also want to [read about btrfs compression](https://thelinuxcode.com/enable-btrfs-filesystem-compression/)

## Creating an Array

```bash
# At any point you can check the status of an array by referencing any member
btrfs filesystem show /dev/vdb
```

```bash
# Raid0
mkfs.btrfs --data raid0 --metadata raid0 /dev/vdb /dev/vdc
btrfs device scan

# Raid1
mkfs.btrfs --data raid1 --metadata raid1 /dev/vdb /dev/vdc
btrfs device scan

# Raid1c3
mkfs.btrfs --data raid1c3 --metadata raid1c3 /dev/vdb /dev/vdc /dev/vdd
btrfs device scan

# Raid10
mkfs.btrfs --data raid10 --metadata raid10 /dev/vdb /dev/vdc /dev/vdd /dev/vde
btrfs device scan

# Convert to raid1
# -dconvert == "data convert"
# -mconvert == "metadata convert"
btrfs balance start -dconvert=raid1 -mconvert=raid1 /btrfs
btrfs balance status
```

## Mounting the Array

One off

```bash
# Create a mount point
mkdir /btrfs

# Mount the top level subvolume
mount /dev/vdb /btrfs -o subvolid=5

# Mount with better SSD support
mount /dev/vdb /btrfs -o subvolid=5,ssd

# Mount with auto defragmentation for HDD support
mount /dev/vdb /btrfs -o subvolid=5,autodefrag

# Mount a subvolume
mount /dev/vdb /btrfs -o subvol=home

# Inspect
btrfs filesystem show /btrfs
```

In fstab

```conf
UUID=btrfs_uuid  /btrfs  btrfs  defaults 0 0
```

## Adding Disks

```bash
# Add a disk
btrfs device add /dev/vdd /btrfs

# Watch the expansion
btrfs filesystem usage /btrfs
```

## Replacing a Disk

```bash
# Remove a disk from the array
btrfs device delete /dev/vdb /btrfs

# Add the new device
btrfs device add /dev/vdg /btrfs
```

## Scrubbing the Array

```bash
# Start a scrub to check for errors
# -B prevents the process from going to the background
# -d prints stats for each device
btrfs scrub start -Bd /btrfs

# Check the status of a scrub
btrfs scrub status /btrfs

# Watch for disk failures
dmesg | grep btrfs
```

## Creating Subvolumes

```bash
# Create a new subvolume (make sure to mount /btrfs as subvolid=5)
btrfs subvolume create /btrfs/foo

# List all subvolumes under a path
btrfs subvolume list -t /btrfs

# Delete a subvolume
btrfs subvolume delete /btrfs/foo
```

## Monitoring Usage

```bash
# Quick info for all btrfs arrays
btrfs filesystem show

# Show usage for a specific array
btrfs filesystem usage /btrfs

# Quick command to filter for data used
btrfs filesystem usage /btrfs | grep 'Data.*Used'
```

## Encrypting BTRFS with LUKS

```bash
export KEYFILE_PATH=/root/btrfs.keyfile
export LUKS_DEVS="sdb sdc sdd sde sdf sdg sdh"

# Create a key file
dd if=/dev/urandom of=${KEYFILE_PATH} bs=128 count=1
chmod 400 ${KEYFILE_PATH}

# Create partitions
for luks_dev in $LUKS_DEVS; do
echo Creating partition for /dev/$luks_dev
parted -s -a optimal -- /dev/$luks_dev mklabel gpt mkpart primary 1MiB 100%
done

# Check that your list is good
for luks_dev in $LUKS_DEVS; do 
echo will encrypt /dev/${luks_dev}1 and create /dev/mapper/luks-$(lsblk -n -o PARTUUID /dev/${luks_dev}1)
done

# Create the luks partitions
# Note that --iter-time 10000 is how long, in milliseconds, to decrypt the key
# -v is verbose
# -q is "batch mode", don't ask for confirmation
# Longer makes it harder to brute-force
for luks_dev in $LUKS_DEVS; do \
    LUKS_UUID=$(lsblk -n -o PARTUUID /dev/${luks_dev}1)
    LUKS_NAME=luks-${LUKS_UUID}
    echo "Encrypting /dev/${luks_dev}1"; \
    cryptsetup luksFormat -v -q --key-file ${KEYFILE_PATH} /dev/${luks_dev}1
    echo "Unlocking /dev/${luks_dev}1 as ${LUKS_NAME}"
    cryptsetup open /dev/${luks_dev}1 ${LUKS_NAME} --key-file=${KEYFILE_PATH}
    echo "Adding ${LUKS_NAME} UUID=${LUKS_UUID} ${KEYFILE_PATH} discard to crypttab"
    echo "${LUKS_NAME} UUID=${LUKS_UUID} none discard" >> /etc/crypttab
done

# List filesystems with UUID
lsblk --fs

# Now create the array using the /dev/mapper entries from above
mkfs.btrfs --data raid1 --metadata raid1 /dev/mapper/crypt-btrfs-vdb /dev/mapper/crypt-btrfs-vdc...
btrfs device scan
```

## Monitoring Disk Health

<https://www.dotlinux.net/blog/how-to-configure-smartd-and-be-notified-of-hard-disk-problems-via-email/#installing-smartd-smartmontools>

```bash
# btrfs device stats shows any errors
# Grep for any line not ending in "0"
btrfs device stats /mnt | grep -vE ' 0$'

# Show the device IDs for the mounted filesystem
btrfs filesystem show /mnt

# Delete a device (with ID 8, for example)
btrfs device delete 8 /mnt

# Add a device to the array
btrfs device add /dev/vdi1 /mnt

# Rebalance the array
btrfs balance start /mnt
```

## Defragmenting and Compressing

```bash
# Defrag a filesystem
btrfs filesystem defragment /mnt

# Defrag and apply compression
# zstd:20 is currently the best compression algorithm
btrfs filesystem defragment -c zstd:20 /mnt
```