# Truenas - [Truenas](#truenas) - [Bios settings](#bios-settings) - [Archiving](#archiving) - [Deleting snapshots](#deleting-snapshots) - [But First, ZFS on RPi](#but-first-zfs-on-rpi) - [Pi Setup](#pi-setup) - [Datasets, Snapshots, and Encryption](#datasets-snapshots-and-encryption) - [Migrating encrypted pools](#migrating-encrypted-pools) - [Migrating Properties](#migrating-properties) - [Backup Task Settings](#backup-task-settings) - [Create and Destroy zfs Datasets](#create-and-destroy-zfs-datasets) - [Create and send snapshots](#create-and-send-snapshots) - [Cleaning up old snapshots](#cleaning-up-old-snapshots) - [VMs](#vms) - [Converting zvol to qcow2](#converting-zvol-to-qcow2) - [Tunables](#tunables) - [Core](#core) - [Scale](#scale) - [ARC Limit](#arc-limit) - [Certs](#certs) - [Testing](#testing) - [iperf](#iperf) - [disk](#disk) - [disk health](#disk-health) - [Dead Disks](#dead-disks) - [Corrupted data](#corrupted-data) - [Stuck VMs](#stuck-vms) - [Mounting ZVOLS](#mounting-zvols) ## Bios settings These are my recommended settings that seem stable and allow GPU passthrough 1. Memory 3200mhz, fabric 1600mhz 2. AC Power - On 3. SVM - On 4. IOMMU - On (Do not touch rebar or other pci encoding stuff) 5. Fans 100% 6. Initial video output: pci 3 7. PCIE slot 1 bifurcation: 4x4x4x4 8. Disable CSM 9. Fast Boot Enabled ## Archiving 1. Create a recursive snapshot called "archive_pool_year_month_day" 2. Create a replication task called "archive_pool_year_month_day" - select all datasets you want to backup - fill in enc0/archives/archive-year-month-day_hour-minute - full filesystem replication - select "Matching naming schema" - Use `archive-%Y-%m-%d_%H-%M` - Deselect run automatically - Save and run ## Deleting snapshots Sometimes you need to delete many snapshots from a certain dataset. The UI is terrible for this, so we need to use `zfs destroy`. xargs is the best way to do this since it allows parallel processing. ```bash # zfs list snapshots with: # -o name: only print the name # -S creation: sort by creation time # -H: don't display headers # -r: recurse through every child dataset zfs list -t snapshot enc0/archives -o name -S creation -H -r # pipe it through xargs with: # -n 1: take only 1 argument from the pipe per command # -P 8: eight parallel processes # Also pass to zfs destroy: # -v: verbose # -n: dryrun zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v -n # if that looks good you can remove the "-n" zfs list -t snapshot enc0/archives -o name -S creation -H -r | xargs -n 1 -P 8 zfs destroy -v ``` ## But First, ZFS on RPi A really good backup server is an RPi running openzfs. See [the openzfs docs](https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS%20for%20Raspberry%20Pi.html#step-2-setup-zfs) for more info. ### Pi Setup Add the vault ssh CA key to your pi. ```bash curl -o /etc/ssh/trusted-user-ca-keys.pem https://vault.ducoterra.net/v1/ssh-client-signer/public_key echo "TrustedUserCAKeys /etc/ssh/trusted-user-ca-keys.pem" >> /etc/ssh/sshd_config service ssh restart ``` Create a pi user. ```bash adduser pi usermod -a -G sudo pi ``` SSH to the pi as the "pi" user. Delete the ubuntu user. ```bash killall -u ubuntu userdel -r ubuntu ``` Disable SSH password authentication ```bash sed -i 's/PasswordAuthentication yes/PasswordAuthentication no/g' /etc/ssh/sshd_config service ssh restart ``` Change the hostname. ```bash echo pi-nas > /etc/hostname ``` Upgrade and restart the pi. ```bash apt update && apt upgrade -y && apt autoremove -y reboot ``` Install ZFS. ```bash apt install -y pv zfs-initramfs ``` Find the disks you want to use to create your pool ```bash fdisk -l ``` Create a pool. ```bash mkdir -p /mnt/backup zpool create \ -o ashift=12 \ -O acltype=posixacl -O canmount=off -O compression=lz4 \ -O dnodesize=auto -O normalization=formD -O relatime=on \ -O xattr=sa -O mountpoint=/mnt/backup \ backup ${DISK} ``` ## Datasets, Snapshots, and Encryption ### Migrating encrypted pools Since you can't use `-R` to send encrypted datasets recursively you'll need to use more creative tactics. Here's my recommendation: 1. Save the datasets from a pool to a text file: ```bash zfs list -r -o name > pool_datasets.txt ``` 2. Next, remove the prefix of the source pool from the list of datasets. Also remove the source pool itself as well as any duplicate pools in the receiving dataset. 3. Now, run a command like the following: ```bash for i in $(cat nvme_pools.txt); do zfs send -v nvme/$i@manual-2021-10-03_22-34 | zfs recv -x encryption enc0/$i; done ``` ### Migrating Properties If you need to migrate your dataset comments you can use the following bash to automate the task. ```bash for i in $(zfs list -H -d 1 -o name backup/nvme/k3os-private); do read -r name desc < <(zfs list -H -o name,org.freenas:description $i) && pvc=$(echo "$name" | awk -F "/" '{print $NF}') && zfs set org.freenas:description=$desc enc1/k3os-private/$pvc; done ``` ### Backup Task Settings | Key | Value | | ------------------------------------ | --------------------- | | Destination Dataset Read-only Policy | SET | | Recursive | true | | Snapshot Retention Policy | Same as Source | | Include Dataset Properties | true | | Periodic Snapshot Tasks | | ### Create and Destroy zfs Datasets ```bash # Create a pool zpool create rpool /dev/disk/by-id/disk-id # Add a cache disk zpool add backup cache /dev/sda # Enable encryption zpool set feature@encryption=enabled rpool # Create a dataset zfs create rpool/d1 # Create an encrypted dataset zfs create -o encryption=on -o keylocation=prompt -o keyformat=passphrase rpool/d1 # Delete a dataset zfs destroy rpool/d1 ``` ### Create and send snapshots ```bash # snapshot pool and all children zfs snapshot -r dataset@now # send all child snapshots zfs send -R dataset@snapshot | zfs recv dataset # use the -w raw flag to send encrypted snapshots zfs send -R -w dataset@snapshot | zfs recv dataset ``` ### Cleaning up old snapshots ```bash wget https://raw.githubusercontent.com/bahamas10/zfs-prune-snapshots/master/zfs-prune-snapshots ``` ## VMs 1. Force UEFI installation 2. `cp /boot/efi/EFI/debian/grubx64.efi /boot/efi/EFI/BOOT/bootx64.efi` ### Converting zvol to qcow2 ```bash dd if=/dev/zvol/enc1/vms/unifi-e373f of=unifi.raw qemu-img convert -f raw -O qcow2 unifi.raw unifi.qcow2 ``` ## Tunables ### Core ```bash sysctl kern.ipc.somaxconn=2048 sysctl kern.ipc.maxsockbuf=16777216 sysctl net.inet.tcp.recvspace=4194304 sysctl net.inet.tcp.sendspace=2097152 sysctl net.inet.tcp.sendbuf_max=16777216 sysctl net.inet.tcp.recvbuf_max=16777216 sysctl net.inet.tcp.sendbuf_auto=1 sysctl net.inet.tcp.recvbuf_auto=1 sysctl net.inet.tcp.sendbuf_inc=16384 sysctl net.inet.tcp.recvbuf_inc=524288 sysctl vfs.zfs.arc_max=34359738368 # set arc size to 32 GiB to prevent eating VMs loader vm.kmem_size=34359738368 # set kmem_size to 32 GiB to force arc_max to apply loader vm.kmem_size_max=34359738368 # set kmem_size_max to 32 GiB to sync with kmem_size ``` Nic options: "mtu 9000 rxcsum txcsum tso4 lro" ### Scale #### ARC Limit Create an Init/Shutdown Script of type `Command` with the following: ```bash echo 34359738368 >> /sys/module/zfs/parameters/zfs_arc_max ``` Set `When` to `Pre Init`. ## Certs 1. Create a new Root certificate (CAs -> ADD -> Internal CA) - Name: Something_Root - Key Length: 4096 - Digest: SHA512 - Lifetime: 825 (Apple's new requirement) - Extend Key Usage: Server Auth - Common Name: Something Root CA - Subject Alternate Names: 2. Create a new intermediate certificate (CAs -> Add -> Intermediate CA) - Name: Something_Intermediate_CA - Key Length: 4096 - Digest: SHA512 - Lifetime: 825 (Apple's new requirement) - Extend Key Usage: Server Auth 3. Create a new Certificate (Certificates -> Add -> Internal Certificate) - Name: Something_Certificate - Key Length: 4096 - Digest: SHA512 - Lifetime: 825 (Apple's new requirement) - Extend Key Usage: Server Auth ## Testing ### iperf ```bash iperf3 -c mainframe -P 4 iperf3 -c mainframe -P 4 -R iperf3 -c pc -P 4 iperf3 -c pc -P 4 -R ``` ### disk ```bash # write 16GB to disk dd if=/dev/zero of=/tmp/test bs=1024k count=16000 # divide result by 1000^3 to get GB/s # read 16GB from disk dd if=/tmp/test of=/dev/null bs=1024k # divide result by 1000^3 to get GB/s ``` ## disk health ```bash # HDD smartctl -a /dev/ada1 | grep "SMART Attributes" -A 18 # NVME smartctl -a /dev/nvme1 | grep "SMART/Health Information" -A 17 ``` ## Dead Disks ```bash === START OF INFORMATION SECTION === Model Family: Western Digital Black Device Model: WDC WD2003FZEX-00Z4SA0 Serial Number: WD-WMC5C0D6PZYZ LU WWN Device Id: 5 0014ee 65a5a19fc Firmware Version: 01.01A01 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Sat Feb 13 18:31:57 2021 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled ``` ## Corrupted data One or more devices has experienced an error resulting in data corruption. Applications may be affected. To get a list of affected files run: ```bash zpool status -v ``` ## Stuck VMs "[EFAULT] 'freeipa' VM is suspended and can only be resumed/powered off" "virsh cannot acquire state change lock monitor=remoteDispatchDomainSuspend" ```bash virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" list virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" destroy virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" start ``` ## Mounting ZVOLS Sometimes you need to mount zvols onto the truenas host. You can do this with the block device in /dev. ```bash for path in $(ls /dev/zvol/enc0/dcsi/apps/); do mount --mkdir /dev/zvol/enc0/dcsi/apps/$path /tmp/pvcs/$path; done for path in $(ls /dev/zvol/enc1/dcsi/apps/); do mount --mkdir /dev/zvol/enc1/dcsi/apps/$path /tmp/pvcs/$path; done # From driveripper rsync --progress -av -e ssh \ driveripper:/mnt/enc1/dcsi/nfs/pvc-ccaace81-bd69-4441-8de1-3b2b24baa7af/ \ /tmp/transfer/ \ --dry-run # To Kube rsync --progress -av --delete -e ssh \ /tmp/transfer/ \ kube:/opt/local-path-provisioner/ssd/pvc-4fca5cad-7640-45ea-946d-7a604a3ac875_minecraft_nimcraft/ \ --dry-run ```