add stuck VM troubleshooting for truenas

infrastructure/graduated/truenas/README.md

@@ -26,6 +26,8 @@
   - [disk health](#disk-health)
   - [Dead Disks](#dead-disks)
   - [Corrupted data](#corrupted-data)
+  - [Stuck VMs](#stuck-vms)
+  - [Mounting ZVOLS](#mounting-zvols)
 
 ## Bios settings
 
@@ -367,3 +369,36 @@ To get a list of affected files run:
 ```bash
 zpool status -v
 ```
+
+## Stuck VMs
+
+"[EFAULT] 'freeipa' VM is suspended and can only be resumed/powered off"
+
+"virsh cannot acquire state change lock monitor=remoteDispatchDomainSuspend"
+
+```bash
+virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" list
+virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" destroy <vm_id>
+virsh -c "qemu+unix:///system?socket=/run/truenas_libvirt/libvirt-sock" start <vm_id>
+```
+
+## Mounting ZVOLS
+
+Sometimes you need to mount zvols onto the truenas host. You can do this with the block device in /dev.
+
+```bash
+for path in $(ls /dev/zvol/enc0/dcsi/apps/); do mount --mkdir /dev/zvol/enc0/dcsi/apps/$path /tmp/pvcs/$path; done
+for path in $(ls /dev/zvol/enc1/dcsi/apps/); do mount --mkdir /dev/zvol/enc1/dcsi/apps/$path /tmp/pvcs/$path; done
+
+# From driveripper
+rsync --progress -av -e ssh \
+    driveripper:/mnt/enc1/dcsi/nfs/pvc-ccaace81-bd69-4441-8de1-3b2b24baa7af/ \
+    /tmp/transfer/ \
+    --dry-run
+
+# To Kube
+rsync --progress -av --delete -e ssh \
+    /tmp/transfer/ \
+    kube:/opt/local-path-provisioner/ssd/pvc-4fca5cad-7640-45ea-946d-7a604a3ac875_minecraft_nimcraft/ \
+    --dry-run
+```