services/homelab
1
1
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

remove metallb, update charts and readme

Browse Source
This commit is contained in:
Reese Wells
2023-09-07 08:51:02 -04:00
parent 813d7e0e5b
commit 6791df6e32
16 changed files with 619 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files

111
README.md
View File

@@ -1,4 +1,4 @@
# Containers!
# Containers
A project to store container-based hosting stuff.
@@ -24,8 +24,6 @@ curl -sfL https://get.k3s.io | sh -s - \
"local-storage" \
"--disable" \
"coredns" \
"--disable" \
"servicelb" \
"--cluster-dns" \
"10.43.0.10"
```
@@ -70,6 +68,12 @@ kubectl run -it --rm --restart=Never --image=infoblox/dnstools:latest dnstools
### Metal LB
The problem with metallb is when a service needs the real IP address of a client. You can
get the real IP with `externalTrafficPolicy: Local` but that prevents shared IP addresses even
if services are running on different ports. Klipper, on the other hand, seems to handle this
just fine. MetalLB isn't great for a local installation for this reason, but I'm leaving
the docs here just in case.
We'll be swapping K3S's default load balancer with Metal LB for more flexibility. ServiceLB was
struggling to allocate IP addresses for load balanced services. MetallLB does make things a little
more complicated- you'll need special annotations (see below) but it's otherwise a well-tested,
@@ -191,9 +195,9 @@ For example: your truenas server might be driveripper.reeselink.com and your kub
containers.reeselink.com. You should also have a democratic-csi-server.reeselink.com and a
democratic-csi-client-1.reeselink.com which might be on 172.20.0.1 and 172.20.0.2.
https://github.com/democratic-csi/democratic-csi
<https://github.com/democratic-csi/democratic-csi>
ISCSI requires a bit of server config before proceeding:
ISCSI requires a bit of server config before proceeding. Run the following on the kubernetes node.
```bash
# Install the following system packages
@@ -211,24 +215,67 @@ sudo systemctl enable iscsi
sudo systemctl start iscsi
```
And now you can install the drivers:
Now you can install the drivers. Note we won't be using the API drivers for Truenas
scale. These have stability issues that happen intermittently (especially when deleting
volumes... as in it won't delete volumes). As of 6/13/23 I don't recommend it.
Note: you can switch between driver types after install so there's no risk in using the
stable driver first and then experimenting with the API driver.
Before we begin you'll need to create a new "democratic" user on Truenas. First you should
create an SSH key for the user:
```bash
ssh-keygen -t rsa -N '' -f secrets/democratic_rsa.prod
chmod 600 secrets/democratic_rsa.prod
```
Now in the web console, use the following options:
| Field | Value |
|----------------------------------------|------------------------------------------------|
| Full Name | democratic |
| Username | democratic |
| Email | blank |
| Disable Password | True |
| Create New Primary Group | True |
| Auxiliary Groups | None |
| Create Home Directory | True |
| Authorized Keys | paste the generated ".pub" key here |
| Shell | bash |
| Allowed sudo commands | /usr/sbin/zfs /usr/sbin/zpool /usr/sbin/chroot |
| Allowed sudo commands with no password | /usr/sbin/zfs /usr/sbin/zpool /usr/sbin/chroot |
| Samba Authentication | False |
Save the user and verify SSH works with
```bash
ssh -i secrets/democratic_rsa.prod democratic@driveripper.reeselink.com
# test forbidden sudo command, should require a password
sudo ls
# test allowed sudo command
sudo zfs list
```
Copy `truenas-iscsi-enc0-stable.yaml` to `secrets/` and populate the secrets. Then
run the following to install it.
```bash
helm repo add democratic-csi https://democratic-csi.github.io/charts/
helm repo update
# enc0 bulk storage (iscsi)
# enc0 stable storage (iscsi)
helm upgrade \
--install \
--values truenas-iscsi-enc0.yaml \
--values secrets/truenas-iscsi-enc0-stable.yaml \
--namespace democratic-csi \
--create-namespace \
zfs-iscsi-enc0 democratic-csi/democratic-csi
# enc1 fast storage (iscsi)
# enc1 stable storage (iscsi)
helm upgrade \
--install \
--values truenas-iscsi-enc1.yaml \
--values secrets/truenas-iscsi-enc1-stable.yaml \
--namespace democratic-csi \
--create-namespace \
zfs-iscsi-enc1 democratic-csi/democratic-csi
@@ -237,13 +284,16 @@ zfs-iscsi-enc1 democratic-csi/democratic-csi
You can test that things worked with:
```bash
kubectl apply -f democratic-csi-pvc-test.yaml
kubectl apply -f tests/democratic-csi-pvc-test.yaml
kubectl delete -f tests/democratic-csi-pvc-test.yaml
```
Because iscsi will mount block devices, troubleshooting mounting issues, data corruption,
and exploring pvc contents must happen on the client device. Here are a few cheat-sheet
and exploring pvc contents must happen on the client device. Here are a few cheat-sheet
commands to make things easier:
Note with iscsi login: set the node.session.auth.username NOT node.session.auth.username_in
```bash
# discover all targets on the server
iscsiadm --mode discovery \
@@ -286,10 +336,10 @@ iscsiadm --mode node \
--targetname $ISCSI_TARGET
# view all connected disks
ls /dev/disk/by-path/
ls /dev/zvol/
# mount a disk
mount -t xfs /dev/disk/by-path/... /mnt/iscsi
mount -t xfs /dev/zvol/... /mnt/iscsi
# emergency - by-path isn't available
# (look for "Attached scsi disk")
@@ -337,7 +387,8 @@ helm upgrade --install \
nextcloud \
./helm/nextcloud \
--namespace nextcloud \
--create-namespace
--create-namespace \
--values secrets/nextcloud-values.yaml
```
Need to copy lots of files? Copy them to the user data dir and then run
@@ -346,6 +397,10 @@ Need to copy lots of files? Copy them to the user data dir and then run
./occ files:scan --all
```
Set up SES with the following links:
https://docs.aws.amazon.com/general/latest/gr/ses.html
### Gitea
Gitea provides a helm chart [here](https://gitea.com/gitea/helm-chart/). We're not
@@ -409,6 +464,30 @@ helm upgrade --install \
--create-namespace
```
### Snapdrop
Snapdrop is a file sharing app that allows airdrop-like functionality over the web
```bash
helm upgrade --install \
snapdrop \
./helm/snapdrop \
--namespace snapdrop \
--create-namespace
```
### Jellyfin
This assumes you have a media NFS share.
```bash
helm upgrade --install \
jellyfin \
./helm/jellyfin \
--namespace jellyfin \
--create-namespace
```
### Troubleshooting
Deleting a stuck namespace
@@ -432,7 +511,7 @@ Mounting an ix-application volume from truenas
# set the mountpoint
zfs set mountpoint=/ix_pvc enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b
#"unset" the mountpoing (back to legacy)
#"unset" the mountpoint (back to legacy)
zfs set mountpoint=legacy enc1/ix-applications/releases/gitea/volumes/pvc-40e27277-71e3-4469-88a3-a39f53435a8b
```