homelab/README.md

Homelab

Welcome to my homelab!

This repo is an in-flux collection of my personal notes, docs, and tutorials of things I find interesting and self-host.

Take a look around!

• "Active" projects (/active) are in use today and generally fall into these categories:
  • aws_ is for aws notes
  • device_ is for hardware
  • kubernetes_ is for helm charts or other kubernetes hosted software
  • os_ is for operating system setup guides and notes
  • podman_ is for containerized projects
  • software_ is for cli tools, projects without a specific way to host them, or other misfits

All active projects will have a markdown file named after the project. This is for quick access via shortcuts like ctrl + p in vscode. For example, I want to check my notes for virsh so I would type ctrl + p "virsh" to open "virsh.md".

"Retired" projects (/retired) is a graveyard of things I didn't want to delete.

"Template" projects (/templates) are quick templates for creating new active projects with sane defaults.

I keep my GPG and SSH keys in keys if you want to add those to your keyring or give me access to your servers.

Table of Contents

• Homelab
  • Table of Contents
  • Fun Facts
    • Keyboard Shortcuts
    • inputrc
    • "find ." shortcuts
    • tmux
    • bash
  • SSH Setup
  • Git GPG Commit Signing
  • Important Dates and Times
  • Project Lifecycle
  • Project Types
  • Active Project Requirements
  • Retirement Requirements
  • Project Structure
  • Creating a Project
  • Order of Operations

Fun Facts

Keyboard Shortcuts

On linux, ctrl+shift+u, then, while holding ctrl+shift, typing b+0 will type a ° (degree) symbol. Also you can enter any unicode symbol this way.

In vim: esc + o will take you to the end of a file and insert a new line.

inputrc

Add this to your ~/.inputrc to allow ctrl + backspace to delete whole words.

"\C-h": backward-kill-word

"find ." shortcuts

# Change file mode for a bunch of directories
find . -type d -exec chmod 755 {} \;

tmux

• Vertical: ctrl + b + "
• Horizontal: ctrl + b + %
• Event Horizontal Distribution: ctrl + b + alt + 1
• Even Vertical Distribution: ctrl + b + alt + 2
• Swap pane order: ctrl + b + : -> swap-pane -t 0

bash

https://tecadmin.net/bash-special-variables/

Here are some handy references for default bash variables

$0 – The name of the script being executed.
$1-$9 – The first nine command-line arguments.
$# – The number of command-line arguments.
$* – All command-line arguments as a single string.
$@ – All command-line arguments as an array.
$? – The exit status of the last executed command.
$$ – The process ID of the current shell.
$! – The process ID of the last background command.
$- – Shows the current shell options or flags. 

And here are the meanings of the shell options

h – Remember the location of commands as they are looked up
i – Interactive shell
m – Job control is enabled
B – Brace expansion is enabled
H – History substitution is enabled

So to check if you are in an interactive shell:

[ $- == *i* ]] && Some command here

SSH Setup

Generate a key (password protect it!)

# Pick one of the below key types
# ed25519
ssh-keygen -C ssh@ducoterra.net -t ed25519
# rsa 4096
ssh-keygen -C ssh@ducoterra.net -t rsa -b 4096

# Inspect a key
ssh-keygen -l -f ~/.ssh/id_rsa

# Change the password
ssh-keygen -p -f ~/.ssh/id_rsa

In your ~/.ssh/config, add the following line to set the default key

IdentityFile ~/.foo/identity

Then add a host to your local computer

Host <hostname>
    Hostname <host.something.com or IP address>
    User <remote user>
    Port <remote port>

And copy the key to a remote computer

# Copy the generated key to the server using password auth. Assumes password auth enabled.
ssh-copy-id -f -i ~/.ssh/id_ed25519 ${REMOTE_USER}@${REMOTE_HOST}

# Log into the server with your key
ssh -i ${KEY_NAME} ${REMOTE_HOST}
# Copy authorized_keys to root
sudo mkdir -p /root/.ssh
sudo cp ~/.ssh/authorized_keys /root/.ssh/authorized_keys
exit

# login and disable password auth
ssh ${REMOTE_HOST}
mkdir -p /etc/ssh/sshd_config.d
echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/01-prohibit-password.conf
systemctl restart sshd

# OPTIONAL: Disable sudo password
echo '%wheel    ALL=(ALL)   NOPASSWD: ALL' > /etc/sudoers.d/01-nopasswd-wheel

exit

# Test if you can SSH with a password
ssh -o PubkeyAuthentication=no ducoterra@${SSH_HOST}.reeselink.com

# Test that you can log into the server with ssh config
ssh $SSH_HOST

Git GPG Commit Signing

1. Use gpg --list-key 'git@ducoterra.net' to find your key
2. Use git config --global user.signingkey 0A46826A... to set the signing key
3. Use gpg --export -a 'git@ducoterra.net' to export the key to copy into Gitea/Github/Gitlab

Now you can sign commits with git commit -S.

Alternatively, you can sign every commit by default with git config --global commit.gpgsign true.

You can verify a commit with git verify-commit e1e551c. If the commit is signed you'll see an output. If not, nothing will show.

Important Dates and Times

Time	Day	Description
00:00	All	Automated builds
00:00	All	NAS Snapshots
02:00	All	Backups
04:00	All	Bare Metal Server Security Updates
05:00	All	VM Server Security Updates
05:00	All	Unifi Protect Firmware Updates
06:00	All	Unifi Network Firmware Updates
06:00	Saturday	Truenas Disk Scrub

Project Lifecycle

Projects will either be active or retired.

Active projects are being actively developed. They are in-use, stable, and production ready. Active projects should meet and track the active project requirements

Retired projects are no longer in use or recommended. They are kept for reference. Retired projects must meet the retirement requirements

You'll notice that most of the active projects have scripts or examples that use the active path as part of their install process. When moved outside the active directory their scripts and examples break. This is intentional. If you want a retired project to work again, bring it back to the active directory.

Project Types

All projects will be prefixed with one of the following categories:

• device_
• os_
• software_
• podman_
• docker_
• kubernetes_

Note, some projects will be named with just the prefix. These are projects for configuring the underlying technology. The podman project, for example, will tell you how to configure and install podman so it works correctly.

device_ will prefix projects that relate to specific machines or equipment. 3D printers, Raspberry Pis, and other IOT devices qualify as specialized hardware that needs documentation and configuration. This is not limited to computer equipment. The furnace is an important part of the homelab. the Air Conditioner is integral to the homelab's function. These projects will also be documented.

os_ will contain projects that set up operating systems. These include best practices, backups, updates, default software, etc.

cloud_ projects are for specific cloud providers. This will contain documentation and errata for things like AWS IAM, Route53, etc. Note these will be prefixed with the cloud's name, not the word "cloud". So AWS services will be prefixed with aws_ and azure would be azure_. This should make them more searchable.

software_ projects record configuration for common software agnostic to operating system or linux flavor.

podman_ projects are either designed to be run as quadlets or as podman containers outright.

kubernetes_ projects are helm, kustomize, kubectl, or some other kubernetes compliant deployment.

Active Project Requirements

• Installation is documented
• Installation configuration examples are provided
• Hardening guidelines are documented
• Upgrade procedures are documented
• Maintenance procedures are documented
• Uninstall procedures are documented
• Backup and restore procedures are documented and tested

Retirement Requirements

• A reason for retirement is documented
• If applicable, a replacement has been identified and documented
• If applicable, backup data locations are documented

Project Structure

All projects will have, at minimum.

1. A README named project-name.md
2. A directory called secrets which will be gitignored.

Creating a Project

Assuming your project name is my-project and it runs on podman

1. Create a new directory called podman_my-project under the active directory
2. Copy the readme template: cp project_readme_template.md active/podman_my-project/my-project.md
3. Populate my-project.md as you work through the install process
4. Create a directory called secrets in podman_my-project. This will be automatically gitignored. Put all secrets here.
5. Push the changes when you have a working product

Order of Operations

1. Configure cloud providers. These usually have no dependencies and typically provide critical services to other projects (DNS, email notifications, etc.)
2. Install infrastructure projects. Usually these only have dependencies on cloud services.
3. Install systemd, kubernetes, docker, podman, and other services.