various updates
This commit is contained in:
@@ -3,10 +3,6 @@
|
||||
- [Self Hosted AI Stack](#self-hosted-ai-stack)
|
||||
- [Notes](#notes)
|
||||
- [Podman Volume Locations](#podman-volume-locations)
|
||||
- [List of Internal Links](#list-of-internal-links)
|
||||
- [Quick Install](#quick-install)
|
||||
- [Text Stack](#text-stack)
|
||||
- [Image Stack](#image-stack)
|
||||
- [Setup](#setup)
|
||||
- [Create the AI user](#create-the-ai-user)
|
||||
- [Helper aliases](#helper-aliases)
|
||||
@@ -36,16 +32,7 @@
|
||||
- [open-webui](#open-webui)
|
||||
- [Upgrade Open Webui](#upgrade-open-webui)
|
||||
- [lite-llm](#lite-llm)
|
||||
- [Install Services with Quadlets](#install-services-with-quadlets)
|
||||
- [API Keys](#api-keys)
|
||||
- [Internal and External Pods](#internal-and-external-pods)
|
||||
- [Llama CPP Server (Port 8000)](#llama-cpp-server-port-8000)
|
||||
- [Llama CPP Embedding Server (Port 8001)](#llama-cpp-embedding-server-port-8001)
|
||||
- [Llama CPP Instruct Server (Port 8002)](#llama-cpp-instruct-server-port-8002)
|
||||
- [Stable Diffusion CPP (Port 1234 and 1235)](#stable-diffusion-cpp-port-1234-and-1235)
|
||||
- [Open Webui (Port 8080)](#open-webui-port-8080)
|
||||
- [Install the update script](#install-the-update-script)
|
||||
- [Install Guest Open Webui with Start/Stop Services](#install-guest-open-webui-with-startstop-services)
|
||||
- [Install the update script](#install-the-update-script)
|
||||
- [Benchmark Results](#benchmark-results)
|
||||
- [Testing with Curl](#testing-with-curl)
|
||||
- [OpenAI API](#openai-api)
|
||||
@@ -77,33 +64,6 @@ hf-download ()
|
||||
|
||||
`~/.local/share/containers/storage/volumes/`
|
||||
|
||||
### List of Internal Links
|
||||
|
||||
- llama-cpp
|
||||
- llama-embed
|
||||
- llama-instruct
|
||||
- image-gen
|
||||
- image-edit
|
||||
- openwebui
|
||||
|
||||
## Quick Install
|
||||
|
||||
### Text Stack
|
||||
|
||||
```bash
|
||||
ansible-playbook \
|
||||
-i ansible/inventory.yaml \
|
||||
active/software_ai_stack/install_ai_text_stack.yaml
|
||||
```
|
||||
|
||||
### Image Stack
|
||||
|
||||
```bash
|
||||
ansible-playbook \
|
||||
-i ansible/inventory.yaml \
|
||||
active/software_ai_stack/install_ai_image_stack.yaml
|
||||
```
|
||||
|
||||
## Setup
|
||||
|
||||
### Create the AI user
|
||||
@@ -186,8 +146,7 @@ rsync -av --progress /home/ai/models/ /srv/models/
|
||||
In my completely subjective opinion: 5 bit quant is usually the sweet spot for
|
||||
unsloth models. Q5_K_S is usually just fine.
|
||||
|
||||
I usually download the F16 mmproj files. This is also completely subjective.
|
||||
BF16 is fine. F32 is overkill.
|
||||
I usually download the BF16 mmproj files. This is also completely subjective.
|
||||
|
||||
#### Text models
|
||||
|
||||
@@ -391,20 +350,43 @@ podman run \
|
||||
--name llama-server-demo \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
-v /home/ai/models/text:/models:z \
|
||||
-v $(pwd):/models:z \
|
||||
-p 8010:8000 \
|
||||
--ipc host \
|
||||
localhost/llama-cpp-vulkan:latest \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
-c 128000 \
|
||||
--perf \
|
||||
--n-gpu-layers all \
|
||||
--jinja \
|
||||
--models-max 1 \
|
||||
--models-dir /models \
|
||||
--chat-template-kwargs '{"enable_thinking": false}' \
|
||||
-m /models/qwen3.5-35b-a3b
|
||||
--model /models/gemma-4-E4B-it-qat-UD-Q4_K_XL.gguf \
|
||||
--mmproj /models/mmproj-F16.gguf \
|
||||
--temp 1.0 \
|
||||
--top-p 0.95 \
|
||||
--top-k 64 \
|
||||
--chat-template-kwargs '{"enable_thinking":true}'
|
||||
|
||||
podman run \
|
||||
--rm \
|
||||
--name llama-server-demo \
|
||||
--device=/dev/kfd \
|
||||
--device=/dev/dri \
|
||||
-v $(pwd):/models:z \
|
||||
-p 8010:8000 \
|
||||
--ipc host \
|
||||
localhost/llama-cpp-vulkan:latest \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--perf \
|
||||
--n-gpu-layers all \
|
||||
--jinja \
|
||||
--model /models/Qwen3.5-9B-UD-Q4_K_XL.gguf \
|
||||
--mmproj /models/mmproj-BF16.gguf \
|
||||
--temp 0.6 \
|
||||
--top-p 0.95 \
|
||||
--top-k 20 \
|
||||
--min-p 0.00 \
|
||||
--chat-template-kwargs '{"enable_thinking":true}'
|
||||
```
|
||||
|
||||
Embedding models
|
||||
@@ -595,96 +577,7 @@ podman run \
|
||||
-p 4000:4000
|
||||
```
|
||||
|
||||
## Install Services with Quadlets
|
||||
|
||||
### API Keys
|
||||
|
||||
```bash
|
||||
mkdir -p /home/ai/.llama-api
|
||||
touch /home/ai/.llama-api/keys.env
|
||||
chmod 600 /home/ai/.llama-api/keys.env
|
||||
vim /home/ai/.llama-api/keys.env
|
||||
|
||||
LLAMA_API_KEY=
|
||||
|
||||
# Generate keys and append to file, then comma separate the keys
|
||||
openssl rand -base64 48 >> keys.env
|
||||
openssl rand -base64 48 >> keys.env
|
||||
openssl rand -base64 48 >> keys.env
|
||||
```
|
||||
|
||||
### Internal and External Pods
|
||||
|
||||
These will be used to restrict internet access to our llama.cpp and
|
||||
stable-diffusion.cpp services while allowing the frontend services to
|
||||
communicate with those containers.
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/ai-internal.* deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user start ai-internal-pod.service
|
||||
```
|
||||
|
||||
### Llama CPP Server (Port 8000)
|
||||
|
||||
Installs the llama.cpp server to run our text models.
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/llama-think.container deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user restart ai-internal-pod.service
|
||||
```
|
||||
|
||||
### Llama CPP Embedding Server (Port 8001)
|
||||
|
||||
Installs the llama.cpp server to run our embedding models
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/llama-embed.container deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user restart ai-internal-pod.service
|
||||
```
|
||||
|
||||
### Llama CPP Instruct Server (Port 8002)
|
||||
|
||||
Installs the llama.cpp server to run a constant instruct (no thinking) model for quick replies
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/llama-instruct.container deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user restart ai-internal-pod.service
|
||||
```
|
||||
|
||||
### Stable Diffusion CPP (Port 1234 and 1235)
|
||||
|
||||
Installs the stable-diffusion.cpp server to run our image models.
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/quadlets_stable_diffusion/* deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user restart ai-internal-pod.service
|
||||
```
|
||||
|
||||
### Open Webui (Port 8080)
|
||||
|
||||
Installs the open webui frontend.
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/quadlets_openwebui/* deskwork-ai:.config/containers/systemd/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user restart ai-external-pod.service
|
||||
```
|
||||
|
||||
Note, all services will be available at `host.containers.internal`. So llama.cpp
|
||||
will be up at `http://host.containers.internal:8000`.
|
||||
|
||||
### Install the update script
|
||||
## Install the update script
|
||||
|
||||
```bash
|
||||
# 1. Builds the latest llama.cpp and stable-diffusion.cpp
|
||||
@@ -696,18 +589,6 @@ chmod +x update-script.sh
|
||||
./update-script.sh
|
||||
```
|
||||
|
||||
### Install Guest Open Webui with Start/Stop Services
|
||||
|
||||
Optionally install a guest openwebui service.
|
||||
|
||||
```bash
|
||||
scp -r active/software_ai_stack/systemd/. deskwork-ai:.config/systemd/user/
|
||||
ssh deskwork-ai
|
||||
systemctl --user daemon-reload
|
||||
systemctl --user enable open-webui-guest-start.timer
|
||||
systemctl --user enable open-webui-guest-stop.timer
|
||||
```
|
||||
|
||||
## Benchmark Results
|
||||
|
||||
Benchmarks are run with [unsloth gpt-oss-20b Q8_0](https://huggingface.co/unsloth/gpt-oss-20b-GGUF/blob/main/gpt-oss-20b-Q8_0.gguf)
|
||||
|
||||
Reference in New Issue
Block a user