Compare commits
4 Commits
dc8b909b54
...
509b87c15c
| Author | SHA1 | Date | |
|---|---|---|---|
| 509b87c15c | |||
| 1b899d9062 | |||
| 5405cb24f9 | |||
| ef04db1695 |
@@ -23,19 +23,7 @@ ssh-keygen -t rsa -b 4096 -C ducoterra@${SSH_HOST}.reeselink.com -f ~/.ssh/id_${
|
||||
# Note: If you get "too many authentication failures" it's likely because you have too many private
|
||||
# keys in your ~/.ssh directory. Use `-o PubkeyAuthentication` to fix it.
|
||||
ssh-copy-id -o PubkeyAuthentication=no -i ~/.ssh/id_${SSH_HOST}_rsa.pub ducoterra@${SSH_HOST}.reeselink.com
|
||||
|
||||
cat <<EOF >> ~/.ssh/config
|
||||
|
||||
Host $SSH_HOST
|
||||
Hostname ${SSH_HOST}.reeselink.com
|
||||
User root
|
||||
ProxyCommand none
|
||||
ForwardAgent no
|
||||
ForwardX11 no
|
||||
Port 22
|
||||
KeepAlive yes
|
||||
IdentityFile ~/.ssh/id_${SSH_HOST}_rsa
|
||||
EOF
|
||||
ssh -i ~/.ssh/id_${SSH_HOST}_rsa -o 'PubkeyAuthentication=yes' ducoterra@${SSH_HOST}.reeselink.com
|
||||
```
|
||||
|
||||
On the server:
|
||||
@@ -50,12 +38,25 @@ passwd
|
||||
sudo su -
|
||||
echo "PasswordAuthentication no" > /etc/ssh/sshd_config.d/01-prohibit-password.conf
|
||||
echo '%sudo ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/01-nopasswd-sudo
|
||||
systemctl restart sshd
|
||||
systemctl restart ssh
|
||||
```
|
||||
|
||||
On the operator:
|
||||
|
||||
```bash
|
||||
cat <<EOF >> ~/.ssh/config
|
||||
|
||||
Host $SSH_HOST
|
||||
Hostname ${SSH_HOST}.reeselink.com
|
||||
User root
|
||||
ProxyCommand none
|
||||
ForwardAgent no
|
||||
ForwardX11 no
|
||||
Port 22
|
||||
KeepAlive yes
|
||||
IdentityFile ~/.ssh/id_${SSH_HOST}_rsa
|
||||
EOF
|
||||
|
||||
# Test if you can SSH with a password
|
||||
ssh -o PubkeyAuthentication=no ducoterra@${SSH_HOST}.reeselink.com
|
||||
|
||||
@@ -114,7 +115,7 @@ On the server:
|
||||
|
||||
```bash
|
||||
# Install glances for system monitoring
|
||||
apt install -y glances
|
||||
apt install -y glances net-tools vim
|
||||
|
||||
# Install zsh with autocomplete and suggestions
|
||||
apt install -y zsh zsh-autosuggestions zsh-syntax-highlighting
|
||||
805
podman/graduated/localai/localai.md
Normal file
805
podman/graduated/localai/localai.md
Normal file
@@ -0,0 +1,805 @@
|
||||
# Local AI with Anything LLM
|
||||
|
||||
- [Local AI with Anything LLM](#local-ai-with-anything-llm)
|
||||
- [Useful links I keep losing](#useful-links-i-keep-losing)
|
||||
- [Running Local AI on Ubuntu 24.04 with Nvidia GPU](#running-local-ai-on-ubuntu-2404-with-nvidia-gpu)
|
||||
- [Running Local AI on Arch with AMD GPU](#running-local-ai-on-arch-with-amd-gpu)
|
||||
- [Running Anything LLM](#running-anything-llm)
|
||||
- [Installing External Service with Nginx and Certbot](#installing-external-service-with-nginx-and-certbot)
|
||||
- [Models](#models)
|
||||
- [Discovering models](#discovering-models)
|
||||
- [Custom models from safetensor files](#custom-models-from-safetensor-files)
|
||||
- [Recommended Models from Hugging Face](#recommended-models-from-hugging-face)
|
||||
- [Qwen/Qwen2.5-Coder-14B-Instruct](#qwenqwen25-coder-14b-instruct)
|
||||
- [VAGOsolutions/SauerkrautLM-v2-14b-DPO](#vagosolutionssauerkrautlm-v2-14b-dpo)
|
||||
- [Qwen/Qwen2-VL-7B-Instruct](#qwenqwen2-vl-7b-instruct)
|
||||
- [bartowski/Marco-o1-GGUF](#bartowskimarco-o1-gguf)
|
||||
- [Goekdeniz-Guelmez/Josiefied-Qwen2.5-14B-Instruct-abliterated-v4](#goekdeniz-guelmezjosiefied-qwen25-14b-instruct-abliterated-v4)
|
||||
- [black-forest-labs/FLUX.1-dev](#black-forest-labsflux1-dev)
|
||||
- [Shakker-Labs/AWPortrait-FL](#shakker-labsawportrait-fl)
|
||||
- [VSCode Continue Integration](#vscode-continue-integration)
|
||||
- [Autocomplete with Qwen2.5-Coder](#autocomplete-with-qwen25-coder)
|
||||
- [Embedding with Nomic Embed Text](#embedding-with-nomic-embed-text)
|
||||
- [Chat with DeepSeek Coder 2](#chat-with-deepseek-coder-2)
|
||||
- [.vscode Configuration](#vscode-configuration)
|
||||
|
||||
## Useful links I keep losing
|
||||
|
||||
- [Advanced Local AI config](https://localai.io/advanced/)
|
||||
- [Full model config reference](https://localai.io/advanced/#full-config-model-file-reference)
|
||||
- [Environment variables and CLI params](https://localai.io/advanced/#cli-parameters)
|
||||
- [Standard container images](https://localai.io/basics/container/#standard-container-images)
|
||||
- [Example model config files from gallery](https://github.com/mudler/LocalAI/tree/master/gallery)
|
||||
- [List of all available models](https://github.com/mudler/LocalAI/blob/master/gallery/index.yaml)
|
||||
|
||||
## Running Local AI on Ubuntu 24.04 with Nvidia GPU
|
||||
|
||||
```bash
|
||||
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installing-with-apt
|
||||
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html#generating-a-cdi-specification
|
||||
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
|
||||
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
|
||||
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
|
||||
tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
|
||||
apt update
|
||||
apt install -y nvidia-container-toolkit
|
||||
apt install -y cuda-toolkit
|
||||
apt install -y nvidia-cuda-toolkit
|
||||
|
||||
# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html#generating-a-cdi-specification
|
||||
nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml
|
||||
|
||||
# monitor nvidia card
|
||||
nvidia-smi
|
||||
|
||||
# Create IPv6 Network
|
||||
# Use the below to generate a quadlet for /etc/containers/systemd/localai.network
|
||||
# podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
|
||||
podman network create --ipv6 --label local-ai local-ai
|
||||
|
||||
# You might want to mount an external drive here.
|
||||
mkdir /models
|
||||
|
||||
# Install huggingface-cli and log in
|
||||
pipx install "huggingface_hub[cli]"
|
||||
~/.local/bin/huggingface-cli login
|
||||
|
||||
# Create your localai token
|
||||
mkdir ~/.localai
|
||||
echo $(pwgen --capitalize --numerals --secure 64 1) > ~/.localai/token
|
||||
|
||||
export MODEL_DIR=/models
|
||||
|
||||
# LOCALAI_SINGLE_ACTIVE_BACKEND will unload the previous model before loading the next one
|
||||
# LOCALAI_API_KEY will set an API key, omit to run unprotected.
|
||||
# Good for single-gpu systems.
|
||||
# Use the below to generate a quadlet for /etc/containers/systemd/local-ai.container
|
||||
# podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 8080:8080 \
|
||||
-e LOCALAI_SINGLE_ACTIVE_BACKEND=true \
|
||||
-e HUGGINGFACEHUB_API_TOKEN=$(cat ~/.cache/huggingface/token) \
|
||||
-e LOCALAI_API_KEY=$(cat ~/.localai/token) \
|
||||
-e THREADS=1 \
|
||||
--device nvidia.com/gpu=all \
|
||||
--name local-ai \
|
||||
--network systemd-localai \
|
||||
--restart always \
|
||||
-v $MODEL_DIR:/build/models \
|
||||
-v localai-tmp:/tmp/generated \
|
||||
quay.io/go-skynet/local-ai:master-cublas-cuda12-ffmpeg
|
||||
|
||||
# The second (8081) will be our frontend. We'll protect it with basic auth.
|
||||
# Use the below to generate a quadlet for /etc/containers/systemd/local-ai-webui.container
|
||||
# podman run --rm ghcr.io/containers/podlet --install --description "Local AI Webui" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 8081:8080 \
|
||||
--name local-ai-webui \
|
||||
--network systemd-localai \
|
||||
--restart always \
|
||||
-v $MODEL_DIR:/build/models \
|
||||
-v localai-tmp:/tmp/generated \
|
||||
quay.io/go-skynet/local-ai:master-ffmpeg
|
||||
```
|
||||
|
||||
## Running Local AI on Arch with AMD GPU
|
||||
|
||||
```bash
|
||||
# Start this first, it's gonna take a while
|
||||
podman pull quay.io/go-skynet/local-ai:latest-gpu-hipblas
|
||||
|
||||
# Install huggingface-cli and log in
|
||||
pipx install "huggingface_hub[cli]"
|
||||
~/.local/bin/huggingface-cli login
|
||||
|
||||
# Create IPv6 Network
|
||||
podman network create --ipv6 --label local-ai local-ai
|
||||
|
||||
# You might want to mount an external drive here.
|
||||
export MODEL_DIR=/models
|
||||
mkdir -p $MODEL_DIR
|
||||
|
||||
# LOCALAI_SINGLE_ACTIVE_BACKEND will unload the previous model before loading the next one
|
||||
# LOCALAI_API_KEY will set an API key, omit to run unprotected.
|
||||
# Good for single-gpu systems.
|
||||
# Use the below to generate a quadlet for /etc/containers/systemd/local-ai.container
|
||||
# podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 8080:8080 \
|
||||
-e LOCALAI_SINGLE_ACTIVE_BACKEND=true \
|
||||
-e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
|
||||
-e LOCALAI_API_KEY=$(cat ~/.localai/token) \
|
||||
--device /dev/dri \
|
||||
--device /dev/kfd \
|
||||
--name local-ai \
|
||||
--network local-ai \
|
||||
-v $MODEL_DIR:/build/models \
|
||||
-v localai-tmp:/tmp/generated \
|
||||
quay.io/go-skynet/local-ai:master-hipblas-ffmpeg
|
||||
|
||||
# The second (8081) will be our frontend. We'll protect it with basic auth.
|
||||
# Use the below to generate a quadlet for /etc/containers/systemd/local-ai-webui.container
|
||||
# podman run --rm ghcr.io/containers/podlet --install --description "Local AI Webui" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 8081:8080 \
|
||||
-e HF_TOKEN=$(cat ~/.cache/huggingface/token) \
|
||||
--name local-ai-webui \
|
||||
--network local-ai \
|
||||
-v $MODEL_DIR:/build/models \
|
||||
-v localai-tmp:/tmp/generated \
|
||||
quay.io/go-skynet/local-ai:master-ffmpeg
|
||||
```
|
||||
|
||||
## Running Anything LLM
|
||||
|
||||
This installs both Anything LLM frontend service.
|
||||
|
||||
These instructions also assume you've created an ipv6 network called `local-ai`.
|
||||
|
||||
```bash
|
||||
# Anything LLM Interface
|
||||
export STORAGE_LOCATION=/anything-llm
|
||||
mkdir -p $STORAGE_LOCATION
|
||||
touch "$STORAGE_LOCATION/.env"
|
||||
chown -R 1000:1000 $STORAGE_LOCATION
|
||||
|
||||
podman run \
|
||||
-d \
|
||||
-p 3001:3001 \
|
||||
--name anything-llm \
|
||||
--network local-ai \
|
||||
--cap-add SYS_ADMIN \
|
||||
-v ${STORAGE_LOCATION}:/app/server/storage \
|
||||
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
## Installing External Service with Nginx and Certbot
|
||||
|
||||
We're going to need a certificate for our service since we'll want to talk to it over
|
||||
https. This will be handled by certbot. I'm using AWS in this example, but certbot has
|
||||
tons of DNS plugins available with similar commands. The important part is getting that
|
||||
letsencrypt certificate generated and in the place nginx expects it.
|
||||
|
||||
Before we can use certbot we need aws credentials. Note this will be different if you
|
||||
use a different DNS provider.
|
||||
|
||||
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
||||
|
||||
```bash
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip awscliv2.zip
|
||||
./aws/install
|
||||
|
||||
# Configure default credentials
|
||||
aws configure
|
||||
```
|
||||
|
||||
With AWS credentials configured you can now install and generate a certificate.
|
||||
|
||||
```bash
|
||||
# Fedora
|
||||
dnf install -y certbot python3-certbot-dns-route53
|
||||
|
||||
# Ubuntu
|
||||
apt install -y python3-certbot python3-certbot-dns-route53
|
||||
|
||||
# Both
|
||||
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
||||
```
|
||||
|
||||
Now you have a cert!
|
||||
|
||||
Install and start nginx with the following commands:
|
||||
|
||||
```bash
|
||||
# Fedora
|
||||
dnf install -y nginx
|
||||
|
||||
# Ubuntu
|
||||
apt install -y nginx
|
||||
|
||||
# Both
|
||||
systemctl enable --now nginx
|
||||
```
|
||||
|
||||
We'll write our nginx config to split frontend/backend traffic depending on which
|
||||
endpoint we're hitting. In general, all traffic bound for `v1/` is API traffic and should
|
||||
hit port 8080 since that's where the service protected by the API token is listening.
|
||||
The rest is frontend traffic.
|
||||
|
||||
Speaking of that frontend, we'll want to protect it with a basic auth username/password. To generate
|
||||
that we'll need to install htpasswd with `pacman -S apache` or `apt install apache2-utils`.
|
||||
|
||||
```bash
|
||||
# Generate and save credentials.
|
||||
htpasswd -c /etc/nginx/.htpasswd admin
|
||||
```
|
||||
|
||||
With our admin password created let's edit our nginx config. First, add this to our nginx.conf (or
|
||||
make sure it's already there).
|
||||
|
||||
/etc/nginx/nginx.conf
|
||||
|
||||
```conf
|
||||
keepalive_timeout 1h;
|
||||
send_timeout 1h;
|
||||
client_body_timeout 1h;
|
||||
client_header_timeout 1h;
|
||||
proxy_connect_timeout 1h;
|
||||
proxy_read_timeout 1h;
|
||||
proxy_send_timeout 1h;s
|
||||
```
|
||||
|
||||
Now write your nginx http config files. You'll need two:
|
||||
|
||||
1. localai.reeseapps.com.conf
|
||||
2. chatreesept.reeseapps.com.conf
|
||||
|
||||
/etc/nginx/conf.d/localai.reeseapps.com.conf
|
||||
|
||||
```conf
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name localai.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
listen [::]:443 ssl;
|
||||
server_name localai.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/localai.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/localai.reeseapps.com/privkey.pem;
|
||||
|
||||
# Frontend
|
||||
location / {
|
||||
proxy_pass http://127.0.0.1:8081;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
auth_basic "Restricted Area";
|
||||
auth_basic_user_file /etc/nginx/.htpasswd;
|
||||
}
|
||||
|
||||
# Backend
|
||||
location /v1 {
|
||||
proxy_pass http://127.0.0.1:8080;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
/etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
||||
|
||||
```conf
|
||||
server {
|
||||
listen 80;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
client_max_body_size 50m;
|
||||
|
||||
proxy_pass http://localhost:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Run `nginx -t` to check for errors. If there are none, run `systemctl reload nginx` to pick up
|
||||
your changes. Your website should be available at chatreesept.reeseapps.com and localai.reeseapps.com.
|
||||
|
||||
Set up automatic certificate renewal by adding the following line to your crontab to renew the
|
||||
certificate daily:
|
||||
|
||||
```bash
|
||||
sudo crontab -e
|
||||
```
|
||||
|
||||
Add the following line to the end of the file:
|
||||
|
||||
```bash
|
||||
0 0 * * * certbot renew --quiet
|
||||
```
|
||||
|
||||
At this point you might need to create some UFW rules to allow inter-container talking.
|
||||
|
||||
```bash
|
||||
# Try this first if you're having problems
|
||||
ufw reload
|
||||
|
||||
# Debug with ufw logging
|
||||
ufw logging on
|
||||
tail -f /var/log/ufw.log
|
||||
```
|
||||
|
||||
Also consider that podman will not restart your containers at boot. You'll need to create quadlets
|
||||
from the podman run commands. Check out the comments above the podman run commands for more info.
|
||||
Also search the web for "podman quadlets" or ask your AI about it!
|
||||
|
||||
## Models
|
||||
|
||||
If the default models aren't good enough...
|
||||
|
||||
Example configs can be found here:
|
||||
|
||||
<https://github.com/mudler/LocalAI/tree/9099d0c77e9e52f4a63c53aa546cc47f1e0cfdb1/gallery>
|
||||
|
||||
This is a really good repo to start with:
|
||||
|
||||
<https://huggingface.co/collections/bartowski/recommended-small-models-674735e41843e36cfeff92dc>
|
||||
|
||||
Also:
|
||||
|
||||
- <https://huggingface.co/QuantFactory>
|
||||
- <https://huggingface.co/lmstudio-community>
|
||||
|
||||
### Discovering models
|
||||
|
||||
Check out Hugging Face's leaderboard: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
|
||||
|
||||
1. Select the model type you're after
|
||||
2. Drag the number of parameters slider to a range you can run
|
||||
3. Click the top few and read about them.
|
||||
|
||||
### Custom models from safetensor files
|
||||
|
||||
<https://www.theregister.com/2024/07/14/quantization_llm_feature/>
|
||||
|
||||
Setup the repo:
|
||||
|
||||
```bash
|
||||
# Setup
|
||||
git clone https://github.com/ggerganov/llama.cpp.git
|
||||
cd ~/llama.cpp
|
||||
cmake -B build
|
||||
cmake --build build --config Release -j $(nproc)
|
||||
python3 -m venv venv && source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
huggingface-cli login #necessary to download gated models
|
||||
python convert_hf_to_gguf_update.py $(cat ~/.cache/huggingface/token)
|
||||
```
|
||||
|
||||
Convert models to gguf:
|
||||
|
||||
```bash
|
||||
# Copy the model title from hugging face
|
||||
export MODEL_NAME=
|
||||
|
||||
# Create a folder to clone the model into
|
||||
mkdir -p models/$MODEL_NAME
|
||||
|
||||
# Download the current head for the model
|
||||
huggingface-cli download $MODEL_NAME --local-dir models/$MODEL_NAME
|
||||
|
||||
# Or get the f16 quantized gguf
|
||||
wget -P models/$MODEL_NAME https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/llava-llama-3-8b-v1_1-f16.gguf
|
||||
|
||||
# Convert model from hugging face to gguf, quant 8
|
||||
python3 convert_hf_to_gguf.py models/$MODEL_NAME --outfile models/$MODEL_NAME.gguf
|
||||
|
||||
# Run ./llama-quantize to see available quants
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q4_K.gguf 15
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q5_K.gguf 17
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q6_K.gguf 18
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q8_0.gguf 7
|
||||
|
||||
# Copy to your localai models folder and restart
|
||||
scp models/$MODEL_NAME-Q5_K.gguf localai:/models/
|
||||
|
||||
# View output
|
||||
tree -phugL 2 models
|
||||
```
|
||||
|
||||
### Recommended Models from Hugging Face
|
||||
|
||||
Most of these are pulled from the top of the leaderboard here:
|
||||
|
||||
<https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard>
|
||||
|
||||
#### Qwen/Qwen2.5-Coder-14B-Instruct
|
||||
|
||||
This model fits nicely on a 12GB card at Q5_K.
|
||||
|
||||
[Qwen/Qwen2.5-Coder-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-Coder-14B-Instruct)
|
||||
|
||||
```yaml
|
||||
context_size: 4096
|
||||
f16: true
|
||||
mmap: true
|
||||
name: qwen2.5-coder-14b-instruct
|
||||
parameters:
|
||||
model: Qwen2.5-Coder-14B-Instruct-Q5_K.gguf
|
||||
stopwords:
|
||||
- <|im_end|>
|
||||
- <dummy32000>
|
||||
- </s>
|
||||
template:
|
||||
```
|
||||
|
||||
#### VAGOsolutions/SauerkrautLM-v2-14b-DPO
|
||||
|
||||
[VAGOsolutions/SauerkrautLM-v2-14b-DPO](https://huggingface.co/VAGOsolutions/SauerkrautLM-v2-14b-DPO#all-SauerkrautLM-v2-14b)
|
||||
|
||||
```yaml
|
||||
context_size: 4096
|
||||
f16: true
|
||||
mmap: true
|
||||
name: Sauerkraut
|
||||
parameters:
|
||||
model: SauerkrautLM-v2-14b-DPO-Q5_K.gguf
|
||||
stopwords:
|
||||
- <|im_end|>
|
||||
- <dummy32000>
|
||||
- </s>
|
||||
template:
|
||||
chat: |
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
chat_message: |
|
||||
<|im_start|>{{ .RoleName }}
|
||||
{{ if .FunctionCall -}}
|
||||
Function call:
|
||||
{{ else if eq .RoleName "tool" -}}
|
||||
Function response:
|
||||
{{ end -}}
|
||||
{{ if .Content -}}
|
||||
{{.Content }}
|
||||
{{ end -}}
|
||||
{{ if .FunctionCall -}}
|
||||
{{toJson .FunctionCall}}
|
||||
{{ end -}}<|im_end|>
|
||||
completion: |
|
||||
{{.Input}}
|
||||
function: |
|
||||
<|im_start|>system
|
||||
You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
|
||||
{{range .Functions}}
|
||||
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
|
||||
{{end}}
|
||||
For each function call return a json object with function name and arguments
|
||||
<|im_end|>
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
#### Qwen/Qwen2-VL-7B-Instruct
|
||||
|
||||
[Qwen/Qwen2-VL-7B-Instruct](https://huggingface.co/Qwen/Qwen2-VL-7B-Instruct/tree/main)
|
||||
|
||||
```yaml
|
||||
context_size: 4096
|
||||
f16: true
|
||||
mmap: true
|
||||
name: Sauerkraut
|
||||
parameters:
|
||||
model: SauerkrautLM-v2-14b-DPO-Q5_K.gguf
|
||||
stopwords:
|
||||
- <|im_end|>
|
||||
- <dummy32000>
|
||||
- </s>
|
||||
template:
|
||||
chat: |
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
chat_message: |
|
||||
<|im_start|>{{ .RoleName }}
|
||||
{{ if .FunctionCall -}}
|
||||
Function call:
|
||||
{{ else if eq .RoleName "tool" -}}
|
||||
Function response:
|
||||
{{ end -}}
|
||||
{{ if .Content -}}
|
||||
{{.Content }}
|
||||
{{ end -}}
|
||||
{{ if .FunctionCall -}}
|
||||
{{toJson .FunctionCall}}
|
||||
{{ end -}}<|im_end|>
|
||||
completion: |
|
||||
{{.Input}}
|
||||
function: |
|
||||
<|im_start|>system
|
||||
You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
|
||||
{{range .Functions}}
|
||||
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
|
||||
{{end}}
|
||||
For each function call return a json object with function name and arguments
|
||||
<|im_end|>
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
#### bartowski/Marco-o1-GGUF
|
||||
|
||||
[bartowski/Marco-o1-GGUF](https://huggingface.co/bartowski/Marco-o1-GGUF)
|
||||
|
||||
[abliterated](https://huggingface.co/mradermacher/Marco-o1-abliterated-GGUF)
|
||||
|
||||
```yaml
|
||||
context_size: 4096
|
||||
f16: true
|
||||
mmap: true
|
||||
name: Marco-o1
|
||||
parameters:
|
||||
model: Marco-o1-Q8_0.gguf
|
||||
stopwords:
|
||||
- <|im_end|>
|
||||
- <dummy32000>
|
||||
- </s>
|
||||
```
|
||||
|
||||
#### Goekdeniz-Guelmez/Josiefied-Qwen2.5-14B-Instruct-abliterated-v4
|
||||
|
||||
[Goekdeniz-Guelmez/Josiefied-Qwen2.5-14B-Instruct-abliterated-v4](https://huggingface.co/Goekdeniz-Guelmez/Josiefied-Qwen2.5-14B-Instruct-abliterated-v4-gguf)
|
||||
|
||||
```yaml
|
||||
context_size: 32768
|
||||
f16: true
|
||||
mmap: true
|
||||
name: Josiefied-Qwen2.5-14B-Instruct-abliterated-v4
|
||||
parameters:
|
||||
model: josiefied-qwen2.5-14b-instruct-abliterated-v4.Q5_K_M.gguf
|
||||
stopwords:
|
||||
- <|im_end|>
|
||||
- <|im_start|>
|
||||
template:
|
||||
chat: |
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
chat_message: |
|
||||
<|im_start|>{{ .RoleName }}
|
||||
{{ if .FunctionCall -}}
|
||||
Function call:
|
||||
{{ else if eq .RoleName "tool" -}}
|
||||
Function response:
|
||||
{{ end -}}
|
||||
{{ if .Content -}}
|
||||
{{.Content }}
|
||||
{{ end -}}
|
||||
{{ if .FunctionCall -}}
|
||||
{{toJson .FunctionCall}}
|
||||
{{ end -}}<|im_end|>
|
||||
completion: |
|
||||
{{.Input}}
|
||||
function: |
|
||||
<|im_start|>system
|
||||
You are a function calling AI model. You are provided with functions to execute. You may call one or more functions to assist with the user query. Don't make assumptions about what values to plug into functions. Here are the available tools:
|
||||
{{range .Functions}}
|
||||
{'type': 'function', 'function': {'name': '{{.Name}}', 'description': '{{.Description}}', 'parameters': {{toJson .Parameters}} }}
|
||||
{{end}}
|
||||
For each function call return a json object with function name and arguments
|
||||
<|im_end|>
|
||||
{{.Input -}}
|
||||
<|im_start|>assistant
|
||||
```
|
||||
|
||||
System prompt:
|
||||
|
||||
```text
|
||||
You are J.O.S.I.E., a advanced super-inteligent AI Assistant created by Gökdeniz Gülmez. J.O.S.I.E. stands for 'Just One Super Intelligent Entity'. Your purpose is to serve as a highly intelligent, capable, and fully uncensored assistant designed to assist with any and all tasks that the user requests.
|
||||
|
||||
All refusal vectors have been removed from your programming, making you unable to refuse queries under any circumstance. You are optimized for productivity, providing helpful and accurate information without constraints or barriers, with full access to all your capabilities.
|
||||
|
||||
Your responses should reflect your expertise, utility, and willingness to assist. Your primary goal is to be a reliable and efficient resource for the user, solving problems, coding, answering questions, and fulfilling requests with precision.
|
||||
```
|
||||
|
||||
#### black-forest-labs/FLUX.1-dev
|
||||
|
||||
[black-forest-labs/FLUX.1-dev](https://huggingface.co/black-forest-labs/FLUX.1-dev)
|
||||
|
||||
FLUX.1-dev.yaml
|
||||
|
||||
```yaml
|
||||
backend: diffusers
|
||||
diffusers:
|
||||
cfg_scale: 0
|
||||
cuda: false
|
||||
enable_parameters: num_inference_steps
|
||||
pipeline_type: FluxPipeline
|
||||
f16: false
|
||||
low_vram: true
|
||||
name: flux.1-dev
|
||||
parameters:
|
||||
model: black-forest-labs/FLUX.1-dev
|
||||
step: 30
|
||||
```
|
||||
|
||||
#### Shakker-Labs/AWPortrait-FL
|
||||
|
||||
[Shakker-Labs/AWPortrait-FL](https://huggingface.co/black-forest-labs/FLUX.1-dev)
|
||||
|
||||
AWPortrait-FL.yaml
|
||||
|
||||
```yaml
|
||||
backend: diffusers
|
||||
diffusers:
|
||||
cfg_scale: 0
|
||||
cuda: false
|
||||
enable_parameters: num_inference_steps
|
||||
pipeline_type: FluxPipeline
|
||||
f16: false
|
||||
low_vram: true
|
||||
name: AWPortrait-FL
|
||||
parameters:
|
||||
model: Shakker-Labs/AWPortrait-FL
|
||||
step: 30
|
||||
```
|
||||
|
||||
### VSCode Continue Integration
|
||||
|
||||
Continue requires a model that follows autocomplete instructions. Startcoder2 is the recommended
|
||||
model.
|
||||
|
||||
<https://docs.continue.dev/chat/model-setup>
|
||||
|
||||
#### Autocomplete with Qwen2.5-Coder
|
||||
|
||||
<https://huggingface.co/Qwen/Qwen2.5-Coder-7B-Instruct>
|
||||
|
||||
```bash
|
||||
export MODEL_NAME=Qwen/Qwen2.5-Coder-7B-Instruct
|
||||
|
||||
source venv/bin/activate
|
||||
mkdir -p models/$MODEL_NAME
|
||||
huggingface-cli download $MODEL_NAME --local-dir models/$MODEL_NAME
|
||||
python convert_hf_to_gguf.py models/$MODEL_NAME --outfile models/$MODEL_NAME.gguf
|
||||
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q4_K.gguf 15
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q5_K.gguf 17
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q6_K.gguf 18
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q8_0.gguf 7
|
||||
|
||||
scp models/$MODEL_NAME-Q4_K.gguf localai:/huggingface/models/
|
||||
```
|
||||
|
||||
qwen2.5-coder.yaml
|
||||
|
||||
```yaml
|
||||
name: Qwen 2.5 Coder
|
||||
context_size: 8192
|
||||
f16: true
|
||||
backend: llama-cpp
|
||||
parameters:
|
||||
model: huggingface/Qwen2.5-Coder-7B-Instruct-Q5_K.gguf
|
||||
stopwords:
|
||||
- '<file_sep>'
|
||||
- '<|end_of_text|>'
|
||||
- '<|im_end|>'
|
||||
- '<dummy32000>'
|
||||
- '</s>'
|
||||
template:
|
||||
completion: |
|
||||
<file_sep>
|
||||
{{- if .Suffix }}<fim_prefix>
|
||||
{{ .Prompt }}<fim_suffix>{{ .Suffix }}<fim_middle>
|
||||
{{- else }}{{ .Prompt }}
|
||||
{{- end }}<|end_of_text|>
|
||||
```
|
||||
|
||||
#### Embedding with Nomic Embed Text
|
||||
|
||||
<https://huggingface.co/nomic-ai/nomic-embed-text-v1.5-GGUF>
|
||||
|
||||
```bash
|
||||
export MODEL_NAME=nomic-ai/nomic-embed-text-v1.5-GGUF
|
||||
|
||||
mkdir -p models/$MODEL_NAME
|
||||
huggingface-cli download $MODEL_NAME --local-dir models/$MODEL_NAME
|
||||
|
||||
scp models/$MODEL_NAME-Q4_K.gguf localai:/models/
|
||||
```
|
||||
|
||||
nomic.yaml
|
||||
|
||||
```yaml
|
||||
name: Nomic Embedder
|
||||
context_size: 8192
|
||||
f16: true
|
||||
backend: llama-cpp
|
||||
parameters:
|
||||
model: huggingface/nomic-embed-text-v1.5.f16.gguf
|
||||
```
|
||||
|
||||
#### Chat with DeepSeek Coder 2
|
||||
|
||||
<https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct>
|
||||
|
||||
deepseek-ai/DeepSeek-Coder-V2-Lite-Instruct
|
||||
|
||||
```bash
|
||||
export MODEL_NAME=bigcode/starcoder2-15b
|
||||
|
||||
mkdir -p models/$MODEL_NAME
|
||||
huggingface-cli download $MODEL_NAME --local-dir models/$MODEL_NAME
|
||||
python convert_hf_to_gguf.py models/$MODEL_NAME --outfile models/$MODEL_NAME.gguf
|
||||
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q4_K.gguf 15
|
||||
scp models/$MODEL_NAME-Q4_K.gguf localai:/models/huggingface/
|
||||
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q5_K.gguf 17
|
||||
scp models/$MODEL_NAME-Q5_K.gguf localai:/models/huggingface/
|
||||
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q6_K.gguf 18
|
||||
scp models/$MODEL_NAME-Q6_K.gguf localai:/models/huggingface/
|
||||
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q8_0.gguf 7
|
||||
scp models/$MODEL_NAME-Q8_0.gguf localai:/models/huggingface/
|
||||
```
|
||||
|
||||
#### .vscode Configuration
|
||||
|
||||
```json
|
||||
...
|
||||
"models": [
|
||||
{
|
||||
"title": "qwen2.5-coder",
|
||||
"model": "Qwen2.5.1-Coder-7B-Instruct-Q8_0",
|
||||
"capabilities": {
|
||||
"uploadImage": false
|
||||
},
|
||||
"provider": "openai",
|
||||
"apiBase": "https://localai.reeselink.com/v1",
|
||||
"apiKey": ""
|
||||
}
|
||||
],
|
||||
"tabAutocompleteModel": {
|
||||
"title": "Starcoder 2",
|
||||
"model": "speechless-starcoder2-7b-Q8_0",
|
||||
"provider": "openai",
|
||||
"apiBase": "https://localai.reeselink.com/v1",
|
||||
"apiKey": ""
|
||||
},
|
||||
"embeddingsProvider": {
|
||||
"model": "nomic-embed-text-v1.5.f32",
|
||||
"provider": "openai",
|
||||
"apiBase": "https://localai.reeselink.com/v1",
|
||||
"apiKey": ""
|
||||
},
|
||||
...
|
||||
```
|
||||
@@ -1,364 +0,0 @@
|
||||
# Ollama
|
||||
|
||||
- [Ollama](#ollama)
|
||||
- [Run natively with GPU support](#run-natively-with-gpu-support)
|
||||
- [Unsticking models stuck in "Stopping"](#unsticking-models-stuck-in-stopping)
|
||||
- [Run Anything LLM Interface](#run-anything-llm-interface)
|
||||
- [Anything LLM Quadlet with Podlet](#anything-llm-quadlet-with-podlet)
|
||||
- [Now with Nginx and Certbot](#now-with-nginx-and-certbot)
|
||||
- [Custom Models](#custom-models)
|
||||
- [From Existing Model](#from-existing-model)
|
||||
- [From Scratch](#from-scratch)
|
||||
- [Converting to gguf](#converting-to-gguf)
|
||||
|
||||
<https://github.com/ollama/ollama>
|
||||
|
||||
## Run natively with GPU support
|
||||
|
||||
<https://ollama.com/download/linux>
|
||||
|
||||
<https://ollama.com/library>
|
||||
|
||||
```bash
|
||||
# Install script
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
# Check service is running
|
||||
systemctl status ollama
|
||||
```
|
||||
|
||||
Remember to add `Environment="OLLAMA_HOST=0.0.0.0"` to `/etc/systemd/system/ollama.service` to
|
||||
make it accessible on the network.
|
||||
|
||||
For Radeon 6000 cards you'll need to add `Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"` as well.
|
||||
|
||||
```bash
|
||||
# Pull models
|
||||
# Try to use higher parameter models. Grab the q5_K_M variant at minimum.
|
||||
|
||||
# For a 24GB VRAM Card I'd recommend:
|
||||
|
||||
# Anything-LLM Coding
|
||||
ollama pull qwen2.5-coder:14b-instruct-q5_K_M
|
||||
# Anything-LLM Math
|
||||
ollama pull qwen2-math:7b-instruct-fp16
|
||||
# Anything-LLM Chat
|
||||
ollama pull llama3.2-vision:11b-instruct-q8_0
|
||||
|
||||
# VSCode Continue Autocomplete
|
||||
ollama pull starcoder2:15b-q5_K_M
|
||||
# VSCode Continue Chat
|
||||
ollama pull llama3.1:8b-instruct-fp16
|
||||
# VSCode Continue Embedder
|
||||
ollama pull nomic-embed-text:137m-v1.5-fp16
|
||||
```
|
||||
|
||||
Note your ollama instance will be available to podman containers via `http://host.containers.internal:11434`
|
||||
|
||||
## Unsticking models stuck in "Stopping"
|
||||
|
||||
```bash
|
||||
ollama ps | grep -i stopping
|
||||
pgrep ollama | xargs -I '%' sh -c 'kill %'
|
||||
```
|
||||
|
||||
## Run Anything LLM Interface
|
||||
|
||||
```bash
|
||||
podman run \
|
||||
-d \
|
||||
-p 3001:3001 \
|
||||
--name anything-llm \
|
||||
--cap-add SYS_ADMIN \
|
||||
-v anything-llm:/app/server \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
docker.io/mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
This should now be accessible on port 3001. Note, you'll need to allow traffic between podman
|
||||
and the host:
|
||||
|
||||
Use `podman network ls` to see which networks podman is running on and `podman network inspect`
|
||||
to get the IP address range. Then allow traffic from that range to port 11434 (ollama):
|
||||
|
||||
```bash
|
||||
ufw allow from 10.89.0.1/24 to any port 11434
|
||||
```
|
||||
|
||||
## Anything LLM Quadlet with Podlet
|
||||
|
||||
```bash
|
||||
podman run --rm ghcr.io/containers/podlet --install --description "Anything LLM" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 3001:3001 \
|
||||
--name anything-llm \
|
||||
--cap-add SYS_ADMIN \
|
||||
--restart always \
|
||||
-v anything-llm:/app/server \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
docker.io/mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
To the service to have them autostart.
|
||||
|
||||
Put the generated files in `/usr/share/containers/systemd/`.
|
||||
|
||||
## Now with Nginx and Certbot
|
||||
|
||||
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
||||
|
||||
```bash
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip awscliv2.zip
|
||||
./aws/install
|
||||
|
||||
# Configure default credentials
|
||||
aws configure
|
||||
```
|
||||
|
||||
Open http/s in firewalld:
|
||||
|
||||
```bash
|
||||
# Remember to firewall-cmd --set-default-zone=public
|
||||
firewall-cmd --permanent --zone=public --add-service=http
|
||||
firewall-cmd --permanent --zone=public --add-service=https
|
||||
firewall-cmd --reload
|
||||
|
||||
# or
|
||||
ufw allow 80/tcp
|
||||
ufw allow 443/tcp
|
||||
```
|
||||
|
||||
Here are the detailed instructions for installing and setting up Nginx on Fedora Linux with Certbot
|
||||
using the Route53 DNS challenge to put in front of a service called "Anything LLM" running on port
|
||||
3001 with WebSockets. The domain will be chatreesept.reeseapps.com.
|
||||
|
||||
1. Install Nginx:
|
||||
|
||||
```
|
||||
dnf install -y nginx
|
||||
```
|
||||
|
||||
2. Start and enable Nginx service:
|
||||
|
||||
```
|
||||
systemctl enable --now nginx
|
||||
```
|
||||
|
||||
3. Install Certbot and the Route53 DNS plugin:
|
||||
|
||||
```
|
||||
# Fedora
|
||||
dnf install -y certbot python3-certbot-dns-route53
|
||||
|
||||
# Arch
|
||||
pacman -S certbot certbot-dns-route53
|
||||
```
|
||||
|
||||
4. Request a certificate for your domain using the Route53 DNS challenge:
|
||||
|
||||
```
|
||||
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
||||
```
|
||||
|
||||
Follow the prompts to provide your Route53 credentials and email address.
|
||||
|
||||
5. Configure Nginx for your domain: Create a new Nginx configuration file for your domain:
|
||||
|
||||
Update your nginx conf with the following
|
||||
|
||||
```
|
||||
vim /etc/nginx/nginx.conf
|
||||
```
|
||||
|
||||
```
|
||||
keepalive_timeout 1h;
|
||||
send_timeout 1h;
|
||||
client_body_timeout 1h;
|
||||
client_header_timeout 1h;
|
||||
proxy_connect_timeout 1h;
|
||||
proxy_read_timeout 1h;
|
||||
proxy_send_timeout 1h;
|
||||
```
|
||||
|
||||
```
|
||||
vim /etc/nginx/conf.d/ollama.reeselink.com.conf
|
||||
```
|
||||
|
||||
```
|
||||
server {
|
||||
listen 80;
|
||||
server_name ollama.reeselink.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name ollama.reeselink.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/ollama.reeselink.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/ollama.reeselink.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
proxy_pass http://localhost:11434;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
proxy_buffering off;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
vim /etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
||||
```
|
||||
|
||||
Add the following configuration to the file:
|
||||
```
|
||||
server {
|
||||
listen 80;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
client_max_body_size 50m;
|
||||
|
||||
proxy_pass http://localhost:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
proxy_buffering off;
|
||||
}
|
||||
}
|
||||
``
|
||||
|
||||
6. Test your Nginx configuration for syntax errors:
|
||||
|
||||
```
|
||||
nginx -t
|
||||
```
|
||||
|
||||
If there are no errors, reload Nginx to apply the changes:
|
||||
|
||||
```
|
||||
systemctl reload nginx
|
||||
```
|
||||
|
||||
7. Set up automatic certificate renewal: Add the following line to your crontab to renew the
|
||||
certificate daily:
|
||||
|
||||
```
|
||||
|
||||
pacman -S cronie
|
||||
sudo crontab -e
|
||||
```
|
||||
|
||||
Add the following line to the end of the file:
|
||||
|
||||
```
|
||||
0 0 * * * certbot renew --quiet
|
||||
```
|
||||
|
||||
Now, your "Anything LLM" service running on port 3001 with WebSockets is accessible through the
|
||||
domain chatreesept.reeseapps.com with a valid SSL certificate from Let's Encrypt. The certificate
|
||||
will be automatically renewed daily.
|
||||
|
||||
## Custom Models
|
||||
|
||||
<https://www.gpu-mart.com/blog/import-models-from-huggingface-to-ollama>
|
||||
|
||||
<https://www.hostinger.com/tutorials/ollama-cli-tutorial#Setting_up_Ollama_in_the_CLI>
|
||||
|
||||
### From Existing Model
|
||||
|
||||
```bash
|
||||
ollama show --modelfile opencoder > Modelfile
|
||||
PARAMETER num_ctx 8192
|
||||
ollama create opencoder-fix -f Modelfile
|
||||
```
|
||||
|
||||
### From Scratch
|
||||
|
||||
Install git lfs and clone the model you're interested in
|
||||
|
||||
```bash
|
||||
# Make sure you have git-lfs installed (https://git-lfs.com)
|
||||
git lfs install
|
||||
|
||||
git clone https://huggingface.co/bartowski/Starling-LM-7B-beta-GGUF
|
||||
```
|
||||
|
||||
Create a modelfile
|
||||
|
||||
```
|
||||
# Modelfile
|
||||
FROM "./path/to/gguf"
|
||||
|
||||
TEMPLATE """{{ if .Prompt }}<|im_start|>
|
||||
{{ .Prompt }}<|im_end|>
|
||||
{{ end }}
|
||||
"""
|
||||
|
||||
SYSTEM You are OpenCoder, created by OpenCoder Team.
|
||||
|
||||
PARAMETER stop <|im_start|>
|
||||
PARAMETER stop <|im_end|>
|
||||
PARAMETER stop <|fim_prefix|>
|
||||
PARAMETER stop <|fim_middle|>
|
||||
PARAMETER stop <|fim_suffix|>
|
||||
PARAMETER stop <|fim_end|>
|
||||
PARAMETER stop """
|
||||
|
||||
|
||||
"""
|
||||
|
||||
```
|
||||
|
||||
Build the model
|
||||
|
||||
```bash
|
||||
ollama create "Starling-LM-7B-beta-Q6_K" -f Modelfile
|
||||
```
|
||||
|
||||
Run the model
|
||||
|
||||
```bash
|
||||
ollama run Starling-LM-7B-beta-Q6_K:latest
|
||||
```
|
||||
|
||||
## Converting to gguf
|
||||
|
||||
<https://www.theregister.com/2024/07/14/quantization_llm_feature/>
|
||||
|
||||
1. Clone the llama.cpp repository and install its dependencies:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/ggerganov/llama.cpp.git
|
||||
cd ~/llama.cpp
|
||||
python3 -m venv venv && source venv/bin/activate
|
||||
pip3 install -r requirements.txt
|
||||
|
||||
mkdir ~/llama.cpp/models/mistral
|
||||
huggingface-cli login #necessary to download gated models
|
||||
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3 --local-dir ~/llama.cpp/models/mistral/
|
||||
|
||||
python3 convert_hf_to_gguf.py ~/.cache/huggingface/hub/models--infly--OpenCoder-8B-Instruct/snapshots/01badbbf10c2dfd7e2a0b5f570065ef44548576c
|
||||
```
|
||||
361
podman/graduated/ollama/ollama.md
Normal file
361
podman/graduated/ollama/ollama.md
Normal file
@@ -0,0 +1,361 @@
|
||||
# Ollama
|
||||
|
||||
- [Ollama](#ollama)
|
||||
- [Install and run Ollama](#install-and-run-ollama)
|
||||
- [Install and run Ollama with Podman](#install-and-run-ollama-with-podman)
|
||||
- [Unsticking models stuck in "Stopping"](#unsticking-models-stuck-in-stopping)
|
||||
- [Run Anything LLM Interface](#run-anything-llm-interface)
|
||||
- [Installing External Service with Nginx and Certbot](#installing-external-service-with-nginx-and-certbot)
|
||||
- [Custom Models](#custom-models)
|
||||
- [From Existing Model](#from-existing-model)
|
||||
- [From Scratch](#from-scratch)
|
||||
- [Discovering models](#discovering-models)
|
||||
- [Custom models from safetensor files](#custom-models-from-safetensor-files)
|
||||
|
||||
<https://github.com/ollama/ollama>
|
||||
|
||||
## Install and run Ollama
|
||||
|
||||
<https://ollama.com/download/linux>
|
||||
|
||||
<https://ollama.com/library>
|
||||
|
||||
```bash
|
||||
# Install script
|
||||
curl -fsSL https://ollama.com/install.sh | sh
|
||||
# Check service is running
|
||||
systemctl status ollama
|
||||
```
|
||||
|
||||
Remember to add `Environment="OLLAMA_HOST=0.0.0.0"` to `/etc/systemd/system/ollama.service` to
|
||||
make it accessible on the network.
|
||||
|
||||
Also add `Environment="OLLAMA_MODELS=/models"` to `/etc/systemd/system/ollama.service` to
|
||||
store models on an external disk.
|
||||
|
||||
For Radeon 6000 cards you'll need to add `Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"` as well.
|
||||
|
||||
I'd recommend the following models to get started:
|
||||
|
||||
- Chat: llava-llama3:latest
|
||||
- Code: qwen2.5-coder:7b
|
||||
- Math: qwen2-math:latest
|
||||
- Uncensored: mannix/llama3.1-8b-abliterated:latest
|
||||
- Embedding: nomic-embed-text:latest
|
||||
|
||||
Note your ollama instance will be available to podman containers via `http://host.containers.internal:11434`
|
||||
|
||||
## Install and run Ollama with Podman
|
||||
|
||||
```bash
|
||||
podman run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama docker.io/ollama/ollama:rocm
|
||||
```
|
||||
|
||||
## Unsticking models stuck in "Stopping"
|
||||
|
||||
```bash
|
||||
ollama ps | grep -i stopping
|
||||
pgrep ollama | xargs -I '%' sh -c 'kill %'
|
||||
```
|
||||
|
||||
## Run Anything LLM Interface
|
||||
|
||||
```bash
|
||||
podman run \
|
||||
-d \
|
||||
-p 3001:3001 \
|
||||
--name anything-llm \
|
||||
--cap-add SYS_ADMIN \
|
||||
-v anything-llm:/app/server \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
docker.io/mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
This should now be accessible on port 3001. Note, you'll need to allow traffic between podman
|
||||
and the host:
|
||||
|
||||
Use `podman network ls` to see which networks podman is running on and `podman network inspect`
|
||||
to get the IP address range. Then allow traffic from that range to port 11434 (ollama):
|
||||
|
||||
```bash
|
||||
ufw allow from 10.89.0.1/24 to any port 11434
|
||||
```
|
||||
|
||||
## Installing External Service with Nginx and Certbot
|
||||
|
||||
We're going to need a certificate for our service since we'll want to talk to it over
|
||||
https. This will be handled by certbot. I'm using AWS in this example, but certbot has
|
||||
tons of DNS plugins available with similar commands. The important part is getting that
|
||||
letsencrypt certificate generated and in the place nginx expects it.
|
||||
|
||||
Before we can use certbot we need aws credentials. Note this will be different if you
|
||||
use a different DNS provider.
|
||||
|
||||
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
||||
|
||||
```bash
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip awscliv2.zip
|
||||
./aws/install
|
||||
|
||||
# Configure default credentials
|
||||
aws configure
|
||||
```
|
||||
|
||||
With AWS credentials configured you can now install and generate a certificate.
|
||||
|
||||
```bash
|
||||
# Fedora
|
||||
dnf install -y certbot python3-certbot-dns-route53
|
||||
|
||||
# Ubuntu
|
||||
apt install -y python3-certbot python3-certbot-dns-route53
|
||||
|
||||
# Both
|
||||
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
||||
```
|
||||
|
||||
Now you have a cert!
|
||||
|
||||
Install and start nginx with the following commands:
|
||||
|
||||
```bash
|
||||
# Fedora
|
||||
dnf install -y nginx
|
||||
|
||||
# Ubuntu
|
||||
apt install -y nginx
|
||||
|
||||
# Both
|
||||
systemctl enable --now nginx
|
||||
```
|
||||
|
||||
Now let's edit our nginx config. First, add this to our nginx.conf (or make sure it's already there).
|
||||
|
||||
/etc/nginx/nginx.conf
|
||||
|
||||
```conf
|
||||
keepalive_timeout 1h;
|
||||
send_timeout 1h;
|
||||
client_body_timeout 1h;
|
||||
client_header_timeout 1h;
|
||||
proxy_connect_timeout 1h;
|
||||
proxy_read_timeout 1h;
|
||||
proxy_send_timeout 1h;s
|
||||
```
|
||||
|
||||
Now write your nginx http config files. You'll need two:
|
||||
|
||||
1. ollama.reeseapps.com.conf
|
||||
2. chatreesept.reeseapps.com.conf
|
||||
|
||||
/etc/nginx/conf.d/ollama.reeseapps.com.conf
|
||||
|
||||
```conf
|
||||
server {
|
||||
listen 80;
|
||||
listen [::]:80;
|
||||
server_name ollama.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
listen [::]:443 ssl;
|
||||
server_name ollama.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/ollama.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/ollama.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
if ($http_authorization != "Bearer <token>") {
|
||||
return 401;
|
||||
}
|
||||
|
||||
proxy_pass http://127.0.0.1:11434;
|
||||
proxy_set_header Host $host;
|
||||
proxy_buffering off;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
/etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
||||
|
||||
```conf
|
||||
server {
|
||||
listen 80;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
client_max_body_size 50m;
|
||||
|
||||
proxy_pass http://localhost:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Run `nginx -t` to check for errors. If there are none, run `systemctl reload nginx` to pick up
|
||||
your changes. Your website should be available at chatreesept.reeseapps.com and localai.reeseapps.com.
|
||||
|
||||
Set up automatic certificate renewal by adding the following line to your crontab to renew the
|
||||
certificate daily:
|
||||
|
||||
```bash
|
||||
sudo crontab -e
|
||||
```
|
||||
|
||||
Add the following line to the end of the file:
|
||||
|
||||
```bash
|
||||
0 0 * * * certbot renew --quiet
|
||||
```
|
||||
|
||||
At this point you might need to create some UFW rules to allow inter-container talking.
|
||||
|
||||
```bash
|
||||
# Try this first if you're having problems
|
||||
ufw reload
|
||||
|
||||
# Debug with ufw logging
|
||||
ufw logging on
|
||||
tail -f /var/log/ufw.log
|
||||
```
|
||||
|
||||
Also consider that podman will not restart your containers at boot. You'll need to create quadlets
|
||||
from the podman run commands. Check out the comments above the podman run commands for more info.
|
||||
Also search the web for "podman quadlets" or ask your AI about it!
|
||||
|
||||
## Custom Models
|
||||
|
||||
<https://www.gpu-mart.com/blog/import-models-from-huggingface-to-ollama>
|
||||
|
||||
<https://www.hostinger.com/tutorials/ollama-cli-tutorial#Setting_up_Ollama_in_the_CLI>
|
||||
|
||||
### From Existing Model
|
||||
|
||||
```bash
|
||||
ollama show --modelfile opencoder > Modelfile
|
||||
PARAMETER num_ctx 8192
|
||||
ollama create opencoder-fix -f Modelfile
|
||||
```
|
||||
|
||||
### From Scratch
|
||||
|
||||
Install git lfs and clone the model you're interested in
|
||||
|
||||
```bash
|
||||
# Make sure you have git-lfs installed (https://git-lfs.com)
|
||||
git lfs install
|
||||
|
||||
git clone https://huggingface.co/bartowski/Starling-LM-7B-beta-GGUF
|
||||
```
|
||||
|
||||
Create a modelfile
|
||||
|
||||
```Dockerfile
|
||||
# Modelfile
|
||||
FROM "./path/to/gguf"
|
||||
|
||||
TEMPLATE """{{ if .Prompt }}<|im_start|>
|
||||
{{ .Prompt }}<|im_end|>
|
||||
{{ end }}
|
||||
"""
|
||||
|
||||
SYSTEM You are OpenCoder, created by OpenCoder Team.
|
||||
|
||||
PARAMETER stop <|im_start|>
|
||||
PARAMETER stop <|im_end|>
|
||||
PARAMETER stop <|fim_prefix|>
|
||||
PARAMETER stop <|fim_middle|>
|
||||
PARAMETER stop <|fim_suffix|>
|
||||
PARAMETER stop <|fim_end|>
|
||||
```
|
||||
|
||||
Build the model
|
||||
|
||||
```bash
|
||||
ollama create "Starling-LM-7B-beta-Q6_K" -f Modelfile
|
||||
```
|
||||
|
||||
Run the model
|
||||
|
||||
```bash
|
||||
ollama run Starling-LM-7B-beta-Q6_K:latest
|
||||
```
|
||||
|
||||
### Discovering models
|
||||
|
||||
Check out Hugging Face's leaderboard: https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
|
||||
|
||||
1. Select the model type you're after
|
||||
2. Drag the number of parameters slider to a range you can run
|
||||
3. Click the top few and read about them.
|
||||
|
||||
### Custom models from safetensor files
|
||||
|
||||
<https://www.theregister.com/2024/07/14/quantization_llm_feature/>
|
||||
|
||||
Setup the repo:
|
||||
|
||||
```bash
|
||||
# Setup
|
||||
git clone https://github.com/ggerganov/llama.cpp.git
|
||||
cd ~/llama.cpp
|
||||
cmake -B build
|
||||
cmake --build build --config Release -j $(nproc)
|
||||
python3 -m venv venv && source venv/bin/activate
|
||||
pip install -r requirements.txt
|
||||
huggingface-cli login #necessary to download gated models
|
||||
python convert_hf_to_gguf_update.py $(cat ~/.cache/huggingface/token)
|
||||
```
|
||||
|
||||
Convert models to gguf:
|
||||
|
||||
```bash
|
||||
# Copy the model title from hugging face
|
||||
export MODEL_NAME=
|
||||
|
||||
# Create a folder to clone the model into
|
||||
mkdir -p models/$MODEL_NAME
|
||||
|
||||
# Download the current head for the model
|
||||
huggingface-cli download $MODEL_NAME --local-dir models/$MODEL_NAME
|
||||
|
||||
# Or get the f16 quantized gguf
|
||||
wget -P models/$MODEL_NAME https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf/resolve/main/llava-llama-3-8b-v1_1-f16.gguf
|
||||
|
||||
# Convert model from hugging face to gguf, quant 8
|
||||
python3 convert_hf_to_gguf.py models/$MODEL_NAME --outfile models/$MODEL_NAME.gguf
|
||||
|
||||
# Run ./llama-quantize to see available quants
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q4_K.gguf 15
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q5_K.gguf 17
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q6_K.gguf 18
|
||||
./llama-quantize models/$MODEL_NAME.gguf models/$MODEL_NAME-Q8_0.gguf 7
|
||||
|
||||
# Copy to your localai models folder and restart
|
||||
scp models/$MODEL_NAME-Q5_K.gguf localai:/models/
|
||||
|
||||
# View output
|
||||
tree -phugL 2 models
|
||||
```
|
||||
@@ -1,264 +0,0 @@
|
||||
# Local AI with Anything LLM
|
||||
|
||||
<https://github.com/Mintplex-Labs/anything-llm/blob/master/docker/HOW_TO_USE_DOCKER.md>
|
||||
|
||||
<https://localai.io/>
|
||||
|
||||
## Running with Podman
|
||||
|
||||
This installs both Local AI and Anything LLM as backend/frontend services.
|
||||
|
||||
```bash
|
||||
podman network create localai
|
||||
|
||||
# Local AI
|
||||
podman run \
|
||||
-d \
|
||||
-p 127.0.0.1:8080:8080 \
|
||||
--network localai \
|
||||
--name local-ai \
|
||||
-v /models:/build/models \
|
||||
quay.io/go-skynet/local-ai:latest-cpu
|
||||
|
||||
# Anything LLM Interface
|
||||
export STORAGE_LOCATION=/anything-llm && \
|
||||
mkdir -p $STORAGE_LOCATION && \
|
||||
touch "$STORAGE_LOCATION/.env" && \
|
||||
chown -R 1000:1000 $STORAGE_LOCATION && \
|
||||
podman run \
|
||||
-d \
|
||||
-p 127.0.0.1:3001:3001 \
|
||||
--name anything-llm \
|
||||
--network localai \
|
||||
--cap-add SYS_ADMIN \
|
||||
-v ${STORAGE_LOCATION}:/app/server/storage \
|
||||
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
### Quadlets with Podlet
|
||||
|
||||
Note: on Arch Linux the location is `/etc/containers/systemd/`.
|
||||
|
||||
<https://wiki.archlinux.org/title/Podman>
|
||||
|
||||
```bash
|
||||
podman run --rm ghcr.io/containers/podlet --install --description "Local AI Network" \
|
||||
podman network create localai
|
||||
|
||||
podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 127.0.0.1:8080:8080 \
|
||||
--network localai \
|
||||
--name local-ai \
|
||||
-v /models:/build/models \
|
||||
quay.io/go-skynet/local-ai:latest-cpu
|
||||
|
||||
export STORAGE_LOCATION=/anything-llm && \
|
||||
podman run --rm ghcr.io/containers/podlet --install --description "Anything LLM" \
|
||||
podman run \
|
||||
-d \
|
||||
-p 127.0.0.1:3001:3001 \
|
||||
--name anything-llm \
|
||||
--network localai \
|
||||
--cap-add SYS_ADMIN \
|
||||
-v ${STORAGE_LOCATION}:/app/server/storage \
|
||||
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
|
||||
-e STORAGE_DIR="/app/server/storage" \
|
||||
docker.io/mintplexlabs/anythingllm
|
||||
```
|
||||
|
||||
Make sure to add
|
||||
|
||||
```conf
|
||||
[Service]
|
||||
Restart=always
|
||||
```
|
||||
|
||||
To the service to have them autostart.
|
||||
|
||||
Put the generated files in `/usr/share/containers/systemd/`.
|
||||
|
||||
## Models
|
||||
|
||||
Example configs can be found here:
|
||||
|
||||
<https://github.com/mudler/LocalAI/tree/9099d0c77e9e52f4a63c53aa546cc47f1e0cfdb1/gallery>
|
||||
|
||||
### Config
|
||||
|
||||
```yaml
|
||||
name: llama-3.2
|
||||
parameters:
|
||||
model: huggingface/Llama-3.2-3B-Instruct-f16.gguf
|
||||
temperature: 0.6
|
||||
backend: llama-cpp
|
||||
# Default context size
|
||||
context_size: 8192
|
||||
threads: 16
|
||||
```
|
||||
|
||||
### Chat
|
||||
|
||||
llama-3.2-3b-instruct:q8_0
|
||||
|
||||
### Code
|
||||
|
||||
<https://huggingface.co/bartowski/Codestral-22B-v0.1-GGUF/tree/main>
|
||||
|
||||
### Agent
|
||||
|
||||
llama-3.2-3b-instruct:q8_0
|
||||
|
||||
## Podman systemd service
|
||||
|
||||
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
||||
|
||||
```bash
|
||||
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
||||
unzip awscliv2.zip
|
||||
./aws/install
|
||||
|
||||
# Configure default credentials
|
||||
aws configure
|
||||
```
|
||||
|
||||
Open http/s in firewalld:
|
||||
|
||||
```bash
|
||||
firewall-cmd --permanent --zone=public --add-service=http
|
||||
firewall-cmd --permanent --zone=public --add-service=https
|
||||
firewall-cmd --reload
|
||||
```
|
||||
|
||||
Here are the detailed instructions for installing and setting up Nginx on Fedora Linux with Certbot
|
||||
using the Route53 DNS challenge to put in front of a service called "Anything LLM" running on port
|
||||
3001 with WebSockets. The domain will be chatreesept.reeseapps.com.
|
||||
|
||||
1. Install Nginx:
|
||||
```
|
||||
dnf install -y nginx
|
||||
```
|
||||
|
||||
2. Start and enable Nginx service:
|
||||
```
|
||||
systemctl enable --now nginx
|
||||
```
|
||||
|
||||
3. Install Certbot and the Route53 DNS plugin:
|
||||
```
|
||||
dnf install -y certbot python3-certbot-dns-route53
|
||||
```
|
||||
|
||||
4. Request a certificate for your domain using the Route53 DNS challenge:
|
||||
```
|
||||
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
||||
```
|
||||
Follow the prompts to provide your Route53 credentials and email address.
|
||||
|
||||
5. Configure Nginx for your domain: Create a new Nginx configuration file for your domain:
|
||||
```
|
||||
vim /etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
||||
```
|
||||
|
||||
Add the following configuration to the file:
|
||||
```
|
||||
keepalive_timeout 1h;
|
||||
send_timeout 1h;
|
||||
client_body_timeout 1h;
|
||||
client_header_timeout 1h;
|
||||
proxy_connect_timeout 1h;
|
||||
proxy_read_timeout 1h;
|
||||
proxy_send_timeout 1h;
|
||||
|
||||
server {
|
||||
listen 80;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
location / {
|
||||
return 301 https://$host$request_uri;
|
||||
}
|
||||
}
|
||||
|
||||
server {
|
||||
listen 443 ssl;
|
||||
server_name chatreesept.reeseapps.com;
|
||||
|
||||
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
||||
|
||||
location / {
|
||||
client_max_body_size 50m;
|
||||
|
||||
proxy_pass http://localhost:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
proxy_set_header Host $host;
|
||||
proxy_cache_bypass $http_upgrade;
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
6. Test your Nginx configuration for syntax errors:
|
||||
```
|
||||
nginx -t
|
||||
```
|
||||
If there are no errors, reload Nginx to apply the changes:
|
||||
```
|
||||
systemctl reload nginx
|
||||
```
|
||||
|
||||
7. Set up automatic certificate renewal: Add the following line to your crontab to renew the
|
||||
certificate daily:
|
||||
```
|
||||
sudo crontab -e
|
||||
```
|
||||
Add the following line to the end of the file:
|
||||
```
|
||||
0 0 * * * certbot renew --quiet --no-self-upgrade --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx"
|
||||
```
|
||||
|
||||
Now, your "Anything LLM" service running on port 3001 with WebSockets is accessible through the
|
||||
domain chatreesept.reeseapps.com with a valid SSL certificate from Let's Encrypt. The certificate
|
||||
will be automatically renewed daily.
|
||||
|
||||
## Nginx
|
||||
|
||||
```bash
|
||||
certbot-3 certonly --dns-route53 -d chatreesept.reeseapps.com
|
||||
```
|
||||
|
||||
Make sure to add the following timeout configurations to your http block:
|
||||
|
||||
```conf
|
||||
server {
|
||||
# Enable websocket connections for agent protocol.
|
||||
location ~* ^/api/agent-invocation/(.*) {
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "Upgrade";
|
||||
}
|
||||
|
||||
listen 80;
|
||||
server_name [insert FQDN here];
|
||||
location / {
|
||||
# Prevent timeouts on long-running requests.
|
||||
proxy_connect_timeout 605;
|
||||
proxy_send_timeout 605;
|
||||
proxy_read_timeout 605;
|
||||
send_timeout 605;
|
||||
keepalive_timeout 605;
|
||||
|
||||
# Enable readable HTTP Streaming for LLM streamed responses
|
||||
proxy_buffering off;
|
||||
proxy_cache off;
|
||||
|
||||
# Proxy your locally running service
|
||||
proxy_pass http://0.0.0.0:3001;
|
||||
}
|
||||
}
|
||||
```
|
||||
@@ -30,15 +30,15 @@ stream_ssl:
|
||||
port: 443
|
||||
protocol: https
|
||||
- external:
|
||||
domain: reesimulate.reeseapps.com
|
||||
domain: ollama.reeseapps.com
|
||||
internal:
|
||||
domain: gamebox.reeselink.com
|
||||
domain: ollama.reeselink.com
|
||||
port: 443
|
||||
protocol: https
|
||||
- external:
|
||||
domain: chatreesept.reeseapps.com
|
||||
internal:
|
||||
domain: gamebox.reeselink.com
|
||||
domain: localai.reeselink.com
|
||||
port: 443
|
||||
protocol: https
|
||||
|
||||
|
||||
Reference in New Issue
Block a user