364 lines
8.8 KiB
Markdown
364 lines
8.8 KiB
Markdown
# Ollama
|
|
|
|
- [Ollama](#ollama)
|
|
- [Run natively with GPU support](#run-natively-with-gpu-support)
|
|
- [Unsticking models stuck in "Stopping"](#unsticking-models-stuck-in-stopping)
|
|
- [Run Anything LLM Interface](#run-anything-llm-interface)
|
|
- [Anything LLM Quadlet with Podlet](#anything-llm-quadlet-with-podlet)
|
|
- [Now with Nginx and Certbot](#now-with-nginx-and-certbot)
|
|
- [Custom Models](#custom-models)
|
|
- [From Existing Model](#from-existing-model)
|
|
- [From Scratch](#from-scratch)
|
|
- [Converting to gguf](#converting-to-gguf)
|
|
|
|
<https://github.com/ollama/ollama>
|
|
|
|
## Run natively with GPU support
|
|
|
|
<https://ollama.com/download/linux>
|
|
|
|
<https://ollama.com/library>
|
|
|
|
```bash
|
|
# Install script
|
|
curl -fsSL https://ollama.com/install.sh | sh
|
|
# Check service is running
|
|
systemctl status ollama
|
|
```
|
|
|
|
Remember to add `Environment="OLLAMA_HOST=0.0.0.0"` to `/etc/systemd/system/ollama.service` to
|
|
make it accessible on the network.
|
|
|
|
For Radeon 6000 cards you'll need to add `Environment="HSA_OVERRIDE_GFX_VERSION=10.3.0"` as well.
|
|
|
|
```bash
|
|
# Pull models
|
|
# Try to use higher parameter models. Grab the q5_K_M variant at minimum.
|
|
|
|
# For a 24GB VRAM Card I'd recommend:
|
|
|
|
# Anything-LLM Coding
|
|
ollama pull qwen2.5-coder:14b-instruct-q5_K_M
|
|
# Anything-LLM Math
|
|
ollama pull qwen2-math:7b-instruct-fp16
|
|
# Anything-LLM Chat
|
|
ollama pull llama3.2-vision:11b-instruct-q8_0
|
|
|
|
# VSCode Continue Autocomplete
|
|
ollama pull starcoder2:15b-q5_K_M
|
|
# VSCode Continue Chat
|
|
ollama pull llama3.1:8b-instruct-fp16
|
|
# VSCode Continue Embedder
|
|
ollama pull nomic-embed-text:137m-v1.5-fp16
|
|
```
|
|
|
|
Note your ollama instance will be available to podman containers via `http://host.containers.internal:11434`
|
|
|
|
## Unsticking models stuck in "Stopping"
|
|
|
|
```bash
|
|
ollama ps | grep -i stopping
|
|
pgrep ollama | xargs -I '%' sh -c 'kill %'
|
|
```
|
|
|
|
## Run Anything LLM Interface
|
|
|
|
```bash
|
|
podman run \
|
|
-d \
|
|
-p 3001:3001 \
|
|
--name anything-llm \
|
|
--cap-add SYS_ADMIN \
|
|
-v anything-llm:/app/server \
|
|
-e STORAGE_DIR="/app/server/storage" \
|
|
docker.io/mintplexlabs/anythingllm
|
|
```
|
|
|
|
This should now be accessible on port 3001. Note, you'll need to allow traffic between podman
|
|
and the host:
|
|
|
|
Use `podman network ls` to see which networks podman is running on and `podman network inspect`
|
|
to get the IP address range. Then allow traffic from that range to port 11434 (ollama):
|
|
|
|
```bash
|
|
ufw allow from 10.89.0.1/24 to any port 11434
|
|
```
|
|
|
|
## Anything LLM Quadlet with Podlet
|
|
|
|
```bash
|
|
podman run --rm ghcr.io/containers/podlet --install --description "Anything LLM" \
|
|
podman run \
|
|
-d \
|
|
-p 3001:3001 \
|
|
--name anything-llm \
|
|
--cap-add SYS_ADMIN \
|
|
--restart always \
|
|
-v anything-llm:/app/server \
|
|
-e STORAGE_DIR="/app/server/storage" \
|
|
docker.io/mintplexlabs/anythingllm
|
|
```
|
|
|
|
To the service to have them autostart.
|
|
|
|
Put the generated files in `/usr/share/containers/systemd/`.
|
|
|
|
## Now with Nginx and Certbot
|
|
|
|
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
|
|
|
```bash
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
|
unzip awscliv2.zip
|
|
./aws/install
|
|
|
|
# Configure default credentials
|
|
aws configure
|
|
```
|
|
|
|
Open http/s in firewalld:
|
|
|
|
```bash
|
|
# Remember to firewall-cmd --set-default-zone=public
|
|
firewall-cmd --permanent --zone=public --add-service=http
|
|
firewall-cmd --permanent --zone=public --add-service=https
|
|
firewall-cmd --reload
|
|
|
|
# or
|
|
ufw allow 80/tcp
|
|
ufw allow 443/tcp
|
|
```
|
|
|
|
Here are the detailed instructions for installing and setting up Nginx on Fedora Linux with Certbot
|
|
using the Route53 DNS challenge to put in front of a service called "Anything LLM" running on port
|
|
3001 with WebSockets. The domain will be chatreesept.reeseapps.com.
|
|
|
|
1. Install Nginx:
|
|
|
|
```
|
|
dnf install -y nginx
|
|
```
|
|
|
|
2. Start and enable Nginx service:
|
|
|
|
```
|
|
systemctl enable --now nginx
|
|
```
|
|
|
|
3. Install Certbot and the Route53 DNS plugin:
|
|
|
|
```
|
|
# Fedora
|
|
dnf install -y certbot python3-certbot-dns-route53
|
|
|
|
# Arch
|
|
pacman -S certbot certbot-dns-route53
|
|
```
|
|
|
|
4. Request a certificate for your domain using the Route53 DNS challenge:
|
|
|
|
```
|
|
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
|
```
|
|
|
|
Follow the prompts to provide your Route53 credentials and email address.
|
|
|
|
5. Configure Nginx for your domain: Create a new Nginx configuration file for your domain:
|
|
|
|
Update your nginx conf with the following
|
|
|
|
```
|
|
vim /etc/nginx/nginx.conf
|
|
```
|
|
|
|
```
|
|
keepalive_timeout 1h;
|
|
send_timeout 1h;
|
|
client_body_timeout 1h;
|
|
client_header_timeout 1h;
|
|
proxy_connect_timeout 1h;
|
|
proxy_read_timeout 1h;
|
|
proxy_send_timeout 1h;
|
|
```
|
|
|
|
```
|
|
vim /etc/nginx/conf.d/ollama.reeselink.com.conf
|
|
```
|
|
|
|
```
|
|
server {
|
|
listen 80;
|
|
server_name ollama.reeselink.com;
|
|
|
|
location / {
|
|
return 301 https://$host$request_uri;
|
|
}
|
|
}
|
|
|
|
server {
|
|
listen 443 ssl;
|
|
server_name ollama.reeselink.com;
|
|
|
|
ssl_certificate /etc/letsencrypt/live/ollama.reeselink.com/fullchain.pem;
|
|
ssl_certificate_key /etc/letsencrypt/live/ollama.reeselink.com/privkey.pem;
|
|
|
|
location / {
|
|
proxy_pass http://localhost:11434;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection "upgrade";
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
proxy_buffering off;
|
|
}
|
|
}
|
|
```
|
|
|
|
```
|
|
vim /etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
|
```
|
|
|
|
Add the following configuration to the file:
|
|
```
|
|
server {
|
|
listen 80;
|
|
server_name chatreesept.reeseapps.com;
|
|
|
|
location / {
|
|
return 301 https://$host$request_uri;
|
|
}
|
|
}
|
|
|
|
server {
|
|
listen 443 ssl;
|
|
server_name chatreesept.reeseapps.com;
|
|
|
|
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
|
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
|
|
|
location / {
|
|
client_max_body_size 50m;
|
|
|
|
proxy_pass http://localhost:3001;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection "upgrade";
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
proxy_buffering off;
|
|
}
|
|
}
|
|
``
|
|
|
|
6. Test your Nginx configuration for syntax errors:
|
|
|
|
```
|
|
nginx -t
|
|
```
|
|
|
|
If there are no errors, reload Nginx to apply the changes:
|
|
|
|
```
|
|
systemctl reload nginx
|
|
```
|
|
|
|
7. Set up automatic certificate renewal: Add the following line to your crontab to renew the
|
|
certificate daily:
|
|
|
|
```
|
|
|
|
pacman -S cronie
|
|
sudo crontab -e
|
|
```
|
|
|
|
Add the following line to the end of the file:
|
|
|
|
```
|
|
0 0 * * * certbot renew --quiet
|
|
```
|
|
|
|
Now, your "Anything LLM" service running on port 3001 with WebSockets is accessible through the
|
|
domain chatreesept.reeseapps.com with a valid SSL certificate from Let's Encrypt. The certificate
|
|
will be automatically renewed daily.
|
|
|
|
## Custom Models
|
|
|
|
<https://www.gpu-mart.com/blog/import-models-from-huggingface-to-ollama>
|
|
|
|
<https://www.hostinger.com/tutorials/ollama-cli-tutorial#Setting_up_Ollama_in_the_CLI>
|
|
|
|
### From Existing Model
|
|
|
|
```bash
|
|
ollama show --modelfile opencoder > Modelfile
|
|
PARAMETER num_ctx 8192
|
|
ollama create opencoder-fix -f Modelfile
|
|
```
|
|
|
|
### From Scratch
|
|
|
|
Install git lfs and clone the model you're interested in
|
|
|
|
```bash
|
|
# Make sure you have git-lfs installed (https://git-lfs.com)
|
|
git lfs install
|
|
|
|
git clone https://huggingface.co/bartowski/Starling-LM-7B-beta-GGUF
|
|
```
|
|
|
|
Create a modelfile
|
|
|
|
```
|
|
# Modelfile
|
|
FROM "./path/to/gguf"
|
|
|
|
TEMPLATE """{{ if .Prompt }}<|im_start|>
|
|
{{ .Prompt }}<|im_end|>
|
|
{{ end }}
|
|
"""
|
|
|
|
SYSTEM You are OpenCoder, created by OpenCoder Team.
|
|
|
|
PARAMETER stop <|im_start|>
|
|
PARAMETER stop <|im_end|>
|
|
PARAMETER stop <|fim_prefix|>
|
|
PARAMETER stop <|fim_middle|>
|
|
PARAMETER stop <|fim_suffix|>
|
|
PARAMETER stop <|fim_end|>
|
|
PARAMETER stop """
|
|
|
|
|
|
"""
|
|
|
|
```
|
|
|
|
Build the model
|
|
|
|
```bash
|
|
ollama create "Starling-LM-7B-beta-Q6_K" -f Modelfile
|
|
```
|
|
|
|
Run the model
|
|
|
|
```bash
|
|
ollama run Starling-LM-7B-beta-Q6_K:latest
|
|
```
|
|
|
|
## Converting to gguf
|
|
|
|
<https://www.theregister.com/2024/07/14/quantization_llm_feature/>
|
|
|
|
1. Clone the llama.cpp repository and install its dependencies:
|
|
|
|
```bash
|
|
git clone https://github.com/ggerganov/llama.cpp.git
|
|
cd ~/llama.cpp
|
|
python3 -m venv venv && source venv/bin/activate
|
|
pip3 install -r requirements.txt
|
|
|
|
mkdir ~/llama.cpp/models/mistral
|
|
huggingface-cli login #necessary to download gated models
|
|
huggingface-cli download mistralai/Mistral-7B-Instruct-v0.3 --local-dir ~/llama.cpp/models/mistral/
|
|
|
|
python3 convert_hf_to_gguf.py ~/.cache/huggingface/hub/models--infly--OpenCoder-8B-Instruct/snapshots/01badbbf10c2dfd7e2a0b5f570065ef44548576c
|
|
``` |