264 lines
6.5 KiB
Markdown
264 lines
6.5 KiB
Markdown
# Local AI with Anything LLM
|
|
|
|
<https://github.com/Mintplex-Labs/anything-llm/blob/master/docker/HOW_TO_USE_DOCKER.md>
|
|
|
|
<https://localai.io/>
|
|
|
|
## Running with Podman
|
|
|
|
This installs both Local AI and Anything LLM as backend/frontend services.
|
|
|
|
```bash
|
|
podman network create localai
|
|
|
|
# Local AI
|
|
podman run \
|
|
-d \
|
|
-p 127.0.0.1:8080:8080 \
|
|
--network localai \
|
|
--name local-ai \
|
|
-v /models:/build/models \
|
|
quay.io/go-skynet/local-ai:latest-cpu
|
|
|
|
# Anything LLM Interface
|
|
export STORAGE_LOCATION=/anything-llm && \
|
|
mkdir -p $STORAGE_LOCATION && \
|
|
touch "$STORAGE_LOCATION/.env" && \
|
|
chown -R 1000:1000 $STORAGE_LOCATION && \
|
|
podman run \
|
|
-d \
|
|
-p 127.0.0.1:3001:3001 \
|
|
--name anything-llm \
|
|
--network localai \
|
|
--cap-add SYS_ADMIN \
|
|
-v ${STORAGE_LOCATION}:/app/server/storage \
|
|
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
|
|
-e STORAGE_DIR="/app/server/storage" \
|
|
mintplexlabs/anythingllm
|
|
```
|
|
|
|
### Quadlets with Podlet
|
|
|
|
Note: on Arch Linux the location is `/etc/containers/systemd/`.
|
|
|
|
<https://wiki.archlinux.org/title/Podman>
|
|
|
|
```bash
|
|
podman run --rm ghcr.io/containers/podlet --install --description "Local AI Network" \
|
|
podman network create localai
|
|
|
|
podman run --rm ghcr.io/containers/podlet --install --description "Local AI" \
|
|
podman run \
|
|
-d \
|
|
-p 127.0.0.1:8080:8080 \
|
|
--network localai \
|
|
--name local-ai \
|
|
-v /models:/build/models \
|
|
quay.io/go-skynet/local-ai:latest-cpu
|
|
|
|
export STORAGE_LOCATION=/anything-llm && \
|
|
podman run --rm ghcr.io/containers/podlet --install --description "Anything LLM" \
|
|
podman run \
|
|
-d \
|
|
-p 127.0.0.1:3001:3001 \
|
|
--name anything-llm \
|
|
--network localai \
|
|
--cap-add SYS_ADMIN \
|
|
-v ${STORAGE_LOCATION}:/app/server/storage \
|
|
-v ${STORAGE_LOCATION}/.env:/app/server/.env \
|
|
-e STORAGE_DIR="/app/server/storage" \
|
|
docker.io/mintplexlabs/anythingllm
|
|
```
|
|
|
|
Make sure to add
|
|
|
|
```conf
|
|
[Service]
|
|
Restart=always
|
|
```
|
|
|
|
To the service to have them autostart.
|
|
|
|
Put the generated files in `/usr/share/containers/systemd/`.
|
|
|
|
## Models
|
|
|
|
Example configs can be found here:
|
|
|
|
<https://github.com/mudler/LocalAI/tree/9099d0c77e9e52f4a63c53aa546cc47f1e0cfdb1/gallery>
|
|
|
|
### Config
|
|
|
|
```yaml
|
|
name: llama-3.2
|
|
parameters:
|
|
model: huggingface/Llama-3.2-3B-Instruct-f16.gguf
|
|
temperature: 0.6
|
|
backend: llama-cpp
|
|
# Default context size
|
|
context_size: 8192
|
|
threads: 16
|
|
```
|
|
|
|
### Chat
|
|
|
|
llama-3.2-3b-instruct:q8_0
|
|
|
|
### Code
|
|
|
|
<https://huggingface.co/bartowski/Codestral-22B-v0.1-GGUF/tree/main>
|
|
|
|
### Agent
|
|
|
|
llama-3.2-3b-instruct:q8_0
|
|
|
|
## Podman systemd service
|
|
|
|
See [generating AWS credentials](cloud/graduated/aws_iam/README.md)
|
|
|
|
```bash
|
|
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
|
|
unzip awscliv2.zip
|
|
./aws/install
|
|
|
|
# Configure default credentials
|
|
aws configure
|
|
```
|
|
|
|
Open http/s in firewalld:
|
|
|
|
```bash
|
|
firewall-cmd --permanent --zone=public --add-service=http
|
|
firewall-cmd --permanent --zone=public --add-service=https
|
|
firewall-cmd --reload
|
|
```
|
|
|
|
Here are the detailed instructions for installing and setting up Nginx on Fedora Linux with Certbot
|
|
using the Route53 DNS challenge to put in front of a service called "Anything LLM" running on port
|
|
3001 with WebSockets. The domain will be chatreesept.reeseapps.com.
|
|
|
|
1. Install Nginx:
|
|
```
|
|
dnf install -y nginx
|
|
```
|
|
|
|
2. Start and enable Nginx service:
|
|
```
|
|
systemctl enable --now nginx
|
|
```
|
|
|
|
3. Install Certbot and the Route53 DNS plugin:
|
|
```
|
|
dnf install -y certbot python3-certbot-dns-route53
|
|
```
|
|
|
|
4. Request a certificate for your domain using the Route53 DNS challenge:
|
|
```
|
|
certbot certonly --dns-route53 -d chatreesept.reeseapps.com
|
|
```
|
|
Follow the prompts to provide your Route53 credentials and email address.
|
|
|
|
5. Configure Nginx for your domain: Create a new Nginx configuration file for your domain:
|
|
```
|
|
vim /etc/nginx/conf.d/chatreesept.reeseapps.com.conf
|
|
```
|
|
|
|
Add the following configuration to the file:
|
|
```
|
|
keepalive_timeout 1h;
|
|
send_timeout 1h;
|
|
client_body_timeout 1h;
|
|
client_header_timeout 1h;
|
|
proxy_connect_timeout 1h;
|
|
proxy_read_timeout 1h;
|
|
proxy_send_timeout 1h;
|
|
|
|
server {
|
|
listen 80;
|
|
server_name chatreesept.reeseapps.com;
|
|
|
|
location / {
|
|
return 301 https://$host$request_uri;
|
|
}
|
|
}
|
|
|
|
server {
|
|
listen 443 ssl;
|
|
server_name chatreesept.reeseapps.com;
|
|
|
|
ssl_certificate /etc/letsencrypt/live/chatreesept.reeseapps.com/fullchain.pem;
|
|
ssl_certificate_key /etc/letsencrypt/live/chatreesept.reeseapps.com/privkey.pem;
|
|
|
|
location / {
|
|
client_max_body_size 50m;
|
|
|
|
proxy_pass http://localhost:3001;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection "upgrade";
|
|
proxy_set_header Host $host;
|
|
proxy_cache_bypass $http_upgrade;
|
|
}
|
|
}
|
|
```
|
|
|
|
6. Test your Nginx configuration for syntax errors:
|
|
```
|
|
nginx -t
|
|
```
|
|
If there are no errors, reload Nginx to apply the changes:
|
|
```
|
|
systemctl reload nginx
|
|
```
|
|
|
|
7. Set up automatic certificate renewal: Add the following line to your crontab to renew the
|
|
certificate daily:
|
|
```
|
|
sudo crontab -e
|
|
```
|
|
Add the following line to the end of the file:
|
|
```
|
|
0 0 * * * certbot renew --quiet --no-self-upgrade --pre-hook "systemctl stop nginx" --post-hook "systemctl start nginx"
|
|
```
|
|
|
|
Now, your "Anything LLM" service running on port 3001 with WebSockets is accessible through the
|
|
domain chatreesept.reeseapps.com with a valid SSL certificate from Let's Encrypt. The certificate
|
|
will be automatically renewed daily.
|
|
|
|
## Nginx
|
|
|
|
```bash
|
|
certbot-3 certonly --dns-route53 -d chatreesept.reeseapps.com
|
|
```
|
|
|
|
Make sure to add the following timeout configurations to your http block:
|
|
|
|
```conf
|
|
server {
|
|
# Enable websocket connections for agent protocol.
|
|
location ~* ^/api/agent-invocation/(.*) {
|
|
proxy_pass http://0.0.0.0:3001;
|
|
proxy_http_version 1.1;
|
|
proxy_set_header Upgrade $http_upgrade;
|
|
proxy_set_header Connection "Upgrade";
|
|
}
|
|
|
|
listen 80;
|
|
server_name [insert FQDN here];
|
|
location / {
|
|
# Prevent timeouts on long-running requests.
|
|
proxy_connect_timeout 605;
|
|
proxy_send_timeout 605;
|
|
proxy_read_timeout 605;
|
|
send_timeout 605;
|
|
keepalive_timeout 605;
|
|
|
|
# Enable readable HTTP Streaming for LLM streamed responses
|
|
proxy_buffering off;
|
|
proxy_cache off;
|
|
|
|
# Proxy your locally running service
|
|
proxy_pass http://0.0.0.0:3001;
|
|
}
|
|
}
|
|
``` |