# Framework Desktop

## BIOS

<https://knowledgebase.frame.work/en_us/changing-memory-allocation-amd-ryzen-ai-max-300-series-By1LG5Yrll>

1. Set GPU memory to 96GB
2. Add the following kernel args to /etc/default/grub (set allocatable VRAM to
   112GB): `ttm.pages_limit=28027343750 ttm.page_pool_size=28027343750`
3. Update grub `grub2-mkconfig -o /boot/grub2/grub.cfg`
4. Reboot

## Docker

```bash
dnf config-manager addrepo \
--from-repofile https://download.docker.com/linux/fedora/docker-ce.repo

dnf install \
docker-ce \
docker-ce-cli \
containerd.io \
docker-buildx-plugin \
docker-compose-plugin
```

## Llama.cpp

```bash
# Build the llama.cpp docker image
git clone https://github.com/ggml-org/llama.cpp.git
export BUILD_TAG=$(date +"%Y-%m-%d-%H-%M-%S")
docker build -t llama-cpp-vulkan:${BUILD_TAG} -f .devops/vulkan.Dockerfile .

# Run llama server
docker run \
-d \
--restart always \
--name=llama-server \
--network=host \
--device=/dev/kfd \
--device=/dev/dri \
-v llama-server-cache:/root/.cache \
llama-cpp-vulkan:${BUILD_TAG} \
-hf ggml-org/gpt-oss-120b-GGUF  --ctx-size 0 --jinja -ub 2048 -b 2048 \
--port 8000 --host 0.0.0.0 -n -1 --n-gpu-layers 999

# Firewall
firewall-cmd --add-port=8000/tcp --permanent
firewall-cmd --reload
```

## Ollama

```bash
# Run ollama
# Will be available on port 11434
docker run \
-d \
--restart always \
--device /dev/kfd \
--device /dev/dri \
-v ollama:/root/.ollama \
-e OLLAMA_VULKAN=1 \
--name ollama \
--network host \
docker.io/ollama/ollama:0.13.5

# Run an image
docker exec -it ollama ollama run gpt-oss:20b

# Firewall
firewall-cmd --add-port=11434/tcp --permanent
firewall-cmd --reload
```

## Anything LLM

Per [the docs](https://docs.anythingllm.com/installation-docker/cloud-docker):

> Note --cap-add SYS_ADMIN is a required command if you want to scrape webpages.
> We use PuppeeteerJS to scrape websites links and --cap-add SYS_ADMIN lets us
> use sandboxed Chromium across all runtimes for best security practices

```bash
# Server will be accessible on port 3001
# Connect llama.cpp as a generic OpenAI LLM provider and use host
# http://172.17.0.1:8000/v1
# Chat model name doesn't matter.
docker run \
-d \
--restart always \
--network host \
--name anythingllm \
--cap-add SYS_ADMIN \
-v anythingllm:/app/server/storage \
-e STORAGE_DIR="/app/server/storage" \
docker.io/mintplexlabs/anythingllm

# Firewall
firewall-cmd --add-port=3001/tcp --permanent
firewall-cmd --reload
```