# Podman litellm - [Podman litellm](#podman-litellm) - [Setup litellm Project](#setup-litellm-project) - [Install litellm](#install-litellm) - [Create the ai user](#create-the-ai-user) - [Write the litellm compose spec](#write-the-litellm-compose-spec) - [A Note on Volumes](#a-note-on-volumes) - [Convert litellm compose spec to quadlets](#convert-litellm-compose-spec-to-quadlets) - [Create the litellm.env file](#create-the-litellmenv-file) - [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet) - [Expose litellm](#expose-litellm) - [Using LiteLLM](#using-litellm) - [Adding Models](#adding-models) - [Testing Models](#testing-models) - [Backup litellm](#backup-litellm) - [Upgrade litellm](#upgrade-litellm) - [Upgrade Quadlets](#upgrade-quadlets) - [Uninstall](#uninstall) - [Notes](#notes) - [SELinux](#selinux) ## Setup litellm Project - [ ] Copy and rename this folder to active/container_litellm - [ ] Find and replace litellm with the name of the service. - [ ] Create the rootless user to run the podman containers - [ ] Write the compose.yaml spec for your service - [ ] Convert the compose.yaml spec to a quadlet - [ ] Install the quadlet on the podman server - [ ] Expose the quadlet service - [ ] Install a backup service and timer ## Install litellm ### Create the ai user ```bash # SSH into your podman server as root useradd ai loginctl enable-linger $(id -u ai) systemctl --user --machine=ai@.host enable podman-restart systemctl --user --machine=ai@.host enable --now podman.socket mkdir -p /home/ai/.config/containers/systemd ``` ### Write the litellm compose spec See the [docker run command here](https://docs.litellm.ai/docs/proxy/docker_quick_start#32-start-proxy) Edit the compose.yaml at active/container_litellm/compose/compose.yaml #### A Note on Volumes Named volumes are stored at `/home/litellm/.local/share/containers/storage/volumes/`. ### Convert litellm compose spec to quadlets Run the following to convert a compose.yaml into the various `.container` files for systemd: ```bash # Generate the systemd service podman run \ --security-opt label=disable \ --rm \ -v $(pwd)/active/container_litellm/compose:/compose \ -v $(pwd)/active/container_litellm/quadlets:/quadlets \ quay.io/k9withabone/podlet \ -f /quadlets \ -i \ --overwrite \ compose /compose/compose.yaml # Copy the files to the server export PODMAN_SERVER=ai-ai scp -r active/container_litellm/quadlets/. $PODMAN_SERVER:/home/ai/.config/containers/systemd/ ``` ### Create the litellm.env file Should look something like: ```env LITELLM_MASTER_KEY="random-string" LITELLM_SALT_KEY="random-string" UI_USERNAME="admin" UI_PASSWORD="random-string" ``` Then copy it to the server ```bash export PODMAN_SERVER=ai scp -r active/container_litellm/config.yaml $PODMAN_SERVER:/home/ai/litellm_config.yaml ssh $PODMAN_SERVER chown -R ai:ai /home/ai/litellm_config.yaml ``` ### Start and enable your systemd quadlet SSH into your podman server as root: ```bash ssh ai machinectl shell ai@ systemctl --user daemon-reload systemctl --user restart litellm journalctl --user -u litellm -f # Enable auto-update service which will pull new container images automatically every day systemctl --user enable --now podman-auto-update.timer ``` ### Expose litellm 1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service) 2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record) 3. Finally, follow your OS's guide for opening ports via its firewall service. ## Using LiteLLM ### Adding Models ```json // qwen3.5-35b-a3b-thinking { "temperature": 1, "top_p": 0.95, "presence_penalty": 1.5, "extra_body": { "top_k": 20, "min_p": 0, "repetition_penalty": 1, "chat_template_kwargs": { "enable_thinking": true } } } // qwen3.5-35b-a3b-coding { "temperature": 0.6, "top_p": 0.95, "presence_penalty": 0, "extra_body": { "top_k": 20, "min_p": 0, "repetition_penalty": 1, "chat_template_kwargs": { "enable_thinking": true } } } // qwen3.5-35b-a3b-instruct { "temperature": 0.7, "top_p": 0.8, "presence_penalty": 1.5, "extra_body": { "top_k": 20, "min_p": 0, "repetition_penalty": 1, "chat_template_kwargs": { "enable_thinking": false } } } ``` ### Testing Models ```bash # List models curl -L -X GET 'https://aipi.reeseapps.com/v1/models' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' curl -L -X POST 'https://aipi.reeseapps.com/v1/chat/completions' \ -H 'Content-Type: application/json' \ -H 'Authorization: Bearer sk-1234' \ -d '{ "model": "gpt-4o-mini", # 👈 REPLACE with 'public model name' for any db-model "messages": [ { "content": "Hey, how's it going", "role": "user" } ], }' ``` ## Backup litellm Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup) ## Upgrade litellm ### Upgrade Quadlets Upgrades should be a repeat of [writing the compose spec](#convert-litellm-compose-spec-to-quadlets) and [installing the quadlets](#start-and-enable-your-systemd-quadlet) ```bash export PODMAN_SERVER= scp -r quadlets/. $PODMAN_SERVER$:/home/litellm/.config/containers/systemd/ ssh litellm systemctl --user daemon-reload ssh litellm systemctl --user restart litellm ``` ## Uninstall ```bash # Stop the user's services systemctl --user disable podman-restart podman container stop --all systemctl --user disable --now podman.socket systemctl --user disable --now podman-auto-update.timer # Delete the user (this won't delete their home directory) # userdel might spit out an error like: # userdel: user litellm is currently used by process 591255 # kill those processes and try again userdel litellm ``` ## Notes ### SELinux :z allows a container to share a mounted volume with all other containers. :Z allows a container to reserve a mounted volume and prevents any other container from accessing.