update litellm docs
All checks were successful
Podman DDNS Image / build-and-push-ddns (push) Successful in 1m22s
All checks were successful
Podman DDNS Image / build-and-push-ddns (push) Successful in 1m22s
This commit is contained in:
@@ -9,9 +9,8 @@
|
||||
- [Convert litellm compose spec to quadlets](#convert-litellm-compose-spec-to-quadlets)
|
||||
- [Create the litellm.env file](#create-the-litellmenv-file)
|
||||
- [Start and enable your systemd quadlet](#start-and-enable-your-systemd-quadlet)
|
||||
- [Install via Ansible](#install-via-ansible)
|
||||
- [Expose litellm](#expose-litellm)
|
||||
- [Using LiteLLM](#using-litellm)
|
||||
- [Adding Models](#adding-models)
|
||||
- [Testing Models](#testing-models)
|
||||
- [Backup litellm](#backup-litellm)
|
||||
- [Upgrade litellm](#upgrade-litellm)
|
||||
@@ -110,63 +109,28 @@ journalctl --user -u litellm -f
|
||||
systemctl --user enable --now podman-auto-update.timer
|
||||
```
|
||||
|
||||
### Install via Ansible
|
||||
|
||||
Preview changes with a dry run:
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/inventory.yaml active/container_litellm/playbook.yml --check --diff
|
||||
```
|
||||
|
||||
Run the playbook from the Homelab root:
|
||||
|
||||
```bash
|
||||
ansible-playbook -i ansible/inventory.yaml active/container_litellm/playbook.yml
|
||||
```
|
||||
|
||||
This copies the quadlets, config, reloads the systemd user daemon, and starts both `litellm-db` and `litellm` services as the `ai` user.
|
||||
|
||||
### Expose litellm
|
||||
|
||||
1. If you need a domain, follow the [DDNS instructions](/active/container_ddns/ddns.md#install-a-new-ddns-service)
|
||||
2. For a web service, follow the [Caddy instructions](/active/container_caddy/caddy.md#adding-a-new-caddy-record)
|
||||
3. Finally, follow your OS's guide for opening ports via its firewall service.
|
||||
|
||||
## Using LiteLLM
|
||||
|
||||
### Adding Models
|
||||
|
||||
```json
|
||||
// qwen3.5-35b-a3b-thinking
|
||||
{
|
||||
"temperature": 1,
|
||||
"top_p": 0.95,
|
||||
"presence_penalty": 1.5,
|
||||
"extra_body": {
|
||||
"top_k": 20,
|
||||
"min_p": 0,
|
||||
"repetition_penalty": 1,
|
||||
"chat_template_kwargs": {
|
||||
"enable_thinking": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// qwen3.5-35b-a3b-coding
|
||||
{
|
||||
"temperature": 0.6,
|
||||
"top_p": 0.95,
|
||||
"presence_penalty": 0,
|
||||
"extra_body": {
|
||||
"top_k": 20,
|
||||
"min_p": 0,
|
||||
"repetition_penalty": 1,
|
||||
"chat_template_kwargs": {
|
||||
"enable_thinking": true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// qwen3.5-35b-a3b-instruct
|
||||
{
|
||||
"temperature": 0.7,
|
||||
"top_p": 0.8,
|
||||
"presence_penalty": 1.5,
|
||||
"extra_body": {
|
||||
"top_k": 20,
|
||||
"min_p": 0,
|
||||
"repetition_penalty": 1,
|
||||
"chat_template_kwargs": {
|
||||
"enable_thinking": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Testing Models
|
||||
|
||||
```bash
|
||||
@@ -191,7 +155,7 @@ curl -L -X POST 'https://aipi.reeseapps.com/v1/chat/completions' \
|
||||
|
||||
## Backup litellm
|
||||
|
||||
Follow the [Borg Backup instructions](/active/systemd_borg/borg.md#set-up-a-client-for-backup)
|
||||
Follow the [Borg Backup instructions](/active/software_borg/borg.md#set-up-a-client-for-backup)
|
||||
|
||||
## Upgrade litellm
|
||||
|
||||
|
||||
Reference in New Issue
Block a user