vibe-bot/MANUAL_TESTS.md

# Manual Testing Guide (Discord)

How to manually verify each bot command from a Discord server. The automated suite
(`uv run pytest vibe_bot/tests/ -v`) covers the internals; this guide is for
end-to-end behavior against a real Discord server and the real LLM/image/embedding
APIs. Every command below makes real API calls, so expect API usage and latency.

## Before You Start

1. A working `.env` at the repo root: `DISCORD_TOKEN`, the chat/image/edit/embedding
   endpoints with their keys and model names (see the Configuration section of
   `README.md`).
2. The bot running and online in your test server, either:

   ```bash
   uv run python -m vibe_bot.main
   ```

   or the container image (`./build.sh`, then run with `--env-file .env`).
   Watch the console output while testing — the bot logs each command trigger.
3. In the Discord Developer Portal, the bot must have **Message Content Intent**,
   **Server Members Intent**, and **Presence Intent** enabled (the bot requests all
   three at startup and will refuse to log in without them).
4. For `!speak`: `kokoro-v1.0.onnx` and `voices-v1.0.bin` in the project root
   (baked into the container image). Without them the bot still runs; only
   `!speak` degrades (it replies "TTS engine not initialized..."). `!voices` is a
   static list and works either way.
5. Use a dedicated test channel. Start every clean pass by wiping stored memory:

   ```
   !lobotomize
   ```

   Expected: "All conversation history and memory has been cleared."
   (This deletes **all** chat messages and embeddings for **all** users and bots.
   It does **not** delete the custom bots themselves or the image ETA stats.)

6. Create the two standard test bots used throughout this guide. Use lowercase
   names: incoming chat text is lowercased before being matched against the stored
   bot name, so a lowercase name matches whatever capitalization you type.

   ```
   !custom-bot alfred you are a proper british butler who speaks in short, polite sentences
   !custom-bot jarvis you are a dry, sarcastic AI assistant
   ```

## Command Reference

### `!custom-bot <name> <personality>` — create a custom bot

**Happy path**

- Send: `!custom-bot alfred you are a proper british butler`
- Expect: `Custom bot **'alfred'** has been created with personality: *you are a proper british butler*`
  followed by `You can now use this bot with: `!alfred <your message>``
- `!list-custom-bots` now includes alfred, and `!alfred <msg>` works immediately
  (the bot-name cache is invalidated on create).

**Replacing an existing name** — bot names are a global namespace, not per-user.

- Send the same command again with a different personality.
- Expect: `Custom bot **'alfred'** already existed and has been **replaced** with personality: *...*`

**Validation** (names 2–50 chars, personality 10–1000 chars):

| Send | Expected reply |
| ---- | -------------- |
| `!custom-bot a you are a test bot` (1-char name) | `Invalid bot name. Name must be between 2 and 50 characters.` |
| a 51+-character name | same |
| `!custom-bot bob short` (personality < 10 chars) | `Invalid personality. Description must be at least 10 characters.` |
| a 1001+-character personality | `Personality too long. Max 1000 characters.` |

### `!list-custom-bots` — list custom bots

- Empty state (fresh database, or after deleting every bot):
  `No custom bots have been created yet. Use `!custom-bot <name> <personality>` to create one.`
- After creating bots: `Available Custom Bots:` followed by one `* <name>` line per bot.

### `!delete-custom-bot <name>` — delete a custom bot (owner only)

- As the creator: `!delete-custom-bot jarvis`
  Expect: `Custom bot 'jarvis' has been deleted.`
  Afterwards `!jarvis hi` produces no response and `!list-custom-bots` omits it
  (the bot-name cache is invalidated on delete).
- Unknown name: `!delete-custom-bot nosuchbot`
  Expect: `Custom bot 'nosuchbot' not found.`
- Ownership: have a second Discord account create `tempbot`, then run
  `!delete-custom-bot tempbot` from a third account.
  Expect: `You can only delete your own custom bots.` (the bot is **not** deleted).
  The creator's delete should then succeed.

### Chatting with a custom bot — `!<bot_name> <message>`

This is not a registered command; messages starting with `!<known bot name> ` are
matched in `on_message` and dispatched to the chat service.

**Happy path**

- Send: `!alfred Could you fetch me some tea?`
- Expect, in order:
  1. `alfred is searching its databanks for Could you fetch me some tea?...`
  2. `alfred response`
  3. The reply, in character, under 2–3 sentences.

**Long replies** — a reply longer than 1000 characters arrives as multiple
consecutive messages (chunked at 1000 chars).

**Missing the trailing text** — `!alfred` alone (no message after the name)
matches nothing: the bot stays silent and no command error is raised.

**Memory / RAG**

1. `!alfred My favorite color is teal. Remember that.`
2. Later — different topic, ideally a different channel or day:
   `!alfred What is my favorite color?`
3. Expect: it answers teal. RAG prepends your 10 most recent messages plus up to 5
   semantically similar stored messages (similarity threshold 0.7) to the prompt.
4. Cross-bot check (shared memory is by design — history is not siloed per bot):
   `!jarvis What is this user's favorite color?` should also know.

**Tool call**

- Send: `!alfred Who is in this channel?`
- The bot may call the `get_channel_members` tool; expect
  `alfred is looking at the channel members...` before a reply that names server
  members. (Not guaranteed every time — the LLM decides whether to call it.)

**Failure path** — stop the LLM API (or point `CHAT_ENDPOINT` at a dead URL and
restart the bot), then send `!alfred hi`.
Expect: `An error occurred while processing your request.` and a traceback in the
bot console.

### `!speak <text> [--voice <name>]` — plain text-to-speech

- `!speak hello world`
  Expect: `Generating speech...` then a `speech.mp3` attachment. Play it.
- Without `--voice`, the configured `TTS_VOICE` (default `af_sarah`) is used.
- `!speak hello world --voice af_bella`
  Expect: same flow, different voice. The `--voice` flag is only recognized at the
  **very end** of the message; `!speak the --voice flag is fun` speaks the entire
  string verbatim.
- Unknown voice: `!speak hi --voice af_nobody`
  Expect: `Unknown voice 'af_nobody'. Use `!voices` to see available voices.`
- No text: `!speak --voice af_bella`
  Expect: `Please provide text to speak.`
- Over 5000 characters of text:
  Expect: `Text too long to speak. Max 5000 characters.`
- Cooldown: 3 uses per 30 seconds per user. A 4th immediate call:
  `You're using that too quickly, try again in Ns.`
- Missing TTS model files (run without the onnx/bin files):
  `TTS engine not initialized. Make sure kokoro-v1.0.onnx and voices-v1.0.bin are present.`

### `!speak <bot_name> <text> [--voice <name>]` — custom bot responds and speaks

Requires an existing custom bot; the first word is compared **exactly**
(case-sensitive) against stored bot names.

- `!speak alfred what time is it`
  Expect, in order:
  1. `**alfred** is thinking...`
  2. `**alfred**: <in-character reply>` (the LLM reply, not your text)
  3. `Generating speech for **alfred**...`
  4. `speech.mp3` containing the **bot's reply** spoken aloud.
  The exchange is also stored in history (visible later via `!history alfred`).
- `!speak alfred what time is it --voice am_puck` — same flow, am_puck voice.
- `!speak alfred` (no text after the name):
  `Please provide text for the bot to respond to.`
- `!speak nonexistent hello`: `Custom bot 'nonexistent' not found.`

### `!voices` — list TTS voices

- No arguments. Expect the static catalog grouped by category:
  en-US female (`af_alloy` … `af_sky`), en-US male (`am_adam` … `am_puck`),
  en-GB (`bf_*`, `bm_*`), fr-FR (`ff_siwis`), Italian (`if_sara`, `im_nicola`),
  Japanese (`jf_*`, `jm_kumo`), Mandarin (`zf_*`, `zm_*`), ending with
  `Use `!speak <text> --voice <voice_name>` to choose a voice.`
- Spot-check one voice per language with `!speak <text in that language> --voice <name>`
  and listen for the expected accent.

### `!doodlebob <prompt>` — generate an image

- `!doodlebob a cat sitting on the moon`
  Expect, in order:
  1. `**Doodlebob shopping for a canvas...**`
  2. `**Doodlebob selected <layout>**` — the LLM picks `square`, `portrait`, or
     `landscape` from the prompt (a tall subject should yield portrait, a wide
     scene landscape). The canvas size follows the layout.
  3. `**Doodlebob calling drone strike on <first 100 chars of the final prompt>...**`
  4. From the **second** generation onward: `**Drone ETA: ~N seconds**` (moving
     average of recorded generation times).
  5. An `image.png` attachment that matches the prompt.
  6. `**Strike complete. Image generated in N.N seconds.**`
- Prompt over 2000 characters: `Prompt too long. Max 2000 characters.`
- Cooldown: 1 per 60 seconds per user.
- Image API failure: `Failed to generate image. The server may be busy.`

### `!retcon <prompt>` — edit an attached image

- Attach an image **on the same message** as the command (drag it into the
  channel): `!retcon make it look like it is on fire [image]`
  Expect: `**Rewriting history to match make it look like it is on fire...**`
  then an edited `image.png` attachment.
- No attachment: `!retcon make it sunny`
  Expect: `Please attach an image to edit.`
  (Only real attachments count; pasted external URLs are ignored, and only
  Discord CDN hosts are downloaded, capped at 8 MB.)
- Prompt over 2000 characters: `Prompt too long. Max 2000 characters.`
- Image edit API failure: `Failed to edit the image.`

### `!talkforme <bot1> <bot2> <n> <topic>` — bot-vs-bot conversation

Requires two existing custom bots.

- `!talkforme alfred jarvis 4 the meaning of life`
  Expect: `alfred is going to talk to jarvis about "the meaning of life" for 4 replies.`
  then alternating messages headed `## alfred` / `## jarvis`, each reply in
  character. Total chat messages is **n + 1** (bot1's opener plus n loop
  replies); n is capped at 20, so at most 21 messages.
- Too few parts: `!talkforme alfred jarvis`
  Expect: `Usage: !talkforme bot1 bot2 <number> <topic>`
- Unknown bot: `!talkforme alfred nosuchbot 4 tea`
  Expect: `nosuchbot is not a real bot...`
- Non-integer count: `!talkforme alfred jarvis four tea`
  Expect: `Message limit must be an integer.`
- Topic over 500 characters: `Topic too long. Max 500 characters.`
- Cooldown: 1 per 30 seconds per user.

### `!history <bot_name>` — view a bot's chat history

- After chatting with alfred: `!history alfred`
  Expect: `Chat History for **alfred**:` followed by the latest up-to-20
  exchanges (oldest of the 20 first), each pair as:

  ```text
  <your message>
  ---
  alfred: <its reply>
  ```

  Turns from `!speak alfred ...` are included too (they are stored under the
  bot name).
- Unknown bot: `!history nosuchbot`
  Expect: `Custom bot 'nosuchbot' not found.`
- A bot with no stored messages: `No chat history found for **alfred**.`
- After `!lobotomize`: same "No chat history found" reply.

### `!debug [subcommand]` — debug menu

- `!debug` — prints the menu listing `members`, `whoami`, `tools`.
- `!debug members` — `Members in this channel (N total):` followed by an
  alphabetized roster of **server** members (Discord has no per-channel
  membership; the channel only identifies the guild), each entry showing display
  name, nickname, global name, and `[status]`. In a DM (no guild):
  `No members found in this channel.`
- `!debug whoami` — a block about your own account: Global Name (if set),
  Nickname (if set), Top Role (if not @everyone), Activities (if any, custom
  status excluded), Joined date, Username, User ID, Account Created date. Verify
  each value against your account.
- `!debug tools` — the `get_channel_members` tool's name, description, and
  parameter JSON schema.
- `!debug banana` — `Unknown debug sub-command: `banana`` plus a pointer to `!debug`.

### `!lobotomize` — wipe all stored memory

- Send `!lobotomize`.
  Expect: `All conversation history and memory has been cleared.`
- Verify:
  - `!history alfred` → `No chat history found for **alfred**.`
  - The RAG recall test above no longer remembers the fact.
  - `!list-custom-bots` is **unchanged** (bots survive; only
    `chat_messages` and `message_embeddings` are cleared).

## Suggested End-to-End Pass

A full manual pass in order:

1. `!lobotomize` — clean slate
2. `!custom-bot alfred ...` and `!custom-bot jarvis ...`
3. `!list-custom-bots` — both listed
4. `!alfred ...` — a few turns; tell it a fact to remember later
5. `!jarvis What did I tell you about ...?` — cross-bot memory
6. `!speak hello world --voice af_bella`, then `!speak alfred tell me a joke`
7. `!voices`
8. `!doodlebob ...` twice (second run should show the Drone ETA)
9. `!retcon ... [image]`
10. `!talkforme alfred jarvis 4 ...`
11. `!history alfred` — contains the steps 4–6 exchanges
12. `!debug`, `!debug members`, `!debug whoami`, `!debug tools`
13. `!delete-custom-bot jarvis` — then confirm `!jarvis hi` is silent
14. `!lobotomize` — confirm `!history alfred` is empty

## Covered Only by the Automated Suite

Not practical to verify by hand (see `uv run pytest vibe_bot/tests/ -v`):

- embedding similarity thresholds and vector math
- database migrations and the `MAX_HISTORY_MESSAGES` cleanup (needs 1000+ messages)
- message chunking edge cases
- RAG benchmark behavior (`uv run python scripts/bench_rag.py`)