vibe-bot/MANUAL_TESTS.md

Manual Testing Guide (Discord)

How to manually verify each bot command from a Discord server. The automated suite (uv run pytest vibe_bot/tests/ -v) covers the internals; this guide is for end-to-end behavior against a real Discord server and the real LLM/image/embedding APIs. Every command below makes real API calls, so expect API usage and latency.

Before You Start

1. A working .env at the repo root: DISCORD_TOKEN, the chat/image/edit/embedding endpoints with their keys and model names (see the Configuration section of README.md).

2. The bot running and online in your test server, either:

   uv run python -m vibe_bot.main
   

   or the container image (./build.sh, then run with --env-file .env). Watch the console output while testing — the bot logs each command trigger.

3. In the Discord Developer Portal, the bot must have Message Content Intent, Server Members Intent, and Presence Intent enabled (the bot requests all three at startup and will refuse to log in without them).

4. For !speak: kokoro-v1.0.onnx and voices-v1.0.bin in the project root (baked into the container image). Without them the bot still runs; only !speak degrades (it replies "TTS engine not initialized..."). !voices is a static list and works either way.

5. Use a dedicated test channel. Start every clean pass by wiping stored memory:

   !lobotomize
   

   Expected: "All conversation history and memory has been cleared." (This deletes all chat messages and embeddings for all users and bots. It does not delete the custom bots themselves or the image ETA stats.)

6. Create the two standard test bots used throughout this guide. Use lowercase names: incoming chat text is lowercased before being matched against the stored bot name, so a lowercase name matches whatever capitalization you type.

   !custom-bot alfred you are a proper british butler who speaks in short, polite sentences
   !custom-bot jarvis you are a dry, sarcastic AI assistant
   

Command Reference

!custom-bot <name> <personality> — create a custom bot

Happy path

• Send: !custom-bot alfred you are a proper british butler
• Expect: Custom bot **'alfred'** has been created with personality: *you are a proper british butler* followed by You can now use this bot with: !alfred ``
• !list-custom-bots now includes alfred, and !alfred <msg> works immediately (the bot-name cache is invalidated on create).

Replacing an existing name — bot names are a global namespace, not per-user.

• Send the same command again with a different personality.
• Expect: Custom bot **'alfred'** already existed and has been **replaced** with personality: *...*

Validation (names 2–50 chars, personality 10–1000 chars):

Send	Expected reply
!custom-bot a you are a test bot (1-char name)	Invalid bot name. Name must be between 2 and 50 characters.
a 51+-character name	same
!custom-bot bob short (personality < 10 chars)	Invalid personality. Description must be at least 10 characters.
a 1001+-character personality	Personality too long. Max 1000 characters.

!list-custom-bots — list custom bots

• Empty state (fresh database, or after deleting every bot): No custom bots have been created yet. Use !custom-bot to create one.
• After creating bots: Available Custom Bots: followed by one * <name> line per bot.

!delete-custom-bot <name> — delete a custom bot (owner only)

• As the creator: !delete-custom-bot jarvis Expect: Custom bot 'jarvis' has been deleted. Afterwards !jarvis hi produces no response and !list-custom-bots omits it (the bot-name cache is invalidated on delete).
• Unknown name: !delete-custom-bot nosuchbot Expect: Custom bot 'nosuchbot' not found.
• Ownership: have a second Discord account create tempbot, then run !delete-custom-bot tempbot from a third account. Expect: You can only delete your own custom bots. (the bot is not deleted). The creator's delete should then succeed.

Chatting with a custom bot — !<bot_name> <message>

This is not a registered command; messages starting with !<known bot name> are matched in on_message and dispatched to the chat service.

Happy path

• Send: !alfred Could you fetch me some tea?
• Expect, in order:
  1. alfred is searching its databanks for Could you fetch me some tea?...
  2. alfred response
  3. The reply, in character, under 2–3 sentences.

Long replies — a reply longer than 1000 characters arrives as multiple consecutive messages (chunked at 1000 chars).

Missing the trailing text — !alfred alone (no message after the name) matches nothing: the bot stays silent and no command error is raised.

Memory / RAG

1. !alfred My favorite color is teal. Remember that.
2. Later — different topic, ideally a different channel or day: !alfred What is my favorite color?
3. Expect: it answers teal. RAG prepends your 10 most recent messages plus up to 5 semantically similar stored messages (similarity threshold 0.7) to the prompt.
4. Cross-bot check (shared memory is by design — history is not siloed per bot): !jarvis What is this user's favorite color? should also know.

Tool call

• Send: !alfred Who is in this channel?
• The bot may call the get_channel_members tool; expect alfred is looking at the channel members... before a reply that names server members. (Not guaranteed every time — the LLM decides whether to call it.)

Failure path — stop the LLM API (or point CHAT_ENDPOINT at a dead URL and restart the bot), then send !alfred hi. Expect: An error occurred while processing your request. and a traceback in the bot console.

!speak <text> [--voice <name>] — plain text-to-speech

• !speak hello world Expect: Generating speech... then a speech.mp3 attachment. Play it.
• Without --voice, the configured TTS_VOICE (default af_sarah) is used.
• !speak hello world --voice af_bella Expect: same flow, different voice. The --voice flag is only recognized at the very end of the message; !speak the --voice flag is fun speaks the entire string verbatim.
• Unknown voice: !speak hi --voice af_nobody Expect: Unknown voice 'af_nobody'. Use !voices to see available voices.
• No text: !speak --voice af_bella Expect: Please provide text to speak.
• Over 5000 characters of text: Expect: Text too long to speak. Max 5000 characters.
• Cooldown: 3 uses per 30 seconds per user. A 4th immediate call: You're using that too quickly, try again in Ns.
• Missing TTS model files (run without the onnx/bin files): TTS engine not initialized. Make sure kokoro-v1.0.onnx and voices-v1.0.bin are present.

!speak <bot_name> <text> [--voice <name>] — custom bot responds and speaks

Requires an existing custom bot; the first word is compared exactly (case-sensitive) against stored bot names.

• !speak alfred what time is it Expect, in order:
  1. **alfred** is thinking...
  2. **alfred**: <in-character reply> (the LLM reply, not your text)
  3. Generating speech for **alfred**...
  4. speech.mp3 containing the bot's reply spoken aloud. The exchange is also stored in history (visible later via !history alfred).
• !speak alfred what time is it --voice am_puck — same flow, am_puck voice.
• !speak alfred (no text after the name): Please provide text for the bot to respond to.
• !speak nonexistent hello: Custom bot 'nonexistent' not found.

!voices — list TTS voices

• No arguments. Expect the static catalog grouped by category: en-US female (af_alloy … af_sky), en-US male (am_adam … am_puck), en-GB (bf_*, bm_*), fr-FR (ff_siwis), Italian (if_sara, im_nicola), Japanese (jf_*, jm_kumo), Mandarin (zf_*, zm_*), ending with Use !speak --voice <voice_name> to choose a voice.
• Spot-check one voice per language with !speak <text in that language> --voice <name> and listen for the expected accent.

!doodlebob <prompt> — generate an image

• !doodlebob a cat sitting on the moon Expect, in order:
  1. **Doodlebob shopping for a canvas...**
  2. **Doodlebob selected <layout>** — the LLM picks square, portrait, or landscape from the prompt (a tall subject should yield portrait, a wide scene landscape). The canvas size follows the layout.
  3. **Doodlebob calling drone strike on <first 100 chars of the final prompt>...**
  4. From the second generation onward: **Drone ETA: ~N seconds** (moving average of recorded generation times).
  5. An image.png attachment that matches the prompt.
  6. **Strike complete. Image generated in N.N seconds.**
• Prompt over 2000 characters: Prompt too long. Max 2000 characters.
• Cooldown: 1 per 60 seconds per user.
• Image API failure: Failed to generate image. The server may be busy.

!retcon <prompt> — edit an attached image

• Attach an image on the same message as the command (drag it into the channel): !retcon make it look like it is on fire [image] Expect: **Rewriting history to match make it look like it is on fire...** then an edited image.png attachment.
• No attachment: !retcon make it sunny Expect: Please attach an image to edit. (Only real attachments count; pasted external URLs are ignored, and only Discord CDN hosts are downloaded, capped at 8 MB.)
• Prompt over 2000 characters: Prompt too long. Max 2000 characters.
• Image edit API failure: Failed to edit the image.

!talkforme <bot1> <bot2> <n> <topic> — bot-vs-bot conversation

Requires two existing custom bots.

• !talkforme alfred jarvis 4 the meaning of life Expect: alfred is going to talk to jarvis about "the meaning of life" for 4 replies. then alternating messages headed ## alfred / ## jarvis, each reply in character. Total chat messages is n + 1 (bot1's opener plus n loop replies); n is capped at 20, so at most 21 messages.
• Too few parts: !talkforme alfred jarvis Expect: Usage: !talkforme bot1 bot2 <number> <topic>
• Unknown bot: !talkforme alfred nosuchbot 4 tea Expect: nosuchbot is not a real bot...
• Non-integer count: !talkforme alfred jarvis four tea Expect: Message limit must be an integer.
• Topic over 500 characters: Topic too long. Max 500 characters.
• Cooldown: 1 per 30 seconds per user.

!history <bot_name> — view a bot's chat history

• After chatting with alfred: !history alfred Expect: Chat History for **alfred**: followed by the latest up-to-20 exchanges (oldest of the 20 first), each pair as:

  <your message>
  ---
  alfred: <its reply>
  

  Turns from !speak alfred ... are included too (they are stored under the bot name).

• Unknown bot: !history nosuchbot Expect: Custom bot 'nosuchbot' not found.

• A bot with no stored messages: No chat history found for **alfred**.

• After !lobotomize: same "No chat history found" reply.

!debug [subcommand] — debug menu

• !debug — prints the menu listing members, whoami, tools.
• !debug members — Members in this channel (N total): followed by an alphabetized roster of server members (Discord has no per-channel membership; the channel only identifies the guild), each entry showing display name, nickname, global name, and [status]. In a DM (no guild): No members found in this channel.
• !debug whoami — a block about your own account: Global Name (if set), Nickname (if set), Top Role (if not @everyone), Activities (if any, custom status excluded), Joined date, Username, User ID, Account Created date. Verify each value against your account.
• !debug tools — the get_channel_members tool's name, description, and parameter JSON schema.
• !debug banana — Unknown debug sub-command: banana`` plus a pointer to !debug.

!lobotomize — wipe all stored memory

• Send !lobotomize. Expect: All conversation history and memory has been cleared.
• Verify:
  • !history alfred → No chat history found for **alfred**.
  • The RAG recall test above no longer remembers the fact.
  • !list-custom-bots is unchanged (bots survive; only chat_messages and message_embeddings are cleared).

Suggested End-to-End Pass

A full manual pass in order:

1. !lobotomize — clean slate
2. !custom-bot alfred ... and !custom-bot jarvis ...
3. !list-custom-bots — both listed
4. !alfred ... — a few turns; tell it a fact to remember later
5. !jarvis What did I tell you about ...? — cross-bot memory
6. !speak hello world --voice af_bella, then !speak alfred tell me a joke
7. !voices
8. !doodlebob ... twice (second run should show the Drone ETA)
9. !retcon ... [image]
10. !talkforme alfred jarvis 4 ...
11. !history alfred — contains the steps 4–6 exchanges
12. !debug, !debug members, !debug whoami, !debug tools
13. !delete-custom-bot jarvis — then confirm !jarvis hi is silent
14. !lobotomize — confirm !history alfred is empty

Covered Only by the Automated Suite

Not practical to verify by hand (see uv run pytest vibe_bot/tests/ -v):

• embedding similarity thresholds and vector math
• database migrations and the MAX_HISTORY_MESSAGES cleanup (needs 1000+ messages)
• message chunking edge cases
• RAG benchmark behavior (uv run python scripts/bench_rag.py)