complete restructure
This commit is contained in:
+299
@@ -0,0 +1,299 @@
|
||||
# Manual Testing Guide (Discord)
|
||||
|
||||
How to manually verify each bot command from a Discord server. The automated suite
|
||||
(`uv run pytest vibe_bot/tests/ -v`) covers the internals; this guide is for
|
||||
end-to-end behavior against a real Discord server and the real LLM/image/embedding
|
||||
APIs. Every command below makes real API calls, so expect API usage and latency.
|
||||
|
||||
## Before You Start
|
||||
|
||||
1. A working `.env` at the repo root: `DISCORD_TOKEN`, the chat/image/edit/embedding
|
||||
endpoints with their keys and model names (see the Configuration section of
|
||||
`README.md`).
|
||||
2. The bot running and online in your test server, either:
|
||||
|
||||
```bash
|
||||
uv run python -m vibe_bot.main
|
||||
```
|
||||
|
||||
or the container image (`./build.sh`, then run with `--env-file .env`).
|
||||
Watch the console output while testing — the bot logs each command trigger.
|
||||
3. In the Discord Developer Portal, the bot must have **Message Content Intent**,
|
||||
**Server Members Intent**, and **Presence Intent** enabled (the bot requests all
|
||||
three at startup and will refuse to log in without them).
|
||||
4. For `!speak`: `kokoro-v1.0.onnx` and `voices-v1.0.bin` in the project root
|
||||
(baked into the container image). Without them the bot still runs; only
|
||||
`!speak` degrades (it replies "TTS engine not initialized..."). `!voices` is a
|
||||
static list and works either way.
|
||||
5. Use a dedicated test channel. Start every clean pass by wiping stored memory:
|
||||
|
||||
```
|
||||
!lobotomize
|
||||
```
|
||||
|
||||
Expected: "All conversation history and memory has been cleared."
|
||||
(This deletes **all** chat messages and embeddings for **all** users and bots.
|
||||
It does **not** delete the custom bots themselves or the image ETA stats.)
|
||||
|
||||
6. Create the two standard test bots used throughout this guide. Use lowercase
|
||||
names: incoming chat text is lowercased before being matched against the stored
|
||||
bot name, so a lowercase name matches whatever capitalization you type.
|
||||
|
||||
```
|
||||
!custom-bot alfred you are a proper british butler who speaks in short, polite sentences
|
||||
!custom-bot jarvis you are a dry, sarcastic AI assistant
|
||||
```
|
||||
|
||||
## Command Reference
|
||||
|
||||
### `!custom-bot <name> <personality>` — create a custom bot
|
||||
|
||||
**Happy path**
|
||||
|
||||
- Send: `!custom-bot alfred you are a proper british butler`
|
||||
- Expect: `Custom bot **'alfred'** has been created with personality: *you are a proper british butler*`
|
||||
followed by `You can now use this bot with: `!alfred <your message>``
|
||||
- `!list-custom-bots` now includes alfred, and `!alfred <msg>` works immediately
|
||||
(the bot-name cache is invalidated on create).
|
||||
|
||||
**Replacing an existing name** — bot names are a global namespace, not per-user.
|
||||
|
||||
- Send the same command again with a different personality.
|
||||
- Expect: `Custom bot **'alfred'** already existed and has been **replaced** with personality: *...*`
|
||||
|
||||
**Validation** (names 2–50 chars, personality 10–1000 chars):
|
||||
|
||||
| Send | Expected reply |
|
||||
| ---- | -------------- |
|
||||
| `!custom-bot a you are a test bot` (1-char name) | `Invalid bot name. Name must be between 2 and 50 characters.` |
|
||||
| a 51+-character name | same |
|
||||
| `!custom-bot bob short` (personality < 10 chars) | `Invalid personality. Description must be at least 10 characters.` |
|
||||
| a 1001+-character personality | `Personality too long. Max 1000 characters.` |
|
||||
|
||||
### `!list-custom-bots` — list custom bots
|
||||
|
||||
- Empty state (fresh database, or after deleting every bot):
|
||||
`No custom bots have been created yet. Use `!custom-bot <name> <personality>` to create one.`
|
||||
- After creating bots: `Available Custom Bots:` followed by one `* <name>` line per bot.
|
||||
|
||||
### `!delete-custom-bot <name>` — delete a custom bot (owner only)
|
||||
|
||||
- As the creator: `!delete-custom-bot jarvis`
|
||||
Expect: `Custom bot 'jarvis' has been deleted.`
|
||||
Afterwards `!jarvis hi` produces no response and `!list-custom-bots` omits it
|
||||
(the bot-name cache is invalidated on delete).
|
||||
- Unknown name: `!delete-custom-bot nosuchbot`
|
||||
Expect: `Custom bot 'nosuchbot' not found.`
|
||||
- Ownership: have a second Discord account create `tempbot`, then run
|
||||
`!delete-custom-bot tempbot` from a third account.
|
||||
Expect: `You can only delete your own custom bots.` (the bot is **not** deleted).
|
||||
The creator's delete should then succeed.
|
||||
|
||||
### Chatting with a custom bot — `!<bot_name> <message>`
|
||||
|
||||
This is not a registered command; messages starting with `!<known bot name> ` are
|
||||
matched in `on_message` and dispatched to the chat service.
|
||||
|
||||
**Happy path**
|
||||
|
||||
- Send: `!alfred Could you fetch me some tea?`
|
||||
- Expect, in order:
|
||||
1. `alfred is searching its databanks for Could you fetch me some tea?...`
|
||||
2. `alfred response`
|
||||
3. The reply, in character, under 2–3 sentences.
|
||||
|
||||
**Long replies** — a reply longer than 1000 characters arrives as multiple
|
||||
consecutive messages (chunked at 1000 chars).
|
||||
|
||||
**Missing the trailing text** — `!alfred` alone (no message after the name)
|
||||
matches nothing: the bot stays silent and no command error is raised.
|
||||
|
||||
**Memory / RAG**
|
||||
|
||||
1. `!alfred My favorite color is teal. Remember that.`
|
||||
2. Later — different topic, ideally a different channel or day:
|
||||
`!alfred What is my favorite color?`
|
||||
3. Expect: it answers teal. RAG prepends your 10 most recent messages plus up to 5
|
||||
semantically similar stored messages (similarity threshold 0.7) to the prompt.
|
||||
4. Cross-bot check (shared memory is by design — history is not siloed per bot):
|
||||
`!jarvis What is this user's favorite color?` should also know.
|
||||
|
||||
**Tool call**
|
||||
|
||||
- Send: `!alfred Who is in this channel?`
|
||||
- The bot may call the `get_channel_members` tool; expect
|
||||
`alfred is looking at the channel members...` before a reply that names server
|
||||
members. (Not guaranteed every time — the LLM decides whether to call it.)
|
||||
|
||||
**Failure path** — stop the LLM API (or point `CHAT_ENDPOINT` at a dead URL and
|
||||
restart the bot), then send `!alfred hi`.
|
||||
Expect: `An error occurred while processing your request.` and a traceback in the
|
||||
bot console.
|
||||
|
||||
### `!speak <text> [--voice <name>]` — plain text-to-speech
|
||||
|
||||
- `!speak hello world`
|
||||
Expect: `Generating speech...` then a `speech.mp3` attachment. Play it.
|
||||
- Without `--voice`, the configured `TTS_VOICE` (default `af_sarah`) is used.
|
||||
- `!speak hello world --voice af_bella`
|
||||
Expect: same flow, different voice. The `--voice` flag is only recognized at the
|
||||
**very end** of the message; `!speak the --voice flag is fun` speaks the entire
|
||||
string verbatim.
|
||||
- Unknown voice: `!speak hi --voice af_nobody`
|
||||
Expect: `Unknown voice 'af_nobody'. Use `!voices` to see available voices.`
|
||||
- No text: `!speak --voice af_bella`
|
||||
Expect: `Please provide text to speak.`
|
||||
- Over 5000 characters of text:
|
||||
Expect: `Text too long to speak. Max 5000 characters.`
|
||||
- Cooldown: 3 uses per 30 seconds per user. A 4th immediate call:
|
||||
`You're using that too quickly, try again in Ns.`
|
||||
- Missing TTS model files (run without the onnx/bin files):
|
||||
`TTS engine not initialized. Make sure kokoro-v1.0.onnx and voices-v1.0.bin are present.`
|
||||
|
||||
### `!speak <bot_name> <text> [--voice <name>]` — custom bot responds and speaks
|
||||
|
||||
Requires an existing custom bot; the first word is compared **exactly**
|
||||
(case-sensitive) against stored bot names.
|
||||
|
||||
- `!speak alfred what time is it`
|
||||
Expect, in order:
|
||||
1. `**alfred** is thinking...`
|
||||
2. `**alfred**: <in-character reply>` (the LLM reply, not your text)
|
||||
3. `Generating speech for **alfred**...`
|
||||
4. `speech.mp3` containing the **bot's reply** spoken aloud.
|
||||
The exchange is also stored in history (visible later via `!history alfred`).
|
||||
- `!speak alfred what time is it --voice am_puck` — same flow, am_puck voice.
|
||||
- `!speak alfred` (no text after the name):
|
||||
`Please provide text for the bot to respond to.`
|
||||
- `!speak nonexistent hello`: `Custom bot 'nonexistent' not found.`
|
||||
|
||||
### `!voices` — list TTS voices
|
||||
|
||||
- No arguments. Expect the static catalog grouped by category:
|
||||
en-US female (`af_alloy` … `af_sky`), en-US male (`am_adam` … `am_puck`),
|
||||
en-GB (`bf_*`, `bm_*`), fr-FR (`ff_siwis`), Italian (`if_sara`, `im_nicola`),
|
||||
Japanese (`jf_*`, `jm_kumo`), Mandarin (`zf_*`, `zm_*`), ending with
|
||||
`Use `!speak <text> --voice <voice_name>` to choose a voice.`
|
||||
- Spot-check one voice per language with `!speak <text in that language> --voice <name>`
|
||||
and listen for the expected accent.
|
||||
|
||||
### `!doodlebob <prompt>` — generate an image
|
||||
|
||||
- `!doodlebob a cat sitting on the moon`
|
||||
Expect, in order:
|
||||
1. `**Doodlebob shopping for a canvas...**`
|
||||
2. `**Doodlebob selected <layout>**` — the LLM picks `square`, `portrait`, or
|
||||
`landscape` from the prompt (a tall subject should yield portrait, a wide
|
||||
scene landscape). The canvas size follows the layout.
|
||||
3. `**Doodlebob calling drone strike on <first 100 chars of the final prompt>...**`
|
||||
4. From the **second** generation onward: `**Drone ETA: ~N seconds**` (moving
|
||||
average of recorded generation times).
|
||||
5. An `image.png` attachment that matches the prompt.
|
||||
6. `**Strike complete. Image generated in N.N seconds.**`
|
||||
- Prompt over 2000 characters: `Prompt too long. Max 2000 characters.`
|
||||
- Cooldown: 1 per 60 seconds per user.
|
||||
- Image API failure: `Failed to generate image. The server may be busy.`
|
||||
|
||||
### `!retcon <prompt>` — edit an attached image
|
||||
|
||||
- Attach an image **on the same message** as the command (drag it into the
|
||||
channel): `!retcon make it look like it is on fire [image]`
|
||||
Expect: `**Rewriting history to match make it look like it is on fire...**`
|
||||
then an edited `image.png` attachment.
|
||||
- No attachment: `!retcon make it sunny`
|
||||
Expect: `Please attach an image to edit.`
|
||||
(Only real attachments count; pasted external URLs are ignored, and only
|
||||
Discord CDN hosts are downloaded, capped at 8 MB.)
|
||||
- Prompt over 2000 characters: `Prompt too long. Max 2000 characters.`
|
||||
- Image edit API failure: `Failed to edit the image.`
|
||||
|
||||
### `!talkforme <bot1> <bot2> <n> <topic>` — bot-vs-bot conversation
|
||||
|
||||
Requires two existing custom bots.
|
||||
|
||||
- `!talkforme alfred jarvis 4 the meaning of life`
|
||||
Expect: `alfred is going to talk to jarvis about "the meaning of life" for 4 replies.`
|
||||
then alternating messages headed `## alfred` / `## jarvis`, each reply in
|
||||
character. Total chat messages is **n + 1** (bot1's opener plus n loop
|
||||
replies); n is capped at 20, so at most 21 messages.
|
||||
- Too few parts: `!talkforme alfred jarvis`
|
||||
Expect: `Usage: !talkforme bot1 bot2 <number> <topic>`
|
||||
- Unknown bot: `!talkforme alfred nosuchbot 4 tea`
|
||||
Expect: `nosuchbot is not a real bot...`
|
||||
- Non-integer count: `!talkforme alfred jarvis four tea`
|
||||
Expect: `Message limit must be an integer.`
|
||||
- Topic over 500 characters: `Topic too long. Max 500 characters.`
|
||||
- Cooldown: 1 per 30 seconds per user.
|
||||
|
||||
### `!history <bot_name>` — view a bot's chat history
|
||||
|
||||
- After chatting with alfred: `!history alfred`
|
||||
Expect: `Chat History for **alfred**:` followed by the latest up-to-20
|
||||
exchanges (oldest of the 20 first), each pair as:
|
||||
|
||||
```text
|
||||
<your message>
|
||||
---
|
||||
alfred: <its reply>
|
||||
```
|
||||
|
||||
Turns from `!speak alfred ...` are included too (they are stored under the
|
||||
bot name).
|
||||
- Unknown bot: `!history nosuchbot`
|
||||
Expect: `Custom bot 'nosuchbot' not found.`
|
||||
- A bot with no stored messages: `No chat history found for **alfred**.`
|
||||
- After `!lobotomize`: same "No chat history found" reply.
|
||||
|
||||
### `!debug [subcommand]` — debug menu
|
||||
|
||||
- `!debug` — prints the menu listing `members`, `whoami`, `tools`.
|
||||
- `!debug members` — `Members in this channel (N total):` followed by an
|
||||
alphabetized roster of **server** members (Discord has no per-channel
|
||||
membership; the channel only identifies the guild), each entry showing display
|
||||
name, nickname, global name, and `[status]`. In a DM (no guild):
|
||||
`No members found in this channel.`
|
||||
- `!debug whoami` — a block about your own account: Global Name (if set),
|
||||
Nickname (if set), Top Role (if not @everyone), Activities (if any, custom
|
||||
status excluded), Joined date, Username, User ID, Account Created date. Verify
|
||||
each value against your account.
|
||||
- `!debug tools` — the `get_channel_members` tool's name, description, and
|
||||
parameter JSON schema.
|
||||
- `!debug banana` — `Unknown debug sub-command: `banana`` plus a pointer to `!debug`.
|
||||
|
||||
### `!lobotomize` — wipe all stored memory
|
||||
|
||||
- Send `!lobotomize`.
|
||||
Expect: `All conversation history and memory has been cleared.`
|
||||
- Verify:
|
||||
- `!history alfred` → `No chat history found for **alfred**.`
|
||||
- The RAG recall test above no longer remembers the fact.
|
||||
- `!list-custom-bots` is **unchanged** (bots survive; only
|
||||
`chat_messages` and `message_embeddings` are cleared).
|
||||
|
||||
## Suggested End-to-End Pass
|
||||
|
||||
A full manual pass in order:
|
||||
|
||||
1. `!lobotomize` — clean slate
|
||||
2. `!custom-bot alfred ...` and `!custom-bot jarvis ...`
|
||||
3. `!list-custom-bots` — both listed
|
||||
4. `!alfred ...` — a few turns; tell it a fact to remember later
|
||||
5. `!jarvis What did I tell you about ...?` — cross-bot memory
|
||||
6. `!speak hello world --voice af_bella`, then `!speak alfred tell me a joke`
|
||||
7. `!voices`
|
||||
8. `!doodlebob ...` twice (second run should show the Drone ETA)
|
||||
9. `!retcon ... [image]`
|
||||
10. `!talkforme alfred jarvis 4 ...`
|
||||
11. `!history alfred` — contains the steps 4–6 exchanges
|
||||
12. `!debug`, `!debug members`, `!debug whoami`, `!debug tools`
|
||||
13. `!delete-custom-bot jarvis` — then confirm `!jarvis hi` is silent
|
||||
14. `!lobotomize` — confirm `!history alfred` is empty
|
||||
|
||||
## Covered Only by the Automated Suite
|
||||
|
||||
Not practical to verify by hand (see `uv run pytest vibe_bot/tests/ -v`):
|
||||
|
||||
- embedding similarity thresholds and vector math
|
||||
- database migrations and the `MAX_HISTORY_MESSAGES` cleanup (needs 1000+ messages)
|
||||
- message chunking edge cases
|
||||
- RAG benchmark behavior (`uv run python scripts/bench_rag.py`)
|
||||
Reference in New Issue
Block a user