increase workers and fix async issues

AGENTS.md

@@ -1,34 +1,47 @@
 # Web On The Fly
 
-Flask app that streams AI-generated fake websites via SSE.
+Flask app that queues requests, then streams AI-generated fake websites via SSE through a Valkey-backed worker.
 
 This is a Podman project. Use `podman` instead of `docker` for all container commands.
 
 ## Commands
 
-- `uv run app.py` — start dev server on port 5000
-- `uv run pyright app.py` — typecheck (strict mode in pyproject.toml)
+- `uv run app.py` — start dev server (includes worker thread) on port 5000
+- `uv run pyright app.py` — typecheck (strict mode)
 - `uv run ruff check app.py` — lint
 - `uv run ruff check app.py --fix` — auto-fix lint issues
+- `podman compose up -d` — start Valkey (required dependency)
 
 Run lint/typecheck after any code changes.
 
 ## Architecture
 
 - Single file: `app.py`
-- `/` → returns a loading page with spinner + live character counter
-- `/stream` → SSE endpoint; streams LLM output base64-encoded per chunk
-- Client JS buffers chunks in-memory (not localStorage), then replaces the document on `[DONE]`
+- Valkey (Redis-compatible) is required for request queuing, chunk buffering, and worker coordination
+- Request flow:
+  - `/` → loading page with SSE client
+  - `/stream` → enqueues request in Valkey, then polls for worker-generated chunks
+  - Background worker (thread in dev, gunicorn_post_fork in prod) pops from queue, calls LLM, buffers base64-encoded chunks
+  - Client decodes chunks in-memory, replaces document on `[RENDER]`
+- After LLM generation: validates HTML with lxml + LLM JS review, auto-fixes up to 2 times
+- THEMES list (50+ items): tuples of `(site_type, description)` — worker picks randomly
 
 ## Env vars (loaded from `.env` via python-dotenv)
 
 - `OPENAI_API_KEY` — required
-- `OPENAI_API_URL` — OpenAI-compatible endpoint (default: aipi.reeseapps.com)
-- `MODEL` — model name (default: turbo)
+- `OPENAI_API_URL` — OpenAI-compatible endpoint (code default: `https://api.openai.com/v1/chat/completions`; `.env.example` uses `https://aipi.reeseapps.com/v1/chat/completions`)
+- `MODEL` — model name (code default: `gpt-4o-mini`; `.env.example` uses `turbo`)
+- `VALKEY_URL` — Valkey/Redis URL (default: `redis://localhost:6379/0`)
+
+## Production
+
+- Runs via gunicorn: 16 workers, 300s timeout
+- `gunicorn_post_fork` hook ensures exactly one worker thread across processes (Valkey-based lock)
+- Containerfile uses `uv sync --frozen --no-dev`, runs with `uv run --no-sync`
 
 ## Key details
 
-- LLM is called with `stream=True`; chunks are base64-encoded to avoid SSE protocol breaks
+- LLM streaming uses `stream=True`; chunks are base64-encoded to avoid SSE protocol breaks
 - HTTP timeout is 300s (long generations)
-- Themes defined in `THEMES` list: tuples of `(site_type, description)` (fake_store, fake_wiki, fake_social, etc.)
 - `pyright` is configured in strict mode; use `# noqa: BLE001` for intentional broad exception catches
+- `lxml.html` is imported dynamically via `importlib.import_module`