skills/interactive-browser/SKILL.md

---
name: interactive-browser
description: "Launch a visible Playwright browser and interact with web pages with human-in-the-loop assistance. Use when the user wants the agent to browse websites, fill forms, click buttons, or automate web tasks while the user watches and guides. The browser window is visible (not headless) and the agent takes screenshots to show the user the current page state."
---

# Interactive Browser

Launch a visible Playwright browser and interact with web pages while the user watches and provides guidance. This is a **human-in-the-loop** browser automation skill — the agent performs actions but the user sees everything and directs what happens next.

## Setup (one-time)

The skill uses Playwright with Chromium. Ensure Playwright is available:

```bash
npx playwright --version
```

If Chromium is not installed, install it:

```bash
npx playwright install chromium
```

The skill installs its own Playwright dependency on first use.

## How It Works

The skill runs a lightweight HTTP server that keeps a visible browser open. The agent communicates with the server via `curl` commands. All screenshots are saved to a temp directory and can be displayed to the user.

## Starting the Browser Server

```bash
# Kill any stale server on the port first
kill $(lsof -ti:9876) 2>/dev/null; sleep 2

# Start detached with logging so it survives shell exit
cd /var/home/ducoterra/.pi/agent/skills/interactive-browser && \
  node scripts/browser-server.js 9876 /tmp/pi-browser-screenshots > /tmp/browser-server.log 2>&1 &
sleep 3

# ALWAYS verify it started
curl -s -g -X POST "http://localhost:9876/status"
```

The server runs on port **9876** by default. Screenshots are saved to the specified directory. Action logs go to `/tmp/browser-server.log` — check it whenever a request misbehaves.

> **Why the redirect?** A background process writing to a dead terminal can get SIGHUP/SIGPIPE and die silently. Always redirect output to a log file and verify with `/status` before use.

## Commands (via curl)

All commands use `POST` requests to `http://localhost:9876`.

**Always use `curl -g`** — selectors like `a[href='/user/login']` contain brackets that curl would otherwise interpret as URL globs (exit code 3: malformed URL).

Parameters can go in the query string OR the request body (JSON or url-encoded — body wins). For complex selectors/JS with special characters, prefer `--data-urlencode`:

```bash
curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=button:has-text('Submit')"
```

### Parameters

| Param | Applies to | Description |
|-------|-----------|-------------|
| `url` | launch, go | Target URL |
| `selector` | click, dblclick, fill, select, hover, get_text | CSS selector |
| `value` | fill, select | Text to fill / option value |
| `expr` | evaluate | JavaScript expression to run in page |
| `force` | click | `true` skips Playwright actionability checks (use when element is obscured or animating) |

### Launch & Navigate

```bash
# Launch browser and navigate to a URL
curl -s -g -X POST "http://localhost:9876/launch?url=https://example.com"

# Navigate to a different URL (browser stays open)
curl -s -g -X POST "http://localhost:9876/go?url=https://other-site.com"

# Reload current page
curl -s -g -X POST "http://localhost:9876/reload"
```

### Interact

```bash
# Click an element (add force=true if it's covered/animating)
curl -s -g -X POST "http://localhost:9876/click?selector=#login-btn"

# Double-click
curl -s -g -X POST "http://localhost:9876/dblclick?selector=.item"

# Fill a form field — use --data-urlencode for selectors with special chars
curl -s -g -X POST "http://localhost:9876/fill" --data-urlencode "selector=input[name='email']" --data-urlencode "value=user@example.com"

# Select an option from a dropdown
curl -s -g -X POST "http://localhost:9876/select" --data-urlencode "selector=select#country" --data-urlencode "value=US"

# Hover over an element
curl -s -g -X POST "http://localhost:9876/hover?selector=.dropdown-trigger"
```

### Inspect

```bash
# Take a screenshot of the current page
curl -s -g -X POST "http://localhost:9876/screenshot"

# Get text content of an element
curl -s -g -X POST "http://localhost:9876/get_text?selector=#main-heading"

# Evaluate JavaScript on the page (always via --data-urlencode for complex expressions)
curl -s -g -X POST "http://localhost:9876/evaluate" --data-urlencode "expr=document.title"
```

### Control

```bash
# Check browser status
curl -s -g -X POST "http://localhost:9876/status"

# Close the browser
curl -s -g -X POST "http://localhost:9876/close"
```

## Output Format

Each command outputs JSON to stdout:

```json
{
  "status": "ok",
  "command": "click",
  "selector": "#login-btn",
  "url": "https://example.com/page",
  "title": "Example Page",
  "screenshot": "/tmp/pi-browser-screenshots/click-2024-01-01T12-00-00.png"
}
```

On error:

```json
{
  "status": "error",
  "message": "Server error",
  "details": "Target closed"
}
```

## Workflow: Human-in-the-Loop Interaction

This is the recommended pattern for interactive browser automation:

### 1. Start the Server

See [Starting the Browser Server](#starting-the-browser-server) — kill stale instances, start with log redirect, verify with `/status`.

### 2. Launch the Browser

```bash
curl -s -g -X POST "http://localhost:9876/launch?url=https://target-site.com"
```

### 3. Show the User the Page

Read the screenshot file from the JSON output and display it. Tell the user what you see and ask what they want to do.

```bash
read /tmp/pi-browser-screenshots/launch-*.png
```

### 4. Get User Instructions

The user looks at the visible browser window AND the screenshot, then tells you what to do:

> "Click the 'Sign In' button in the top right"
> "Fill the email field with my address"
> "Take a screenshot, I need to see what's on the page"

### 5. Execute the Action

Run the appropriate curl command:

```bash
curl -s -g -X POST "http://localhost:9876/click?selector=.nav-signin"
```

### 6. Show the Result

Read the new screenshot and show it to the user. Repeat from step 4.

### 7. Close When Done

```bash
curl -s -g -X POST "http://localhost:9876/close"
```

## Finding Selectors (Don't Guess)

**When a click fails with `Timeout ... waiting for locator(...)`, the selector doesn't match the page** — the site uses different markup than you assumed (e.g. Gitea's sign-in link is `a[href='/user/login']`, not `/user/sign_in`). Never guess hrefs. Enumerate elements first:

```bash
# List all links with their text and href
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('a')).map(function(a){return a.textContent.trim()+' -> '+a.getAttribute('href')}).join('\n')"

# List form fields
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('input,textarea,select')).map(function(e){return e.tagName+' name='+e.getAttribute('name')+' type='+e.getAttribute('type')}).join('\n')"
```

Selector tips:
- Use CSS selectors: `#id`, `.class`, `tag`, `[attr="value"]`
- For form fields: `input[type="email"]`, `textarea[name="message"]`, `select#country`
- For precise targeting: `input[name="username"]` is more stable than text-matching
- Use `force=true` when an element is found but covered/animating: `.../click?selector=X&force=true`

## Reading Page Content (DOM > Screenshots)

For extracting text (post lists, comments, tables), use `evaluate` on the DOM — it's exact, cheap, and scales to whole pages. Screenshots are for *showing the user state*, not for reading data.

```bash
# Example: extract Reddit post titles + scores
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('shreddit-post')).map(function(p){return p.getAttribute('score')+' pts | '+p.getAttribute('post-title')}).join('\n')"
```

### Lazy-loaded / infinite-scroll pages

Pages like Reddit load content as you scroll. Scroll several times with pauses before extracting:

```bash
for i in 1 2 3 4 5; do
  curl -s -g -X POST "http://localhost:9876/evaluate" --data-urlencode "expr=window.scrollBy(0, 1200)" > /dev/null
  sleep 0.8
done
# then run your extraction evaluate
```

### Writing JS for `evaluate`

- Pass expressions via `--data-urlencode` — never hand-encode in the query string
- Prefer `function(){}` over arrow functions, and IIFEs `(function(){...})()` for multi-statement code
- Return a single string/number (the result is `String()`-ed); join multi-line output with `'\n'`

## Important Notes

- **Requires a display** — the browser is visible, so this needs X11/Wayland (or X forwarding / a virtual display). It will fail on a headless box with no display server.
- **The browser is visible** — the user should see it open on their screen
- **slowMo is set to 100ms** — actions have a slight delay so the user can follow along
- **Viewport is 1280x900** — reasonably sized for most tasks
- **Persistent session** — the browser keeps state (cookies, localStorage) between commands, so logged-in sessions persist
- **One browser, one page** — the server drives a single page; no parallel tabs
- **Secrets** — when possible, ask the user to type passwords themselves in the visible window rather than passing them through chat/commands
- **Screenshots are saved** to the temp directory with timestamps — you can review them later
- **Interaction timeouts are 8s** — failures return a JSON error fast; check the error's `details` for Playwright's call log
- **Always close the browser** when done to free resources
- **The server stays running** until explicitly closed — you can launch, do other work, then come back and continue interacting

## Troubleshooting

| Symptom | Cause | Fix |
|---------|-------|-----|
| curl exit 3 (URL malformed) | Selector contains `[...]` interpreted as curl glob | Add `-g` to every curl call |
| curl exit 52 (empty reply) | Server dead | Restart it (see Starting the Browser Server), check `/tmp/browser-server.log` |
| curl exit 28 (timeout) | Server request stuck | Should not happen anymore (8s internal timeouts); check the log |
| `Timeout ... waiting for locator` | Selector doesn't match the page | Enumerate elements with `evaluate` (see Finding Selectors) |
| Click succeeds but nothing happens | Element is covered or JS-gated | Retry with `force=true`, then screenshot to verify |
| `Missing required parameter: X` | Param in URL but not decoded properly | Move it to the body with `--data-urlencode "X=..."` |
| Stale page content after navigation | Page still loading / lazy content | `reload`, or scroll before extracting |
| Server port busy | Old server instance | `kill $(lsof -ti:9876)` before starting |

## Example Session

```
User: "Help me log into my GitHub account"

Agent: "I'll open a visible browser so you can see everything. Starting the server..."
Agent: $ kill $(lsof -ti:9876) 2>/dev/null; sleep 2
Agent: $ cd .../interactive-browser && node scripts/browser-server.js 9876 /tmp/pi-browser-screenshots > /tmp/browser-server.log 2>&1 &
Agent: $ curl -s -g -X POST "http://localhost:9876/status"   # verify it started
Agent: $ curl -s -g -X POST "http://localhost:9876/launch?url=https://github.com/login"
Agent: [reads screenshot, shows it to user]
Agent: "I can see the GitHub login page. What would you like to do?"

User: "Fill in my email: ducoterra@example.com"

Agent: $ curl -s -g -X POST "http://localhost:9876/fill" --data-urlencode "selector=input[name='login']" --data-urlencode "value=ducoterra@example.com"
Agent: [shows new screenshot]
Agent: "Email filled in. What next?"

User: "Click the 'Continue' button"

Agent: $ curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=input[name='commit']"
Agent: [shows new screenshot]
Agent: "It's asking for a password. What should I do?"

User: "I'll type the password myself. You wait."

[User types password in the visible browser]

User: "Done, click Sign in"

Agent: $ curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=input[data-commit-button]"
Agent: [shows new screenshot]
Agent: "You're logged in! Welcome to your dashboard."

Agent: $ curl -s -g -X POST "http://localhost:9876/close"
```