skills/interactive-browser/SKILL.md

name, description

name	description
interactive-browser	Launch a visible Playwright browser and interact with web pages with human-in-the-loop assistance. Use when the user wants the agent to browse websites, fill forms, click buttons, or automate web tasks while the user watches and guides. The browser window is visible (not headless) and the agent takes screenshots to show the user the current page state.

Interactive Browser

Launch a visible Playwright browser and interact with web pages while the user watches and provides guidance. This is a human-in-the-loop browser automation skill — the agent performs actions but the user sees everything and directs what happens next.

Setup (one-time)

The skill uses Playwright with Chromium. Ensure Playwright is available:

npx playwright --version

If Chromium is not installed, install it:

npx playwright install chromium

The skill installs its own Playwright dependency on first use.

How It Works

The skill runs a lightweight HTTP server that keeps a visible browser open. The agent communicates with the server via curl commands. All screenshots are saved to a temp directory and can be displayed to the user.

Starting the Browser Server

# Kill any stale server on the port first
kill $(lsof -ti:9876) 2>/dev/null; sleep 2

# Start detached with logging so it survives shell exit
cd /var/home/ducoterra/.pi/agent/skills/interactive-browser && \
  node scripts/browser-server.js 9876 /tmp/pi-browser-screenshots > /tmp/browser-server.log 2>&1 &
sleep 3

# ALWAYS verify it started
curl -s -g -X POST "http://localhost:9876/status"

The server runs on port 9876 by default. Screenshots are saved to the specified directory. Action logs go to /tmp/browser-server.log — check it whenever a request misbehaves.

Why the redirect? A background process writing to a dead terminal can get SIGHUP/SIGPIPE and die silently. Always redirect output to a log file and verify with /status before use.

Commands (via curl)

All commands use POST requests to http://localhost:9876.

Always use curl -g — selectors like a[href='/user/login'] contain brackets that curl would otherwise interpret as URL globs (exit code 3: malformed URL).

Parameters can go in the query string OR the request body (JSON or url-encoded — body wins). For complex selectors/JS with special characters, prefer --data-urlencode:

curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=button:has-text('Submit')"

Parameters

Param	Applies to	Description
url	launch, go	Target URL
selector	click, dblclick, fill, select, hover, get_text	CSS selector
value	fill, select	Text to fill / option value
expr	evaluate	JavaScript expression to run in page
force	click	true skips Playwright actionability checks (use when element is obscured or animating)

Launch & Navigate

# Launch browser and navigate to a URL
curl -s -g -X POST "http://localhost:9876/launch?url=https://example.com"

# Navigate to a different URL (browser stays open)
curl -s -g -X POST "http://localhost:9876/go?url=https://other-site.com"

# Reload current page
curl -s -g -X POST "http://localhost:9876/reload"

Interact

# Click an element (add force=true if it's covered/animating)
curl -s -g -X POST "http://localhost:9876/click?selector=#login-btn"

# Double-click
curl -s -g -X POST "http://localhost:9876/dblclick?selector=.item"

# Fill a form field — use --data-urlencode for selectors with special chars
curl -s -g -X POST "http://localhost:9876/fill" --data-urlencode "selector=input[name='email']" --data-urlencode "value=user@example.com"

# Select an option from a dropdown
curl -s -g -X POST "http://localhost:9876/select" --data-urlencode "selector=select#country" --data-urlencode "value=US"

# Hover over an element
curl -s -g -X POST "http://localhost:9876/hover?selector=.dropdown-trigger"

Inspect

# Take a screenshot of the current page
curl -s -g -X POST "http://localhost:9876/screenshot"

# Get text content of an element
curl -s -g -X POST "http://localhost:9876/get_text?selector=#main-heading"

# Evaluate JavaScript on the page (always via --data-urlencode for complex expressions)
curl -s -g -X POST "http://localhost:9876/evaluate" --data-urlencode "expr=document.title"

Control

# Check browser status
curl -s -g -X POST "http://localhost:9876/status"

# Close the browser
curl -s -g -X POST "http://localhost:9876/close"

Output Format

Each command outputs JSON to stdout:

{
  "status": "ok",
  "command": "click",
  "selector": "#login-btn",
  "url": "https://example.com/page",
  "title": "Example Page",
  "screenshot": "/tmp/pi-browser-screenshots/click-2024-01-01T12-00-00.png"
}

On error:

{
  "status": "error",
  "message": "Server error",
  "details": "Target closed"
}

Workflow: Human-in-the-Loop Interaction

This is the recommended pattern for interactive browser automation:

1. Start the Server

See Starting the Browser Server — kill stale instances, start with log redirect, verify with /status.

2. Launch the Browser

curl -s -g -X POST "http://localhost:9876/launch?url=https://target-site.com"

3. Show the User the Page

Read the screenshot file from the JSON output and display it. Tell the user what you see and ask what they want to do.

read /tmp/pi-browser-screenshots/launch-*.png

4. Get User Instructions

The user looks at the visible browser window AND the screenshot, then tells you what to do:

"Click the 'Sign In' button in the top right" "Fill the email field with my address" "Take a screenshot, I need to see what's on the page"

5. Execute the Action

Run the appropriate curl command:

curl -s -g -X POST "http://localhost:9876/click?selector=.nav-signin"

6. Show the Result

Read the new screenshot and show it to the user. Repeat from step 4.

7. Close When Done

curl -s -g -X POST "http://localhost:9876/close"

Finding Selectors (Don't Guess)

When a click fails with Timeout ... waiting for locator(...), the selector doesn't match the page — the site uses different markup than you assumed (e.g. Gitea's sign-in link is a[href='/user/login'], not /user/sign_in). Never guess hrefs. Enumerate elements first:

# List all links with their text and href
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('a')).map(function(a){return a.textContent.trim()+' -> '+a.getAttribute('href')}).join('\n')"

# List form fields
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('input,textarea,select')).map(function(e){return e.tagName+' name='+e.getAttribute('name')+' type='+e.getAttribute('type')}).join('\n')"

Selector tips:

• Use CSS selectors: #id, .class, tag, [attr="value"]
• For form fields: input[type="email"], textarea[name="message"], select#country
• For precise targeting: input[name="username"] is more stable than text-matching
• Use force=true when an element is found but covered/animating: .../click?selector=X&force=true

Reading Page Content (DOM > Screenshots)

For extracting text (post lists, comments, tables), use evaluate on the DOM — it's exact, cheap, and scales to whole pages. Screenshots are for showing the user state, not for reading data.

# Example: extract Reddit post titles + scores
curl -s -g -X POST "http://localhost:9876/evaluate" \
  --data-urlencode "expr=Array.from(document.querySelectorAll('shreddit-post')).map(function(p){return p.getAttribute('score')+' pts | '+p.getAttribute('post-title')}).join('\n')"

Lazy-loaded / infinite-scroll pages

Pages like Reddit load content as you scroll. Scroll several times with pauses before extracting:

for i in 1 2 3 4 5; do
  curl -s -g -X POST "http://localhost:9876/evaluate" --data-urlencode "expr=window.scrollBy(0, 1200)" > /dev/null
  sleep 0.8
done
# then run your extraction evaluate

Writing JS for evaluate

• Pass expressions via --data-urlencode — never hand-encode in the query string
• Prefer function(){} over arrow functions, and IIFEs (function(){...})() for multi-statement code
• Return a single string/number (the result is String()-ed); join multi-line output with '\n'

Important Notes

• Requires a display — the browser is visible, so this needs X11/Wayland (or X forwarding / a virtual display). It will fail on a headless box with no display server.
• The browser is visible — the user should see it open on their screen
• slowMo is set to 100ms — actions have a slight delay so the user can follow along
• Viewport is 1280x900 — reasonably sized for most tasks
• Persistent session — the browser keeps state (cookies, localStorage) between commands, so logged-in sessions persist
• One browser, one page — the server drives a single page; no parallel tabs
• Secrets — when possible, ask the user to type passwords themselves in the visible window rather than passing them through chat/commands
• Screenshots are saved to the temp directory with timestamps — you can review them later
• Interaction timeouts are 8s — failures return a JSON error fast; check the error's details for Playwright's call log
• Always close the browser when done to free resources
• The server stays running until explicitly closed — you can launch, do other work, then come back and continue interacting

Troubleshooting

Symptom	Cause	Fix
curl exit 3 (URL malformed)	Selector contains [...] interpreted as curl glob	Add -g to every curl call
curl exit 52 (empty reply)	Server dead	Restart it (see Starting the Browser Server), check /tmp/browser-server.log
curl exit 28 (timeout)	Server request stuck	Should not happen anymore (8s internal timeouts); check the log
Timeout ... waiting for locator	Selector doesn't match the page	Enumerate elements with evaluate (see Finding Selectors)
Click succeeds but nothing happens	Element is covered or JS-gated	Retry with force=true, then screenshot to verify
Missing required parameter: X	Param in URL but not decoded properly	Move it to the body with --data-urlencode "X=..."
Stale page content after navigation	Page still loading / lazy content	reload, or scroll before extracting
Server port busy	Old server instance	kill $(lsof -ti:9876) before starting

Example Session

User: "Help me log into my GitHub account"

Agent: "I'll open a visible browser so you can see everything. Starting the server..."
Agent: $ kill $(lsof -ti:9876) 2>/dev/null; sleep 2
Agent: $ cd .../interactive-browser && node scripts/browser-server.js 9876 /tmp/pi-browser-screenshots > /tmp/browser-server.log 2>&1 &
Agent: $ curl -s -g -X POST "http://localhost:9876/status"   # verify it started
Agent: $ curl -s -g -X POST "http://localhost:9876/launch?url=https://github.com/login"
Agent: [reads screenshot, shows it to user]
Agent: "I can see the GitHub login page. What would you like to do?"

User: "Fill in my email: ducoterra@example.com"

Agent: $ curl -s -g -X POST "http://localhost:9876/fill" --data-urlencode "selector=input[name='login']" --data-urlencode "value=ducoterra@example.com"
Agent: [shows new screenshot]
Agent: "Email filled in. What next?"

User: "Click the 'Continue' button"

Agent: $ curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=input[name='commit']"
Agent: [shows new screenshot]
Agent: "It's asking for a password. What should I do?"

User: "I'll type the password myself. You wait."

[User types password in the visible browser]

User: "Done, click Sign in"

Agent: $ curl -s -g -X POST "http://localhost:9876/click" --data-urlencode "selector=input[data-commit-button]"
Agent: [shows new screenshot]
Agent: "You're logged in! Welcome to your dashboard."

Agent: $ curl -s -g -X POST "http://localhost:9876/close"