# Browser use

> Give an agent a real browser. It clicks through your app like a person would, you watch it live, and its screenshots land in pull requests, issues and chat.

An agent that changes a UI should look at it. Give an agent a browser, and each of its runs gets a real Chromium right next to its sandbox. The agent opens your dev server, clicks, types and fills in forms, reads the console and the network requests, and takes screenshots, the way you'd check a change before calling it done.

You can watch all of it. The run page shows the agent's browser live, with a ring around each thing it clicks. And when the agent has something to show, its screenshots go into what it writes: the pull request, an issue, the chat thread or its summary.

The run page's Browser tab shows the agent's browser live. The agent opens its dev server's checkout page, types Berlin into the city field, clicks Continue, and takes a screenshot. The screenshot then appears in the pull request's description and in the team's Slack thread.

Everyone who can see the run watches the same browser while the agent works. Screenshots it takes go into what it writes: pull requests, issues, chat threads and its summary.

Ask for it in your own words, and show it what you mean. Agents see images, so you can point at a problem with a screenshot and get one back:

A thread in Slack’s #checkout:

- **Priya:** @ryph there’s a little visual glitch in the order summary on my phone *(with a marked-up screenshot: the VAT price, €68.50, runs into its label “Includes VAT (19%)”)*
- In its browser, Ada opens the checkout at 375 × 812 and sees the same overlap.
- **Ryph:** [Ada] The VAT price had a fixed width, so a longer label ran into it below 400px. It wraps now. Fixed in PR #215, here it is at the same size: *(with a screenshot of the fixed order summary)*

Agents see images. A screenshot you post with a mention, in a task or in an issue reaches the agent with your words, and it can answer with screenshots of its own.

## What the agent can do

The browser is a set of tools the agent calls, all starting with `browser__`:

| To              | Tools                                                                                        |
| --------------- | -------------------------------------------------------------------------------------------- |
| Get around      | `navigate`, `back`, `forward`, `reload`, and `tabs` to open, switch and close tabs           |
| Read the page   | `snapshot`, `evaluate` to run JavaScript in the page, `console_messages`, `network_requests` |
| Act on the page | `click`, `type`, `press_key`, `hover`, `select_option`, `scroll`, `wait_for`                 |
| See it          | `screenshot`, of the whole page or one element, and `resize` for other screen sizes          |
| Set the stage   | `add_init_script`, to stub a request, freeze the clock or seed storage before each load      |

A snapshot lists the page's buttons, links and fields, each with a short reference, so the agent clicks `button "Continue"` by its reference, with no guessing at coordinates. Every action answers with what happened: the new URL and title, a fresh snapshot, any console errors since the last step, and any dialog that popped up.

Agents running on a model that can't see images get no screenshot tool, and work from snapshots alone.

## Watch it live

1. **Open the run**

   Open the task in the web app. While the agent works, its run is on the task page.

2. **Switch to the Browser tab**

   The **Browser** tab sits next to the transcript. It shows the agent's browser as it is right now: the page, the open tabs and their address, and a ring around whatever the agent just clicked or typed into.

3. **Follow along, or dig in**

   Under the page, **Actions** lists each step the agent took in the browser. Click one to jump to it in the transcript. **Console** and **Network** show the page's console output and requests, with errors and failed requests marked.

Anyone who can see the run can watch. The transcript keeps every browser action and every screenshot.

## Screenshots where your team is

Each screenshot the agent takes is kept with the run, and the agent can put it into what it writes. It works in pull request descriptions, issues and comments, chat posts and the summary at the end of the run, and each place shows it in its own way:

| Where              | How the screenshot shows                                                                  |
| ------------------ | ----------------------------------------------------------------------------------------- |
| GitHub and GitLab  | As a link to the image in Ryph, which opens for anyone signed in who can see the run      |
| Radicle            | Embedded in the issue or patch, as a file that's part of it                               |
| Slack and Telegram | As images posted in the thread with the message, each labeled so the text can refer to it |
| Zulip              | Inline in the message                                                                     |
| Ryph's web app     | Inline in the transcript and the run's summary                                            |

GitHub gets a link because GitHub fetches images through its own proxy, which can't sign in to Ryph, and nothing from a run is public. GitLab gets one too, for now.

An agent can only use screenshots from its own run, up to ten in one post. If it names one that doesn't exist, the post fails with a message saying so, and the agent fixes it, so pull requests don't end up with broken images.

## Test against your own app

The browser shares the run's network, so `localhost:5173` in the browser is the dev server the agent started in its shell. Together with an [environment](https://ryph.ai/docs/environments.md) that brings up your database, the agent can run the app the way you do and click through the change it just made.

The agent's browser is for the agent. To click through the app yourself, ask the agent for a [preview](https://ryph.ai/docs/previews.md).

## Set your app up for agents

An agent gets the most out of its browser when it can bring your app up on its own, quickly, with something worth looking at. Ryph's own repo is a good example. Its agents work on Ryph's web app with the [playground](https://github.com/ryph-hq/ryph/tree/main/packages/playground): `bun run playground` starts all of Ryph from source in the run's container, with the web app's dev server on `localhost:5173` and a seeded team, repos, agents and finished and live runs behind it.

**Make it run without your infrastructure.** The playground needs no container runtime, no cloud account and no real data. Ryph runs on an embedded database, and a mock model provider and mock Slack, Zulip and Telegram stand in for the outside world. An agent can start it in any run, and a page it opens can't touch anything real. If your app does need a database, a cache or a queue to start, declare them in [`.ryph/services.yml`](https://ryph.ai/docs/environments.md#servicesyml), and every run gets its own fresh copy of each. What matters is that the agent's app talks to throwaway copies, and stays away from shared or production systems.

**Seed data that reaches every screen.** An empty app shows the agent very little. The playground seeds a believable deployment, and presets pick the scenario: `--preset ui`, the default, for working on the web app, `--preset full` for every state a run or patch can be in, `--preset minimal` for a near-empty setup, and `--preset empty` for the sign-up flow.

**Give it test accounts.** The playground signs in as `owner@ryph.local` with a known password, and has a `member@`, `operator@` and `viewer@` for checking each role. Seeded accounts keep real passwords out of the run.

**Start it early, as a service.** Booting takes a while, so we instruct our own agents in [`environment.md`](https://github.com/ryph-hq/ryph/blob/main/.ryph/environment.md) to start the playground before they read the code, with `ryph__background_run` set to keep it running and tell the agent when it's up:

```json
{
	"command": "bun run playground",
	"service": true,
	"readyWhen": "RYPH_PLAYGROUND_READY"
}
```

The playground prints `RYPH_PLAYGROUND_READY` once seeding and its HTTP checks are done, so the agent gets one message when it can open the page, and keeps working until then. Print a marker like this from your own dev command, after it's really ready.

**Write it down where the agent looks.** Ryph's [`.ryph/environment.md`](https://github.com/ryph-hq/ryph/blob/main/.ryph/environment.md) tells its agents when to start the playground (for layout, styling and navigation, and to skip it for backend-only work), how to start it, how to sign in, and that the app hot-reloads while backend changes need a restart. Ryph adds the file to every run's prompt, so the agent knows all this from the start.

## How it's kept apart

The browser opens pages anyone may have written, so Ryph treats them as hostile:

- **It belongs to one run.** Each run gets its own browser, which keeps nothing on disk and goes away with the run.
- **It holds no keys.** The browser has no Ryph tokens or forge credentials. It gets only what it needs to answer the agent and to use the run's internet connection.
- **It reaches what the run may.** Pages load through the run's [egress proxy](https://ryph.ai/docs/egress.md), so the agent's network allowlist applies to the browser as well. Its own `localhost` is always reachable.
- **Pages can't speak for Ryph.** Everything a page says reaches the agent marked as untrusted content from a third party, fenced so the page can't break out of the quote.
- **A page can't hang the agent.** Every browser action has a time limit. A page stuck in a loop has its script stopped, and a page that still won't answer is replaced with a blank tab.

On Kubernetes with gVisor, Chromium also runs with its own sandbox. Under Docker's default runtime that sandbox can't start, so the browser is held by the run's container confinement. With `RYPH_DOCKER_RUNTIME=runsc`, the browser tries Chromium's sandbox first.

> [!WARNING]
> **A logged-in browser can act as you**
>
> The browser starts signed in to nothing. If you give an agent a password or a token to sign in somewhere, anything that reaches the agent, like a hostile page or issue, could steer what it does there. Use test accounts with as little access as the job needs.

## Turn it on

Turn it on for each agent that should have one.

**For the agent,** open **Agents**, pick the agent and click **Edit**. Under **Browser**, turn on **Give this agent a browser**. Only an administrator can turn it on. If the deployment can't give the agent a browser, the switch says why.

**For the deployment,** there's nothing to do. Every way of running Ryph, from the [binary on your machine](https://ryph.ai/docs/try-locally.md) to [Kubernetes](https://ryph.ai/docs/kubernetes.md), offers browsers out of the box. To turn them off, set `RYPH_BROWSER_IMAGE=off` for the binary, or `runs.browserSidecar: false` in the Helm values.

A browser adds to its run's resources. Under Docker it's uncapped. Elsewhere it adds 1 CPU and 1536 MiB by default, which `RYPH_BROWSER_CPUS` and `RYPH_BROWSER_MEMORY` change.