Browser use

An agent that changes a UI should look at it. Give an agent a browser, and each of its runs gets a real Chromium right next to its sandbox. The agent opens your dev server, clicks, types and fills in forms, reads the console and the network requests, and takes screenshots, the way you’d check a change before calling it done.

You can watch all of it. The run page shows the agent’s browser live, with a ring around each thing it clicks. And when the agent has something to show, its screenshots go into what it writes: the pull request, an issue, the chat thread or its summary.

Ask for it in your own words, and show it what you mean. Agents see images, so you can point at a problem with a screenshot and get one back:

What the agent can do

The browser is a set of tools the agent calls, all starting with browser__:

A snapshot lists the page’s buttons, links and fields, each with a short reference, so the agent clicks button "Continue" by its reference, with no guessing at coordinates. Every action answers with what happened: the new URL and title, a fresh snapshot, any console errors since the last step, and any dialog that popped up.

Agents running on a model that can’t see images get no screenshot tool, and work from snapshots alone.

Watch it live

1. Open the run

   Open the task in the web app. While the agent works, its run is on the task page.

2. Switch to the Browser tab

   The Browser tab sits next to the transcript. It shows the agent’s browser as it is right now: the page, the open tabs and their address, and a ring around whatever the agent just clicked or typed into.

3. Follow along, or dig in

   Under the page, Actions lists each step the agent took in the browser. Click one to jump to it in the transcript. Console and Network show the page’s console output and requests, with errors and failed requests marked.

Anyone who can see the run can watch. The transcript keeps every browser action and every screenshot.

Screenshots where your team is

Each screenshot the agent takes is kept with the run, and the agent can put it into what it writes. It works in pull request descriptions, issues and comments, chat posts and the summary at the end of the run, and each place shows it in its own way:

GitHub gets a link because GitHub fetches images through its own proxy, which can’t sign in to Ryph, and nothing from a run is public. GitLab gets one too, for now.

An agent can only use screenshots from its own run, up to ten in one post. If it names one that doesn’t exist, the post fails with a message saying so, and the agent fixes it, so pull requests don’t end up with broken images.

Test against your own app

The browser shares the run’s network, so localhost:5173 in the browser is the dev server the agent started in its shell. Together with an environment that brings up your database, the agent can run the app the way you do and click through the change it just made.

The agent’s browser is for the agent. To click through the app yourself, ask the agent for a preview.

Set your app up for agents

An agent gets the most out of its browser when it can bring your app up on its own, quickly, with something worth looking at. Ryph’s own repo is a good example. Its agents work on Ryph’s web app with the playground: bun run playground starts all of Ryph from source in the run’s container, with the web app’s dev server on localhost:5173 and a seeded team, repos, agents and finished and live runs behind it.

Make it run without your infrastructure. The playground needs no container runtime, no cloud account and no real data. Ryph runs on an embedded database, and a mock model provider and mock Slack, Zulip and Telegram stand in for the outside world. An agent can start it in any run, and a page it opens can’t touch anything real. If your app does need a database, a cache or a queue to start, declare them in .ryph/services.yml, and every run gets its own fresh copy of each. What matters is that the agent’s app talks to throwaway copies, and stays away from shared or production systems.

Seed data that reaches every screen. An empty app shows the agent very little. The playground seeds a believable deployment, and presets pick the scenario: --preset ui, the default, for working on the web app, --preset full for every state a run or patch can be in, --preset minimal for a near-empty setup, and --preset empty for the sign-up flow.

Give it test accounts. The playground signs in as owner@ryph.local with a known password, and has a member@, operator@ and viewer@ for checking each role. Seeded accounts keep real passwords out of the run.

Start it early, as a service. Booting takes a while, so we instruct our own agents in environment.md to start the playground before they read the code, with ryph__background_run set to keep it running and tell the agent when it’s up:

{
	"command": "bun run playground",
	"service": true,
	"readyWhen": "RYPH_PLAYGROUND_READY"
}

The playground prints RYPH_PLAYGROUND_READY once seeding and its HTTP checks are done, so the agent gets one message when it can open the page, and keeps working until then. Print a marker like this from your own dev command, after it’s really ready.

Write it down where the agent looks. Ryph’s .ryph/environment.md tells its agents when to start the playground (for layout, styling and navigation, and to skip it for backend-only work), how to start it, how to sign in, and that the app hot-reloads while backend changes need a restart. Ryph adds the file to every run’s prompt, so the agent knows all this from the start.

How it’s kept apart

The browser opens pages anyone may have written, so Ryph treats them as hostile:

• It belongs to one run. Each run gets its own browser, which keeps nothing on disk and goes away with the run.
• It holds no keys. The browser has no Ryph tokens or forge credentials. It gets only what it needs to answer the agent and to use the run’s internet connection.
• It reaches what the run may. Pages load through the run’s egress proxy, so the agent’s network allowlist applies to the browser as well. Its own localhost is always reachable.
• Pages can’t speak for Ryph. Everything a page says reaches the agent marked as untrusted content from a third party, fenced so the page can’t break out of the quote.
• A page can’t hang the agent. Every browser action has a time limit. A page stuck in a loop has its script stopped, and a page that still won’t answer is replaced with a blank tab.

On Kubernetes with gVisor, Chromium also runs with its own sandbox. Under Docker’s default runtime that sandbox can’t start, so the browser is held by the run’s container confinement. With RYPH_DOCKER_RUNTIME=runsc, the browser tries Chromium’s sandbox first.

Turn it on

Turn it on for each agent that should have one.

For the agent, open Agents, pick the agent and click Edit. Under Browser, turn on Give this agent a browser. Only an administrator can turn it on. If the deployment can’t give the agent a browser, the switch says why.

For the deployment, there’s nothing to do. Every way of running Ryph, from the binary on your machine to Kubernetes, offers browsers out of the box. To turn them off, set RYPH_BROWSER_IMAGE=off for the binary, or runs.browserSidecar: false in the Helm values.

A browser adds to its run’s resources. Under Docker it’s uncapped. Elsewhere it adds 1 CPU and 1536 MiB by default, which RYPH_BROWSER_CPUS and RYPH_BROWSER_MEMORY change.