Best Practices for Using Lightpanda

Céline Debled

Developer Relations

TL;DR

Lightpanda runs the requests you ask for. The correct behavior comes from you. Install the agent skill if a model is driving. Identify your traffic with --user-agent whatever you’re doing. Let the workflow decide the rest: --obey-robots and --http-nav-delay if you’re fetching pages in bulk, scoping if a model is paying for the tokens, and a fresh browser per task if you’re following links you didn’t choose.

Start with the right mental model: Lightpanda is a browser for machines

Lightpanda is a headless browser for machines. People make agents that use it to read and act on the web, for data extraction at scale, for indexing, prerendering, and for monitoring.

Unlike a crawler framework, Lightpanda (like curl and all browsers) doesn’t have a politeness policy . The policy is yours to set, and Lightpanda gives you the options to enforce that inside the browser, rather than in every script that talks to it.

The useful questions to ask yourself:

• How many pages will this touch, and against how many hosts?
• Does a model read the output, and pay for it in tokens and turns?
• Are you following links you chose, or links the page gave you?

Each section below outlines best practices depending on your answers.

Choose the right Lightpanda interface for your use case

The Lightpanda binary exposes multiple interfaces for you to control the browser.

Configure the browser correctly

The code snippets below can be converted to all interfaces: fetch, serve, mcp, agent and run.

Say who you are

This applies regardless of the volume. Site operators treat traffic they can identify differently from traffic they can’t. They route known clients separately in analytics, apply their own rate limits, and allowlist the ones they recognize.

--user-agent replaces the user agent string outright. Put your name and a URL in it:

./lightpanda serve --user-agent "MyCompany/1.0 (+https://example.com/bot)"

An operator who wants to know what your traffic does can read that page, then contact you instead of writing a firewall rule.

--user-agent rejects any value containing “mozilla”, which rules out impersonating Chrome, Firefox, Safari and Edge, since they all still ship a legacy Mozilla/5.0 token. Whatever you set, Lightpanda always sends Sec-Ch-Ua: "Lightpanda";v="1" alongside it, so the engine stays visible to any server that checks.

Prove it with Web Bot Auth. A user agent string is a claim, and nothing stops another client from sending the same one. Web Bot Auth replaces the claim with a signature: Lightpanda signs every outgoing request with an Ed25519 key, and sites verify it against a key directory you publish. Cloudflare, Akamai, Google and AWS all verify signatures. It takes three flags: --web-bot-auth-key-file, --web-bot-auth-keyid and --web-bot-auth-domain. Our Web Bot Auth post  covers the full setup.

Respect robots.txt when you’re fetching in bulk

A site uses robots.txt to declare which paths automated clients may fetch, in the format specified by RFC 9309 . --obey-robots makes Lightpanda check it on every request, fetching each host’s file once and sharing the result for the session.

./lightpanda serve --obey-robots --dump markdown https://example.com/some/page

Turn it on when you’re pulling pages in volume. Disallowed paths are often the ones that cost the site most to serve: infinite diff views, search result permutations, faceted filters that hit the database on every load. Owners disallow them because they’re expensive. Honoring the list is the cheapest way to be sure you aren’t doing damage.

Pace your navigations once the volume goes up

A run that fires navigations as fast as the event loop degrades the site it’s reading from. Rate limits, CAPTCHAs and bans are the consequence rather than the problem.

Most people handle pacing in the automation script, which means every script reimplements it and any bug lands on the target site. PR #3239  moves it into the browser:

./lightpanda serve --http-nav-delay 1000

Navigations to the same host are now spaced at least one second apart, whatever the thing driving the browser does. The limit is per host, so broad, shallow work pays nothing and only concentrated load feels it. Start at 1000 ms and adjust. A stable run that finishes overnight beats a fast one that gets blocked at 2am.

Only top-level navigations are paced, because delaying every subrequest would make page load times erratic. And pacing is not parallelism: --http-max-host-open caps simultaneous connections to one host and defaults to 6.

Give every task its own browser

Scope each task to its own browser instance. This is the model we described in browser security in the age of AI agents . It’s possible with Lightpanda in a way it isn’t with Chrome, because startup is close to instant and memory per instance is small.

A run that reads untrusted content, holds session state and can act on the web leaves you vulnerable to prompt injections. A browser scoped to one task, with only the privileges that task needs, is more secure. Consider --block-private-networks too, since a browser inside your network is a server-side request forgery waiting to happen.

Even where nothing is untrusted, a fresh instance has no cookies, storage or leftover state, so a run either works from scratch or fails visibly.

Only CDP and MCP need explicit configuration. Other interfaces handle isolation automatically.

CDP server. You control the lifecycle. In Puppeteer or Playwright, create a new browser instance for each task. Each one connects to a separate Lightpanda process, or you create separate contexts within the same connection.

MCP server. The behavior depends on your mode. The stdio default mode gives you one browser for the session. HTTP mode (start with --port 3000) lets you request multiple sessions, each with its own browser. Use HTTP mode if you need per-task isolation.

The cheapest request is the one you never make

Every flag above shapes the requests you send. The larger win is sending fewer, and that happens in your design.

1. Take the bulk source when one exists. Walking a web-based git forge to reconstruct history means thousands of page loads, each asking the server to generate a diff on demand. A single git clone hands you the same data locally, and you query it at memory speed. The same goes for a documented API instead of the interface on top of it, or a sitemap.xml instead of link discovery.
2. Use search APIs instead of navigating search engines. APIs like Exa, Tavily or Brave Search return results directly; no render-and-parse overhead. MCP agents can also use the native search tool.
3. Back off when a site tells you to. A 429 or 503 with a Retry-After header is the server asking for room. Reading that header and pausing is your code’s job. Lightpanda doesn’t handle retries, and --http-nav-delay doesn’t cover it: a rate limit means your steady pace was still too fast.
4. Cache between runs. Lightpanda’s HTTP cache follows RFC 9111 and revalidates with If-None-Match and If-Modified-Since, so unchanged resources come back as a 304 with no body. It is off by default unless you point it somewhere:

./lightpanda serve --http-cache-dir /var/cache/lightpanda

5. Don’t fetch what you’ll throw away. If you want text, you don’t need the ad networks, analytics beacons and tracking pixels. --adblock-lists takes filter lists and --block-urls takes your own patterns. External stylesheets are off unless you pass --load-resources stylesheet, except in agent mode with an LLM where they load by default so the model can tell what’s visible.
6. Stay out of infinite URL spaces. Calendars, diffs between arbitrary revisions, faceted search, sort parameters. These generate unbounded URLs, and each is usually a fresh database query on the other end. A run that wanders into one never terminates, and it looks like an attack while it does.
7. Cap depth and breadth per site. Decide up front how deep you’ll follow links and how many URLs you’ll keep from one domain. Linkup wrote about this  for AI search crawling: without a cap, the tail of a large site absorbs most of your budget and returns almost nothing.

When a flow settles, stop rediscovering it

The first time a model works out how to log in and pull a report, that’s reasoning. The thousandth time, it’s waste and a fresh chance to get it wrong. Capture the flow as a PandaScript and replay it with lightpanda run script.js. You stop paying for the rediscovery, and the failure mode becomes a script error instead of a plausible wrong answer.

For agents, give the model only what it needs

Once a model consumes the output, your bottleneck stops being bandwidth and becomes context. A page dumped in full costs tokens and it crowds out the reasoning you wanted.

We measured this ourselves : the tool surface, not the engine, was the dominant variable in accuracy.

1. Start with tree. The semantic tree gives you roles, accessible names and interactivity in one pass. It tells the model what kind of page this is and where the interesting part lives. LP.getSemanticTree does the same over CDP.
2. Drill down with nodeDetails or findElement. Locate the region you care about instead of pulling the whole document.
3. Then call markdown on that subtree. Pass a node ID or selector. Full-page markdown is still right sometimes, but it should be the fallback.

Where this is going

Browser automation is growing fast, so it’s essential to preserve websites from associated huge load. Serving a page costs a site money. Cost, quality of service and security are correlated. Every site that blocks bots makes automation harder for everyone, so we would rather machine traffic behave well.

Cloudflare, which sits in front of roughly a fifth of the web, began blocking AI crawlers by default  on new domains in July 2025. The same rule that stops a scraper reselling your content stops an agent that was about to buy something for a customer. Cloudflare also has an option of pay per crawl  so a site can name a price instead of refusing the request. That turns automated traffic from a cost into a transaction.

If a handful of verification allowlists decide which agents get through, a small number of companies decide who reads the web. A web that’s decentralised by design is worth defending: identity should remain a way in rather than a gate.

Putting it together

If you are using an AI coding agent, the agent skill  explains how to set up and use Lightpanda by integrating best practices.

To install it:

npx skills add https://github.com/lightpanda-io/agent-skill --skill lightpanda

Or as a Claude Code plugin:

/plugin marketplace add lightpanda-io/agent-skill

/plugin install lightpanda@lightpanda

To set up Lightpanda manually, the quickstart  gets you running in ten minutes.

Install the binary with bash scripts/install.sh, or through Homebrew, the AUR, or the .deb package. If wanted, register the MCP server with claude mcp add lightpanda -- lightpanda mcp.

And start here:

./lightpanda serve --user-agent "MyCompany/1.0 (+https://example.com/bot)"

• --obey-robots and --http-nav-delay 1000 once the run touches pages in bulk.
• --http-cache-dir once it runs more than once.
• --block-private-networks with one browser per task once you’re following links a page gives you.

FAQ

Why shouldn’t I dump the whole page as markdown?

It costs turns. Full-page markdown returns 10 to 30 KB per call, which slows every turn and eats context the model needs for reasoning. Reading the semantic tree first and converting only the relevant subtree gets the same information for a fraction of the tokens.

Do these practices apply if I’m only loading a few pages?

Identity always applies. Pacing and robots.txt matter once you’re concentrating load on one host, and caching matters once the same run happens twice. Per-task isolation depends on trust rather than volume, so it applies from the first page if you’re following links you didn’t choose.

Should I use —user-agent or —user-agent-suffix?

Use --user-agent and put your own identity in it, with a URL where an operator can read about your bot. That gives site owners something specific to allowlist, rate limit or contact. --user-agent-suffix appends to Lightpanda/1.0 instead, which keeps you matched to robots.txt rules written for Lightpanda. Either way, Sec-Ch-Ua still identifies the engine.

How can I control Lightpanda traffic to my site?

Sec-Ch-Ua is the reliable signal. Lightpanda always sends "Lightpanda";v="1" and the header is not overridable, so an edge or WAF rule matching it catches every Lightpanda request whatever user agent the operator configured.

User-Agent is Lightpanda/1.0 by default, but operators can and should replace it with their own identity. Treat it as a way to tell operators apart rather than a way to detect the engine. It can never contain “mozilla”, so it will never be disguised as a mainstream browser.

robots.txt works as it does for any client: a User-agent: Lightpanda block is honored by anyone running --obey-robots and ignored by anyone who isn’t.

Céline Debled

Developer Relations

Céline runs Developer Relations at Lightpanda. She started out as a full-stack web developer, then spent four years working with DevTools founders at Agoranov (a Paris startup incubator). At Lightpanda she writes the docs and demos, talks to developers building on the browser, and gets their feedback back to the engineering team. If you are stuck on something, she is the person to ask.