Web Bot Auth Support in Lightpanda

Muki Kiboigo

Software Engineer

TL;DR

Lightpanda now signs outgoing HTTP requests using Web Bot Auth , the emerging IETF standard based on HTTP Message Signatures (RFC 9421) . This means your automation traffic can cryptographically prove who it is to websites and CDNs. Pass your Ed25519 private key, and Lightpanda handles the rest: signing every request with the right headers so that providers like Cloudflare and Akamai recognize your bot as verified.

What Is Web Bot Auth?

Web Bot Auth is an IETF draft protocol  that applies HTTP Message Signatures (RFC 9421) to bot identification. The core idea is straightforward.

Before sending a request, your bot signs specific components of that request (at minimum, the target domain) with an Ed25519 private key. The corresponding public key lives in a directory you host at a well-known URL. When a server receives your request, it fetches your public key from that directory and verifies the signature. If the signature is valid, the server knows the request came from you and nobody else.

Each signed request carries three HTTP headers:

• Signature-Input: Declares what components were signed, the key ID, validity timestamps, and the tag web-bot-auth
• Signature: The actual cryptographic signature
• Signature-Agent: A URL pointing to your public key directory

Here’s what a signed request looks like on the wire:

GET /products HTTP/1.1
Host: www.example.com
User-Agent: MyBot/1.0
Signature-Agent: "https://mybot.mydomain.com"
Signature-Input: sig=("@authority" "signature-agent");\
                 created=1700000000;\
                 expires=1700000060;\
                 keyid="ba3e64==";\
                 tag="web-bot-auth"
Signature: sig=abc123==

Cloudflare and Akamai already verify these signatures at the edge. OpenAI signs Operator  requests this way. The IETF is considering a dedicated Web Bot Auth working group  to formalize the standard.

How Lightpanda Implements Web Bot Auth

Thanks to PR #1609 , Lightpanda now handles Web Bot Auth signing at the HTTP layer. You provide your Ed25519 private key, your key id and your Signature-Agent URL, and Lightpanda attaches the correct Signature-Input, Signature, and Signature-Agent headers to every outgoing request.

This happens inside the browser, at the network level. You don’t need to modify your Puppeteer or Playwright scripts to add headers manually. Every HTTP request that Lightpanda makes, whether it’s the initial page fetch, a JavaScript-triggered XHR gets signed automatically.

Why Browser-Level Signing Matters

You could sign requests yourself in your automation script, but that only covers the requests you explicitly make. A real page load triggers dozens of subrequests: scripts, and API calls from JavaScript. If those subrequests go out unsigned, a verifier might still flag your traffic as unverified.

Because Lightpanda controls the HTTP client (we use libcurl  under the hood), we sign at the point where requests leave the browser.

Getting Started

Pass your key and Signature-Agent URL when starting Lightpanda:

lightpanda serve \
  --web-bot-auth-key /path/to/private-key.pem \
  --web-bot-auth-agent "https://mybot.example.com" \
  --web_bot_auth_keyid "_f1ywmr7gZ-T2JBMHEAV8FRE7ntLjHFanj9kDccmacU" \
  --host 127.0.0.1 --port 9222

Or with the fetch command:

lightpanda fetch \
  --web-bot-auth-key /path/to/private-key.pem \
  --web-bot-auth-agent "https://mybot.mydomain.com" \
  --web_bot_auth_keyid "_f1ywmr7gZ-T2JBMHEAV8FRE7ntLjHFanj9kDccmacU" \
  https://example.com

Connect to Lightpanda’s CDP server as usual, and every request goes out signed.

Setting Up Your Keys and Directory

Before you sign requests, you need a keypair and a public key directory.

Generate an Ed25519 Key Pair

openssl genpkey -algorithm ed25519 -out private-key.pem
openssl pkey -in private-key.pem -pubout -out public-key.pem

Host Your Key Directory

The protocol requires your public key to be available at /.well-known/http-message-signatures-directory on the domain specified in your Signature-Agent header. You need to serve a JSON Web Key Set (JWKS) containing your public key, and you also need to sign the directory response itself with the same key.

The directory must be served over HTTPS with the content type application/http-message-signatures-directory+json. The response needs to include its own Signature, Signature-Input, and Content-Digest headers.

To make this easier, we’ve open-sourced wbauth , a small Go server that handles key directory hosting for you. It takes care of the signed response, the correct content type, and the JWKS format. You can see a working example at our test directory endpoint .

Here’s what the server response body looks like:

HTTP/2 200
content-type: application/http-message-signatures-directory+json
date: Fri, 20 Mar 2026 11:27:05 GMT
signature: sig1=:FfEUMOUhGMZFt+ZdvcrNcE8B3iFRXdTTrh0A8WyA0RK3fM1YPc3kBWJKPtvWvs3MZ7rIP4SI9k205WxuA2YZAA==:
signature-input: sig1=("@authority";req);created=1774006025;expires=1774006085;alg="ed25519";tag="http-message-signatures-directory";keyid="_f1ywmr7gZ-T2JBMHEAV8FRE7ntLjHFanj9kDccmacU"
content-length: 90
{
  "keys": [
    {
      "crv": "Ed25519",
      "kty": "OKP",
      "x": "o7omQIOBiRF7qGjLbAA2BI2-hgxq5-AWFOgN8VcZf3w"
    }
  ]
}

Cloudflare provides a test endpoint  where you can verify that your signing setup works correctly before deploying.

Why This Matters for AI Agents

The web is increasingly hostile to automated traffic. CDNs and bot management systems are getting more aggressive.

Web Bot Auth gives your bot a verifiable identity. Cloudflare’s Verified Bots Program  now accepts applications from bots that sign their requests with Web Bot Auth. Akamai has integrated Web Bot Auth  into their bot management platform as well.

The protocol is also a natural fit for Lightpanda’s architecture. We already wrote about browser security in the age of AI agents  and our “one browser, one task” model. Each browser instance carries its own cryptographic identity, and you control exactly which key is used for which task.

Trade-offs and Limitations

Web Bot Auth is an emerging standard. A few things to keep in mind:

Adoption is growing but not universal. Cloudflare, Akamai, and Stytch verify signatures today. Many other platforms and CDNs do not yet. Your signed requests will work fine against non-supporting servers (the headers are ignored), but you won’t get the verification benefits everywhere.

Key management is on you. You need to generate, store, and rotate your Ed25519 keys. You need to host a reliable key directory over HTTPS. If your directory goes down, verifiers can’t fetch your public key.

The spec is still evolving. The IETF draft is at version 05  as of writing. There may be changes to the protocol before it becomes an RFC. We’ll track any updates and adjust our implementation accordingly.

Replay protection is limited. The protocol supports nonces and short expiry windows to mitigate replay attacks, but there is no server-side nonce validation database in current implementations. Cloudflare recommends short expiry times (around one minute) as the primary defense.

Get Started

Try Web Bot Auth today. The quickstart guide  will get you running in under 10 minutes.

FAQ

What is Web Bot Auth?

Web Bot Auth is an emerging IETF standard that uses HTTP Message Signatures (RFC 9421) to let bots and AI agents cryptographically prove their identity. Instead of relying on User-Agent strings or IP addresses, your bot signs each request with an Ed25519 private key. Servers verify the signature using your published public key.

Do I need to register with every website I want to crawl?

No. Web Bot Auth is decentralized. You host your public key directory on your own domain, and any server that supports the protocol can verify your requests independently. You do not need to coordinate with each website. You may need to register with providers, such as Cloudflare, to be considered verified.

Does Web Bot Auth work with Puppeteer and Playwright?

Yes. Lightpanda signs requests at the HTTP layer, so your existing Puppeteer and Playwright scripts work without modification. Connect to Lightpanda’s CDP server as usual. All requests, including subrequests triggered by JavaScript, go out with valid signatures.

What happens if a server doesn’t support Web Bot Auth?

The server ignores the Signature-Input, Signature, and Signature-Agent headers. Your request proceeds normally. Web Bot Auth is additive; it doesn’t break anything for non-supporting servers.

How do I rotate my signing keys?

Generate a new Ed25519 key pair, add the new public key to your JWKS directory alongside the old one, and update the private key path in your Lightpanda configuration. Remove the old public key from your directory after a transition period. Verifiers use the keyid parameter to match signatures to the correct key.

Is Web Bot Auth the same as mTLS for bots?

No. Both achieve authentication, but they work at different layers. mTLS operates at the TLS connection level and requires the server to actively request a client certificate. Web Bot Auth operates at the HTTP layer and doesn’t require any special TLS configuration. Cloudflare is pursuing both approaches, but is prioritizing HTTP Message Signatures  because they’re easier to deploy incrementally.

Muki Kiboigo

Software Engineer

Muki is a computer engineer from Seattle with a background in systems programming. He is the author of open-source Zig projects like zzz, an HTTP web framework, and tardy, an async runtime. He also runs a website at muki.gg. At Lightpanda, he works on the core browser engine.