Puppeteer, Playwright & other client libraries

Connect Puppeteer, Playwright, Chromedp, or Selenium to Lightpanda in under 10 minutes, and read content that only exists once a page’s JavaScript runs. Get the same automation you already know, running faster and lighter than Chrome.

Steps

Install Lightpanda


curl -fsSL https://pkg.lightpanda.io/install.sh | bash
export PATH="$HOME/.local/bin:$PATH"

Other options: WSL2 on Windows, Docker, package managers.

Install your client library

Puppeteer

Needs Node.js . In a new project directory:


npm init -y
npm pkg set type=module
npm install --save puppeteer-core

Playwright

JavaScript

Needs Node.js . In a new project directory:


npm init -y
npm pkg set type=module
npm install --save playwright-core

Chromedp

Needs Go . In a new project directory:


go mod init quickstart
go get github.com/chromedp/chromedp

Selenium

WebDriver support, classic and BiDi, is new and doesn’t cover the full specs yet. If a command your scripts need is missing, open an issue .

JavaScript

Needs Node.js . In a new project directory:


npm init -y
npm pkg set type=module
npm install selenium-webdriver

Start Lightpanda

Puppeteer


lightpanda serve

Playwright


lightpanda serve

Chromedp


lightpanda serve

Selenium

Selenium needs the webdriver protocol enabled:


lightpanda serve --protocol webdriver

Lightpanda listens on 127.0.0.1:9222 by default, the address the script connects to. Find every serve flag, including host/port binding, CDP connection limits, and logging, in the serve reference.

ℹ️

Respect robots.txt and avoid high-frequency requests against small sites. Pass --obey-robots to have Lightpanda follow robots.txt for you.

Connect and read JS-rendered content

The script opens quotes.toscrape.com/js and prints the first quote. The quotes are rendered by JavaScript, so a plain HTTP fetch returns an empty list: Lightpanda runs that JavaScript before your script reads the page.

Puppeteer

Save as index.js:


'use strict'
 
import puppeteer from 'puppeteer-core'
 
const browser = await puppeteer.connect({
  browserWSEndpoint: 'ws://127.0.0.1:9222',
})
 
const context = await browser.createBrowserContext()
const page = await context.newPage()
 
await page.goto('https://quotes.toscrape.com/js/')
 
const quote = await page.$eval('.quote .text', el => el.textContent)
console.log(quote)
 
await page.close()
await context.close()
await browser.disconnect()

Find more Puppeteer options.

Playwright

JavaScript

Save as index.js:


import { chromium } from 'playwright-core';
 
const browser = await chromium.connectOverCDP('ws://127.0.0.1:9222');
 
const context = await browser.newContext({});
const page = await context.newPage();
 
await page.goto('https://quotes.toscrape.com/js/');
 
const quote = await page.locator('.quote .text').first().textContent();
console.log(quote);
 
await page.close();
await context.close();
await browser.close();

Find more Playwright options.

Chromedp

Save as main.go:


package main
 
import (
	"context"
	"log"
 
	"github.com/chromedp/chromedp"
)
 
func main() {
	allocCtx, cancel := chromedp.NewRemoteAllocator(context.Background(),
		"ws://127.0.0.1:9222", chromedp.NoModifyURL,
	)
	defer cancel()
 
	ctx, cancel := chromedp.NewContext(allocCtx)
	defer cancel()
 
	var quote string
	if err := chromedp.Run(ctx,
		chromedp.Navigate("https://quotes.toscrape.com/js/"),
		chromedp.Text(".quote .text", &quote, chromedp.ByQuery),
	); err != nil {
		log.Fatalf("failed getting the quote: %v", err)
	}
 
	log.Println("quote:", quote)
}

Find more Chromedp options.

Selenium

JavaScript

This example drives the browser over BiDi: request a BiDi session by setting the webSocketUrl capability to true. Without it, the session uses classic WebDriver commands (driver.get(), findElement(), …). Save as index.js:


'use strict'
 
import { Builder, Capabilities } from 'selenium-webdriver'
import { BrowsingContext } from 'selenium-webdriver/bidi/generated/browsing_context.js'
import { Script } from 'selenium-webdriver/bidi/generated/script.js'
 
const caps = new Capabilities()
caps.set('browserName', 'lightpanda')
caps.set('webSocketUrl', true)
 
const driver = await new Builder()
  .usingServer('http://127.0.0.1:9222')
  .withCapabilities(caps)
  .build()
 
const browsingContext = await BrowsingContext.create(driver)
const script = await Script.create(driver)
 
const { context } = await browsingContext.create({ type: 'tab' })
await browsingContext.navigate({ context, url: 'https://quotes.toscrape.com/js/', wait: 'complete' })
 
const quote = await script.evaluate({
  expression: "document.querySelector('.quote .text').textContent",
  target: { context },
  awaitPromise: false,
})
console.log(quote.result.value)
 
await browsingContext.close({ context })
await driver.quit()

Find more Selenium options.

Run it

Puppeteer


node index.js

Playwright

JavaScript


node index.js

Chromedp


go run main.go

Selenium

JavaScript


node index.js

You should see the first quote printed to your terminal. For more complex examples, like filling a form and waiting for dynamic results, see Interact with a webpage.

Go to production

Switch to Lightpanda Cloud by exporting your token and connecting over wss:// instead of ws://:


export LPD_TOKEN="your token here"

Puppeteer


const browser = await puppeteer.connect({
  browserWSEndpoint: 'wss://euwest.cloud.lightpanda.io/ws?token=' + process.env.LPD_TOKEN,
})

Playwright


const browser = await chromium.connectOverCDP(
  'wss://euwest.cloud.lightpanda.io/ws?token=' + process.env.LPD_TOKEN,
);

Chromedp


import (
	"context"
	"log"
	"os"
 
	"github.com/chromedp/chromedp"
)
 
func main() {
	allocCtx, cancel := chromedp.NewRemoteAllocator(context.Background(),
		"wss://euwest.cloud.lightpanda.io/ws?token="+os.Getenv("LPD_TOKEN"), chromedp.NoModifyURL,
	)
	// ...

Puppeteer, Playwright & other client libraries

Steps

Install Lightpanda

Install your client library

Puppeteer

Playwright

JavaScript

Python

Chromedp

Selenium

JavaScript

Python

Start Lightpanda

Puppeteer

Playwright

Chromedp

Selenium

Connect and read JS-rendered content

Puppeteer

Playwright

JavaScript

Python

Chromedp

Selenium

JavaScript

Python

Run it

Puppeteer

Playwright

JavaScript

Python

Chromedp

Selenium

JavaScript

Python

Go to production

Puppeteer

Playwright

Chromedp

Selenium