dig2browser

Anti-detection browser automation library for Rust with Web Bot Auth support. Custom CDP + WebDriver + BiDi backends — zero external browser-automation dependencies.

Multi-browser support: Chrome, Edge, Firefox. Built-in anti-detection via 16 dedicated scripts, cookie management (Chrome DPAPI + Firefox plaintext), and agent-friendly DevTools access.

Why

Existing Rust browser libraries (chromiumoxide, fantoccini, thirtyfour) each support one protocol. dig2browser implements all three protocols from scratch in ~7.5K lines:

Protocol	Browsers	What it gives
CDP (Chrome DevTools Protocol)	Chrome, Edge	Full DevTools: network interception, pre-navigation script injection, DOM access, Input events
W3C WebDriver	Chrome, Edge, Firefox	Element interaction, screenshots, cookies, Actions API
WebDriver BiDi	Firefox, Chrome	Pre-navigation scripts (`addPreloadScript`), network interception, typed events — CDP-equivalent for Firefox

One unified API (StealthBrowser / StealthPage) regardless of backend.

Architecture

dig2browser/
├── crates/
│   ├── cdp/        # CDP WebSocket client, 8 typed domains (1600 LOC)
│   ├── webdriver/  # W3C WebDriver REST client (1100 LOC)
│   ├── bidi/       # WebDriver BiDi WebSocket client (780 LOC)
│   ├── stealth/    # 16 JS anti-detection scripts (690 LOC)
│   ├── cookie/     # Chrome DPAPI + Firefox plaintext readers (780 LOC)
│   ├── detect/     # Browser binary detection + launch args (280 LOC)
│   ├── bot_auth/   # Web Bot Auth: Ed25519 signing, JWKS, key management (250 LOC)
│   └── core/       # StealthBrowser, StealthPage, BrowserPool (2400 LOC)
└── src/lib.rs      # Re-export facade

Dependency Graph

dig2browser (facade)
  └── core
        ├── cdp
        ├── webdriver
        ├── bidi
        ├── stealth
        ├── cookie
        ├── detect
        └── bot_auth (web-bot-auth crate)

No circular dependencies. Leaf crates (stealth, cookie, detect) have zero protocol deps.

Quick Start

use dig2browser::{StealthBrowser, LaunchConfig, BrowserPreference};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    // Auto-detects Chrome/Edge, launches with anti-detection enabled
    let browser = StealthBrowser::launch().await?;

    let page = browser.new_page("https://example.com").await?;

    // Get page HTML
    let html = page.html().await?;
    println!("Title length: {}", html.len());

    // Execute JavaScript
    let title = page.eval("document.title").await?;
    println!("Title: {}", title);

    // Find and interact with elements
    let heading = page.find("h1").await?;
    let text = heading.text().await?;
    println!("Heading: {}", text);

    // Screenshot
    let png = page.screenshot().await?;
    std::fs::write("screenshot.png", &png)?;

    browser.close().await?;
    Ok(())
}

Firefox

use dig2browser::{StealthBrowser, LaunchConfig, StealthConfig, BrowserPreference};

let launch = LaunchConfig {
    browser_pref: BrowserPreference::Firefox,
    geckodriver_url: "http://localhost:4444".into(),
    ..Default::default()
};

// Requires geckodriver running: geckodriver --port 4444
let browser = StealthBrowser::launch_with(launch, StealthConfig::default()).await?;

Wait Builder

use std::time::Duration;

// Wait for element to appear
let element = page.wait()
    .at_most(Duration::from_secs(10))
    .every(Duration::from_millis(200))
    .for_element(".results")
    .await?;

// Wait for URL change
page.wait()
    .at_most(Duration::from_secs(5))
    .for_url("/dashboard")
    .await?;

// Wait for JS condition
page.wait()
    .for_condition("window.dataLoaded === true")
    .await?;

Browser Pool

use dig2browser::{BrowserPool, PoolConfig};

let pool = BrowserPool::new(PoolConfig {
    size: 4,
    max_pages_per_browser: 20,
    ..Default::default()
}).await?;

let page = pool.acquire().await?;
page.page().goto("https://example.com").await?;
// Page returned to pool on drop

DevTools Events

Works on all three browsers — CDP events on Chrome/Edge, BiDi events on Firefox.

let mut devtools = page.devtools().await?;
while let Some(event) = devtools.next_event().await {
    match event {
        DevToolsEvent::Network(ev) => println!("Request: {} {}", ev.method, ev.url.unwrap_or_default()),
        DevToolsEvent::Console(ev) => println!("[{}] {}", ev.level, ev.text),
    }
}

Features

Anti-detection (16 scripts, auto-injected)

navigator.webdriver → false
window.chrome mock
Canvas fingerprint randomization
WebGL vendor/renderer spoofing
Plugin/mimeType simulation
Hardware concurrency + device memory
Connection type, battery API, media devices
WebRTC leak prevention
Screen resolution + outer window size
Performance timing noise
UserAgentData branding

Element Interaction

let input = page.find("input[name=query]").await?;
input.type_text("search term").await?;

let button = page.find("button[type=submit]").await?;
button.click().await?;

let result = page.find(".result").await?;
let text = result.text().await?;
let html = result.html().await?;
let bbox = result.bounding_box().await?;

Cookies

Cross-protocol cookie management + reading from browser profiles:

// Read cookies from browser profile (Chrome DPAPI / Firefox plaintext)
use dig2browser::{InterceptConfig, CookieJar};

// Get cookies from current page
let jar = page.get_cookies().await?;

// Set cookies
page.set_cookies(&jar).await?;

PDF Export

use dig2browser::PrintOptions;

let pdf = page.pdf(PrintOptions {
    landscape: true,
    print_background: true,
    ..Default::default()
}).await?;
std::fs::write("page.pdf", &pdf)?;

Crawler (optional, off by default: `--features crawler`)

Ported from the retired standalone dig2crawl crate. Adds agent-driven CSS-selector discovery (a multi-turn Claude session escalates CSS → interactive clicks/scrolls → screenshot+vision, building a reusable site profile), a fast CSS-selector extractor that then runs without an agent, plain-HTTP and browser-driven fetch behind one Fetcher trait, and the dig2crawl CLI (discover / extract / export-spec / fetch / test-selector). The site-profile model, structured-record parsers (selector / JSON-LD / antibot / metadata / embedded SPA JSON), and SQLite job/record storage + CSV/JSONL export live in the browser-agnostic dig2browser-crawler crate (dig2browser_crawler::profile); this crate adds only the parts that need a real browser or an agent CLI transport (gate4agent). A default build of dig2browser links neither.

cargo run --release --features crawler --bin dig2crawl -- \
    discover https://example.com/pricing --goal "VPS plans: name, price, ram"
cargo run --release --features crawler --bin dig2crawl -- \
    extract https://example.com/pricing --profile output/example.com/profile.json

Web Bot Auth

Cryptographic bot identity using RFC 9421 HTTP Message Signatures. Instead of anti-detection scripting, your crawler proves its identity to CDN providers (Cloudflare, Akamai, DataDome, HUMAN Security, AWS) with Ed25519 signatures.

One implementation covers all providers — they all support the same Web Bot Auth standard.

Setup

use dig2browser::bot_auth::*;

// 1. Generate a keypair (or load existing)
let keypair = BotKeyPair::load_or_generate(Path::new("keys/my-bot.key"))?;

// 2. Generate JWKS directory for hosting
let jwks = JwksDirectory::from_keypair(&keypair);
jwks.save_to_file(Path::new("public/.well-known/http-message-signatures-directory"))?;
println!("JWKS:\n{}", jwks.to_json());

// 3. Create identity (from env: BOT_AUTH_JWKS_URL, BOT_AUTH_KEY_PATH)
let identity = BotIdentity::from_env(
    "my-crawler",
    "https://github.com/you/my-crawler",
);
// Or manually:
// let identity = BotIdentity::new(
//     "my-crawler",
//     "https://github.com/you/my-crawler",
//     "https://you.github.io/.well-known/http-message-signatures-directory",
//     "keys/my-bot.key",
// );

// 4. Sign requests
let signer = RequestSigner::from_identity(identity)?;
let headers = signer.sign_request("GET", "https://example.com/data")?;

// 5. Attach to reqwest
let resp = client.get(url)
    .header("Signature-Agent", &headers.signature_agent)
    .header("Signature-Input", &headers.signature_input)
    .header("Signature", &headers.signature)
    .send().await?;

How to Register Your Bot

Provider	Registration	Docs
Cloudflare	Verified Bots form	Web Bot Auth docs
Akamai	Bot Registration	Blog post
DataDome	Bot Authentication	Automatic if JWKS hosted
HUMAN Security	Contact via site	Announcement
AWS Bedrock	AgentCore docs	Automatic

Steps:

Generate keypair: BotKeyPair::generate() or BotKeyPair::load_or_generate(path)
Host the JWKS JSON at a public URL (GitHub Pages works: /.well-known/http-message-signatures-directory)
Register with each provider using the links above (provide your JWKS URL + bot homepage)
Sign all requests with RequestSigner — the 3 headers are added automatically

Environment variables (set in consumer's .env):

Variable	Description
`BOT_AUTH_JWKS_URL`	Public URL where JWKS directory is hosted
`BOT_AUTH_KEY_PATH`	Path to Ed25519 private key (32 bytes raw)

BotIdentity::from_env(name, homepage) reads both from env. Panics if missing.

Security: Never commit your private key (*.key). Add keys/, *.key, and .env to .gitignore.

CDP Domains

Hand-written typed helpers for 10 domains:

Domain	Methods
Target	create, attach, close, list
Page	navigate, content, screenshot, PDF, addScript, frameTree
Runtime	evaluate, callFunctionOn, addBinding
Network	enable, getCookies, setCookie, deleteCookies, getResponseBody
Fetch	enable, continue, fail, fulfill, rewrite headers
DOM	querySelector, getBoxModel, resolveNode, outerHTML, focus, scrollIntoView
Input	mouse click/move, keyboard type/press, touch
Emulation	timezone, UA, device metrics, geolocation, locale, media
Security	ignoreCertificateErrors
Log	enable (events via typed stream)

Typed Event Stream

use dig2browser_cdp::{EventStream, CdpEventType, FetchRequestPaused, NetworkResponseReceived};

let mut fetch_events: EventStream<FetchRequestPaused> = session.event_stream();
while let Some(event) = fetch_events.next().await {
    println!("Intercepted: {} {}", event.resource_type, event.request.url);
}

WebDriver BiDi

Firefox-equivalent of CDP capabilities:

Module	Capabilities
script	`addPreloadScript` (pre-navigation injection), evaluate, callFunction
network	`subscribeNetwork` (events), `addIntercept`, continueRequest, provideResponse, failRequest
browsingContext	navigate, getTree, create/close, screenshot, print
input	performActions, releaseActions
log	subscribe to console/error events

Browser Support

Browser	Protocol	Anti-detection	Status
Chrome	CDP	Full (pre-nav injection)	Production
Edge	CDP	Full (pre-nav injection)	Production
Firefox	WebDriver BiDi	Full (preloadScript)	Production

CLI Tools

`cdp-control` — attach-only CDP panel (Python)

Drive the owner's already-open headed Chrome/Edge. Never launches a browser.

python tools/cdp-control/cdp_control.py tabs
python tools/cdp-control/cdp_control.py --tab dev shot NAME
python tools/cdp-control/cdp_control.py --tab prod click 100 200

See tools/cdp-control/README.md.

dig2browser ships three standalone binaries:

`keygen` — Generate Ed25519 keypair for Web Bot Auth

cargo run --bin keygen -- keys/my-bot.key

`dev-fetch` — DevTools in your terminal

Fetch a URL through the browser and inspect everything — no code needed.

# Basic: fetch URL, show title/size/time
dev-fetch https://example.com

# Full DevTools inspection
dev-fetch https://cloud.vk.com/pricing \
  --fingerprint russian.json \
  --network-log \
  --cookies \
  --console \
  --save-html out.html \
  --save-screenshot out.png

# Execute JS
dev-fetch https://example.com --eval "document.title"

# DOM inspection
dev-fetch https://example.com --dom "div.pricing-card"

# Headed mode + keep open for manual inspection
dev-fetch https://example.com --headed --keep-open 60

# With persistent profile (reuse cookies from cookie-auth)
dev-fetch https://yandex.cloud --profile /tmp/dig2crawl-profiles/yandex.cloud --cookies

Flag	Description
`--fingerprint <PATH>`	JSON fingerprint config (browser, locale, timezone, viewport, anti-detection level)
`--headed`	Visible browser window
`--wait-selector <CSS>`	Wait for element before capturing
`--save-html <PATH>`	Save HTML to file
`--save-screenshot <PATH>`	Save screenshot PNG
`--profile <PATH>`	Persistent browser profile directory
`--network-log`	Show all network requests/responses
`--cookies`	Dump cookies after page load
`--console`	Show console.log/warn/error messages
`--eval <JS>`	Execute JavaScript and print result
`--dom <selector>`	Find elements and print outer HTML
`--keep-open <SECONDS>`	Keep browser open (useful with --headed)

`dig2-wasm-test` — run `cargo test` in a real browser, zero setup

A native wasm-bindgen-test runner. Runs cargo test --target wasm32-unknown-unknown inside a real headless browser without wasm-pack, wasm-bindgen-cli, or a manually-installed driver on PATH — it auto-detects the browser, downloads the matching WebDriver, drives it, and reports results back to cargo.

Set it as the cargo runner in the crate under test:

# .cargo/config.toml
[target.wasm32-unknown-unknown]
runner = "dig2-wasm-test"

Then just:

cargo test --target wasm32-unknown-unknown

dig2-wasm-test (invoked by cargo with the compiled .wasm):

Generates the wasm-bindgen JS shim in-process (no external wasm-bindgen binary).
Serves the shim + test harness from an ephemeral loopback HTTP server.
Auto-detects an installed browser (Chrome → Firefox → Edge), reads its version.
Downloads the matching driver (chromedriver via Chrome-for-Testing, geckodriver latest, msedgedriver) into ~/.cache/dig2browser/drivers/ and caches it.
Spawns the driver, runs the tests headless, scrapes the result, and forwards the exit code (0 = all passed). Driver + browser are killed cleanly on exit.

Test-name filters

# Run only tests whose name contains "ws_binance"
cargo test --target wasm32-unknown-unknown -- ws_binance

# Exact match
cargo test --target wasm32-unknown-unknown -- --exact ws_binance

# Show console.log output inline
cargo test --target wasm32-unknown-unknown -- --nocapture

# Include #[ignore]d tests
cargo test --target wasm32-unknown-unknown -- --include-ignored

Positional arguments (not starting with --) are test-name substring filters. With --exact, the trailing wasm-bindgen hash segment (_<hexsuffix>) is stripped before comparing, so --exact ws_binance matches __wbgt_ws_binance_a1b2c3.

Flag	Effect
`<name>` (positional)	Substring filter — only tests whose name contains `<name>` are run
`--exact`	Exact match instead of substring (hash suffix stripped)
`--nocapture`	Route `console.log` / `console.error` etc. to stdout
`--include-ignored`	Also run `#[ignore]`d tests
`--ignored`	Same as `--include-ignored`

Environment variables

Env var	Effect
`DIG2_WASM_BROWSER`	Force `chrome` / `firefox` / `edge` instead of auto-detect
`WASM_BINDGEN_TEST_TIMEOUT`	Per-run global timeout in seconds (default 20)
`DIG2_WASM_PER_TEST_TIMEOUT`	Stall watchdog: if `#output` stops advancing for this many seconds and no `test result:` has appeared, the runner exits 124 and prints the last output so you can see which test was last running. `0` or unset = disabled. Note: because wasm runs single-threaded the hung test cannot be skipped — the watchdog surfaces the hang quickly instead of waiting for the global timeout.
`DIG2_WASM_HEADLESS`	Set to `0` or `false` to open a visible browser window (useful to eyeball a hung test)
`DIG2_WASM_ESTABLISH_TIMEOUT`	Seconds allowed for the browser/driver establishment phase (`new_session` + initial `goto`). Default 60. If the browser or driver wedges during startup the run is aborted and exits 124 instead of hanging forever. Example: `DIG2_WASM_ESTABLISH_TIMEOUT=30` to fail faster on slow CI.
`DIG2_WASM_BROWSER_ARGS`	Space-separated extra browser flags appended after the built-in flags (`--headless=new`, `--disable-gpu`, `--no-sandbox`, `--disable-dev-shm-usage`, `--user-data-dir=…`) for Chrome and Edge, and after `-headless` for Firefox. Useful for memory/crash tuning under heavy load, e.g. `DIG2_WASM_BROWSER_ARGS="--js-flags=--max-old-space-size=4096 --disable-features=NetworkService"`. Args must not contain spaces. Empty/unset = current behavior.
`CHROMEDRIVER`	Path to a pre-installed chromedriver binary — skips auto-download
`MSEDGEDRIVER`	Path to a pre-installed msedgedriver binary — skips auto-download
`GECKODRIVER`	Path to a pre-installed geckodriver binary — skips auto-download

Each run uses an isolated temporary browser profile (e.g. %TEMP%\dig2wasm-profile-<uuid>). Chrome and Edge hold an exclusive lock on their user-data-dir; without isolation, back-to-back or parallel runs would fail immediately after "resolving driver…" because a still-exiting prior instance holds the profile lock. The profile dir is created before the browser launches and removed on exit (best-effort, same as the harness temp dir).

Exit codes

Code	Meaning
0	All tests passed
1	Test failures, parse error, or driver/browser error
124	Global timeout elapsed or stall watchdog fired (no test progress for `DIG2_WASM_PER_TEST_TIMEOUT` seconds) or establishment timeout (`DIG2_WASM_ESTABLISH_TIMEOUT`) exceeded during `new_session`/`goto`

Version coupling: the test crate's wasm-bindgen version must match the wasm-bindgen-cli-support version dig2browser is built against (currently 0.2.118). A mismatch yields a clear "schema version mismatch" error — pin wasm-bindgen = "=0.2.118" in the test crate.

Process Lifecycle

Browser processes are automatically cleaned up — no zombie Chrome/Edge left behind:

Layer	Mechanism	Covers
Graceful	`browser.close()` sends `Browser.close` CDP command + `child.kill()`	Normal exit
kill_on_drop	`tokio::process::Command::kill_on_drop(true)`	Panic, early return, forgotten close
Drop safety net	`CdpBrowserBackend::drop()` calls `start_kill()`	Edge cases where Child drop doesn't fire

For Firefox/BiDi: geckodriver manages Firefox lifecycle. DELETE /session tells geckodriver to terminate Firefox.

Requirements

Chrome/Edge: No external driver needed — connects directly via CDP
Firefox: Requires geckodriver running (geckodriver --port 4444)
Rust: 1.75+ (2021 edition)

Roadmap

Support the Project

If you find this tool useful, consider supporting development:

Currency	Network	Address
USDT	TRC20	`TNxMKsvVLYViQ5X5sgCYmkzH4qjhhh5U7X`
USDC	Arbitrum	`0xEF3B94Fe845E21371b4C4C5F2032E1f23A13Aa6e`
ETH	Ethereum	`0xEF3B94Fe845E21371b4C4C5F2032E1f23A13Aa6e`
BTC	Bitcoin	`bc1qjgzthxja8umt5tvrp5tfcf9zeepmhn0f6mnt40`
SOL	Solana	`DZJjmH8Cs5wEafz5Ua86wBBkurSA4xdWXa3LWnBUR94c`

Changelog

0.4.12

New dig2-wasm-test binary — a native wasm-bindgen-test runner for cargo test --target wasm32-unknown-unknown. No wasm-pack / wasm-bindgen-cli / manually-installed driver required: auto-detects Chrome/Firefox/Edge, auto-downloads + caches the matching WebDriver, generates the shim in-process via wasm-bindgen-cli-support, serves it from a loopback HTTP server, drives the headless browser, and forwards the exit code. See the "CLI Tools" section.
New wasmtest module exposing the runner internals (shim, harness, server, driver, download).
New detect::version::browser_version — reads the installed Chrome/Edge version on Windows.

0.4.11

StealthPage::set_bypass_csp(enabled: bool) — CDP backend calls Page.setBypassCSP, allowing Runtime.evaluate script injection on CSP-locked sites. BiDi backend is a silent no-op.
StealthPage::add_script_to_evaluate_on_new_document(source: &str) -> String — CDP backend calls Page.addScriptToEvaluateOnNewDocument and returns the identifier (use for removal). BiDi backend is a silent no-op returning "".

License

MIT

Name		Name	Last commit message	Last commit date
crates		crates
docs/plans		docs/plans
examples		examples
patches/saturating-time		patches/saturating-time
src		src
tests		tests
tools		tools
.gitignore		.gitignore
Cargo.lock		Cargo.lock
Cargo.toml		Cargo.toml
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation