CrawlGenie architecture

Tiers, the agent interface, policy, recorder, data.

CrawlGenie is a single Node.js process (plus the Chromium processes it manages) that turns Playwright into a governed, multi-tenant, agent-facing service. This document explains what runs where and why.

┌──────────────────────────────────────────────────────────────────────────────┐
│  Clients                                                                      │
│  REST/SDK (any language)   MCP (Claude, Cursor, Codex…)   CDP (Playwright,   │
│                                                            Puppeteer)        │
└───────────────┬───────────────────────┬─────────────────────────┬────────────┘
                │                       │                         │ ws relay (isolated tier)
┌───────────────▼───────────────────────▼─────────────────────────▼────────────┐
│  Fastify API  ─ auth (API keys) ─ zod validation ─ error mapping             │
│   /v1/sessions/*   /v1/routines/*   /v1/profiles   /v1/approvals   /mcp      │
├──────────────────────────────────────────────────────────────────────────────┤
│  SessionManager                                                              │
│   • capacity + profile locks   • fork (rewind)   • crash recovery            │
│   ┌──────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐     │
│   │ Session      │  │ Policy       │  │ Recorder     │  │ Routines     │     │
│   │ act queue    │  │ domains/SSRF │  │ events+jpeg  │  │ record →     │     │
│   │ snapshot     │  │ resources    │  │ storageState │  │ parameterise │     │
│   │ approvals    │  │ approvals    │  │ SSE stream   │  │ → self-heal  │     │
│   └──────┬───────┘  └──────────────┘  └──────┬───────┘  └──────────────┘     │
├──────────┼───────────────────────────────────┼───────────────────────────────┤
│  BrowserPool (warm Chromium processes)       │  node:sqlite (WAL) + files    │
│   shared tier: BrowserContext per session    │  sessions, events, profiles,  │
│   isolated tier: own process + real CDP port │  approvals, routines          │
└──────────────────────────────────────────────┴───────────────────────────────┘

Session tiers#

The pool keeps VESSEL_WARM_BROWSERS processes with spare capacity, tops up asynchronously after each acquisition, and recycles a process once it is empty and older than an hour to bound renderer leaks. A disconnected process is dropped and replaced; sessions on it close with reason context_closed.

The agent interface#

The core idea is that an agent should never see HTML. Every observation is Playwright's ariaSnapshot({ mode: 'ai' }): a YAML-like accessibility tree where actionable nodes carry [ref=eN]. CrawlGenie's interactive mode (default) filters that further: interactive nodes, headings, short content lines, landmark ancestors; layout wrappers (generic, table/row/cell) and /url: lines are dropped and the tree is re-indented. Real pages land at 2–8 k characters instead of 30–80 k.

Actions reference those refs. The server resolves aria-ref=eN to the element, so there is no selector engineering on the model side. A batch of actions stops at the first failure and returns a fresh snapshot, which is the natural observe→act loop with one round trip per step.

Refs are only valid for the last snapshot taken on that page. Anything that takes another snapshot (including locator.ariaSnapshot()) invalidates them — this is why target descriptions are computed in-page rather than via a nested snapshot.

Policy (enforced server-side)#

Attached at creation, immutable for the session's life, applied on every request through context.route('**/*'):

• allowDomains / blockDomains (wildcards) — applies to sub-resources, redirects and link clicks, not just API-initiated navigations.
• SSRF guard — scheme allow-list, literal private/link-local/metadata ranges, DNS resolution check on every top-level document request. Defence in depth, not a sandbox: Chromium resolves names itself, so production deployments should also put an egress proxy / network policy in front of the browser.
• blockResources, blockTrackers, maxBytes — bandwidth and noise control; images are answered with a 1×1 GIF so layouts don't collapse.
• maxDurationSec, idleTimeoutSec, maxPages, maxActions.
• approvals — destructive holds clicks/Enter on targets whose text matches pay/buy/send/delete/… (plus custom patterns) until a human approves via API or dashboard; all holds everything.

Recorder and rewind#

Each action records an event row, a low-quality JPEG of the active page, and a storageState snapshot (cookies + localStorage). POST /sessions/:id/fork {seq} starts a new session seeded from the storage at that step and navigates to the recorded URL. The original is untouched; branch as often as you like. Events stream over SSE for live dashboards.

Routines#

A routine is the list of successful actions from a session, with each ref-based action annotated by the target's accessible identity (role, name, text) and literal typed values replaced by {{param}} placeholders. Replay takes a full snapshot before each step and re-finds the element by identity (exact role+name → loose → any role → text). Steps execute through Session.act, so policies and approvals still apply. Sensitive param names (pass/secret/token/…) are never stored.

Live view#

Page.startScreencast over CDP pushes JPEG frames only when the page repaints; viewers receive them over a WebSocket and can inject mouse/keyboard events (Input.dispatch*). No VNC/X server. The view follows the agent's active tab.

Data#

node:sqlite in WAL mode holds metadata; screenshots and storage snapshots are files under VESSEL_DATA_DIR/sessions/<id>/. Single-node by design. For several API nodes, put sessions behind a sticky router (a session lives in exactly one process) and move metadata to Postgres.

What CrawlGenie deliberately does not do#

• Solve CAPTCHAs or supply residential proxies. Bring your own (proxy per session); see docs/COMPARISON.md.
• Run an LLM. CrawlGenie is model-agnostic infrastructure; the agent decides, CrawlGenie observes/acts/governs.
• Pretend headless Chromium is undetectable. The UA is normalised (HeadlessChrome → Chrome) and AutomationControlled is disabled; anything beyond that is site-specific and changes monthly.

Section: Build · This page as markdown · Questions? hello@crawlgenie.com