# CrawlGenie architecture CrawlGenie is a single Node.js process (plus the Chromium processes it manages) that turns Playwright into a governed, multi-tenant, agent-facing service. This document explains what runs where and why. ``` ┌──────────────────────────────────────────────────────────────────────────────┐ │ Clients │ │ REST/SDK (any language) MCP (Claude, Cursor, Codex…) CDP (Playwright, │ │ Puppeteer) │ └───────────────┬───────────────────────┬─────────────────────────┬────────────┘ │ │ │ ws relay (isolated tier) ┌───────────────▼───────────────────────▼─────────────────────────▼────────────┐ │ Fastify API ─ auth (API keys) ─ zod validation ─ error mapping │ │ /v1/sessions/* /v1/routines/* /v1/profiles /v1/approvals /mcp │ ├──────────────────────────────────────────────────────────────────────────────┤ │ SessionManager │ │ • capacity + profile locks • fork (rewind) • crash recovery │ │ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ │ │ │ Session │ │ Policy │ │ Recorder │ │ Routines │ │ │ │ act queue │ │ domains/SSRF │ │ events+jpeg │ │ record → │ │ │ │ snapshot │ │ resources │ │ storageState │ │ parameterise │ │ │ │ approvals │ │ approvals │ │ SSE stream │ │ → self-heal │ │ │ └──────┬───────┘ └──────────────┘ └──────┬───────┘ └──────────────┘ │ ├──────────┼───────────────────────────────────┼───────────────────────────────┤ │ BrowserPool (warm Chromium processes) │ node:sqlite (WAL) + files │ │ shared tier: BrowserContext per session │ sessions, events, profiles, │ │ isolated tier: own process + real CDP port │ approvals, routines │ └──────────────────────────────────────────────┴───────────────────────────────┘ ``` ## Session tiers | | shared (default) | isolated | |---|---|---| | Unit | `BrowserContext` in a pooled Chromium | dedicated Chromium (`launchPersistentContext`) | | Create latency | ~100–300 ms (process already warm) | ~400–900 ms | | Memory | ~60–130 MB per context | ~250–400 MB per process | | Raw CDP (`connectOverCDP`) | no (would expose other tenants' targets) | yes, via tokenised ws relay | | Crash blast radius | all contexts in that process | one session | | Per-session proxy | yes (`proxy: {server:'per-context'}` at launch) | yes | The pool keeps `VESSEL_WARM_BROWSERS` processes with spare capacity, tops up asynchronously after each acquisition, and recycles a process once it is empty and older than an hour to bound renderer leaks. A disconnected process is dropped and replaced; sessions on it close with reason `context_closed`. ## The agent interface The core idea is that an agent should never see HTML. Every observation is Playwright's `ariaSnapshot({ mode: 'ai' })`: a YAML-like accessibility tree where actionable nodes carry `[ref=eN]`. CrawlGenie's `interactive` mode (default) filters that further: interactive nodes, headings, short content lines, landmark ancestors; layout wrappers (`generic`, `table/row/cell`) and `/url:` lines are dropped and the tree is re-indented. Real pages land at 2–8 k characters instead of 30–80 k. Actions reference those refs. The server resolves `aria-ref=eN` to the element, so there is no selector engineering on the model side. A batch of actions stops at the first failure and returns a fresh snapshot, which is the natural observe→act loop with one round trip per step. Refs are only valid for the last snapshot taken on that page. Anything that takes another snapshot (including `locator.ariaSnapshot()`) invalidates them — this is why target descriptions are computed in-page rather than via a nested snapshot. ## Policy (enforced server-side) Attached at creation, immutable for the session's life, applied on every request through `context.route('**/*')`: * `allowDomains` / `blockDomains` (wildcards) — applies to sub-resources, redirects and link clicks, not just API-initiated navigations. * SSRF guard — scheme allow-list, literal private/link-local/metadata ranges, DNS resolution check on every top-level document request. Defence in depth, not a sandbox: Chromium resolves names itself, so production deployments should also put an egress proxy / network policy in front of the browser. * `blockResources`, `blockTrackers`, `maxBytes` — bandwidth and noise control; images are answered with a 1×1 GIF so layouts don't collapse. * `maxDurationSec`, `idleTimeoutSec`, `maxPages`, `maxActions`. * `approvals` — `destructive` holds clicks/Enter on targets whose text matches pay/buy/send/delete/… (plus custom patterns) until a human approves via API or dashboard; `all` holds everything. ## Recorder and rewind Each action records an event row, a low-quality JPEG of the active page, and a `storageState` snapshot (cookies + localStorage). `POST /sessions/:id/fork {seq}` starts a new session seeded from the storage at that step and navigates to the recorded URL. The original is untouched; branch as often as you like. Events stream over SSE for live dashboards. ## Routines A routine is the list of successful actions from a session, with each ref-based action annotated by the target's accessible identity (`role`, `name`, `text`) and literal typed values replaced by `{{param}}` placeholders. Replay takes a full snapshot before each step and re-finds the element by identity (exact role+name → loose → any role → text). Steps execute through `Session.act`, so policies and approvals still apply. Sensitive param names (pass/secret/token/…) are never stored. ## Live view `Page.startScreencast` over CDP pushes JPEG frames only when the page repaints; viewers receive them over a WebSocket and can inject mouse/keyboard events (`Input.dispatch*`). No VNC/X server. The view follows the agent's active tab. ## Data `node:sqlite` in WAL mode holds metadata; screenshots and storage snapshots are files under `VESSEL_DATA_DIR/sessions//`. Single-node by design. For several API nodes, put sessions behind a sticky router (a session lives in exactly one process) and move metadata to Postgres. ## What CrawlGenie deliberately does not do * Solve CAPTCHAs or supply residential proxies. Bring your own (`proxy` per session); see docs/COMPARISON.md. * Run an LLM. CrawlGenie is model-agnostic infrastructure; the agent decides, CrawlGenie observes/acts/governs. * Pretend headless Chromium is undetectable. The UA is normalised (`HeadlessChrome` → `Chrome`) and `AutomationControlled` is disabled; anything beyond that is site-specific and changes monthly.