Sign inStart free
Home/Docs/How CrawlGenie works — a walkthrough for the owner

How CrawlGenie works — a walkthrough for the owner

Modes, a session end to end, routines, profiles, secrets, how to test.

You run one server (Docker image). It owns a pool of Chromium browsers. Two kinds of users talk to it:

  • Humans use the dashboard (https://your-domain/) to watch, approve, replay, teach routines and manage keys/secrets.
  • Agents / code use the API (/v1/...), MCP (/mcp) or the function-calling tools (/v1/tools). They never see the dashboard; they get compact snapshots and act by ref.

Who can use it: two modes#

Self-hosted (default)Cloud mode (VESSEL_SIGNUP=true)
Who gets inKeys you put in VESSEL_API_KEYSAnyone who signs up in the dashboard (email + password)
Where accounts livenowhere — keys are configusers, orgs, api_keys tables in /data/vessel.db
Isolationall keys see everything (optionally quotas per key)each organisation sees only its own sessions, profiles, routines, secrets, keys; env keys are admins
Keysfixedminted/revoked in Setup → API keys, shown once, stored hashed

So: yes, the database is in-house. Sign-ups are rows in SQLite on your VM; nothing goes to a third party.

A session, end to end#

  1. Create — POST /v1/sessions (or the New session button). Everything is optional:
    • url – where to start.
    • tier – shared (pooled Chromium, fastest, default) or isolated (own Chromium + raw CDP).
    • profile – resume a saved login; profileLock exclusive by default.
    • proxy – { pool, country, sticky } or your own { server, username, password }.
    • policy – the guardrails: allowDomains, denyDomains, blockResources (images/media/fonts), maxActions, maxDurationSec, idleTimeoutSec, maxBytes, approvals: { mode: off|destructive|all, patterns }, record.
    • snapshotDefaults – { mode: interactive|full, maxChars, delta }.
    • viewport, userAgent, locale, timezoneId, extensions (isolated), metadata (labels for the dashboard). You get back SessionInfo: id, liveUrl (dashboard page), cdpUrl (isolated), expiresAt, usage.
  2. Observe — GET /v1/sessions/:id/snapshot returns the page as a short accessibility tree with [ref=eN] markers (2–8k chars instead of 50–500k of HTML). delta=true returns only what changed.
  3. Act — POST /v1/sessions/:id/act { actions: [...] } with click/fill/type/press/select/check/scroll/ upload/goto/wait/assert/evaluate/tab/dialog by ref. The response includes the next snapshot, so one round trip per step. Credentials go in as {{secret:name}}.
  4. Guardrails run server-side — a navigation outside allowDomains is blocked; a click that looks destructive (pay/send/delete, or your patterns) is held until a human approves it in the dashboard (or via POST .../approvals/:aid, or your webhook).
  5. Everything is recorded — each step with a screenshot and storage state. The dashboard shows the live screen (WebSocket screencast), the timeline, the exact snapshot the model saw, usage and the network log. You can take over with mouse/keyboard, fork a new session from any step, or save as routine.
  6. Close — DELETE /v1/sessions/:id, or idle/max timers do it. usage reports actions, bandwidth, observation tokens read vs raw, and the estimated dollars saved.

Routines, profiles, secrets, one-shots#

  • Routine = a recorded flow with {{params}}; replays re-find elements by role + name (self-healing), cost zero model tokens, and can carry assert steps so they double as monitors. Run from the dashboard, POST /v1/routines/:name/run, or the browser_run_routine tool.
  • Profile = cookies + localStorage saved from a session; pass profile on create to resume logged in.
  • Secret = a credential stored server-side; the model types the placeholder, the server types the value.
  • One-shot = POST /v1/scrape, GET /v1/screenshot, GET /v1/pdf when you only want the page.

How to test it end to end right now#

On your own machine (already running at http://localhost:7070 during development) or on the VM after deploy/install.sh:

  1. Open the dashboard → Get started. Click New session, URL https://news.ycombinator.com, approvals destructive. Watch the live view; open Snapshot and note the token count vs raw.
  2. Click Take over, log in to any site you own, then Save profile. Create another session with that profile: it starts logged in.
  3. Scrub the Timeline, click Save as routine, then run it from Routines — zero model tokens.
  4. Secrets → add demo_pw; in a session, fill a password field with {{secret:demo_pw}} (via API or MCP) and confirm the recording shows only the placeholder.
  5. Hand it to a model: claude mcp add --transport http vessel http://localhost:7070/mcp --header "Authorization: Bearer <key>" and ask Claude to "open example.com and tell me the heading". Or pip install vessel-browser and run packages/python/tests/smoke.py.
  6. Cloud mode: start with VESSEL_SIGNUP=true, open the dashboard, sign up, mint a key under API keys, and repeat step 5 with that key. Sign up a second account and confirm it cannot see the first one's sessions.

Section: Build · This page as markdown · Questions? hello@crawlgenie.com