Deploying CrawlGenie

Docker, Compose, Kubernetes, env vars, scaling, backups.

Local (no Docker)#

npm install
npx playwright install chromium     # once
npm run build
VESSEL_API_KEYS=dev-key npm start    # http://localhost:7070

Docker / Compose (recommended)#

VESSEL_API_KEYS=$(openssl rand -hex 24) docker compose up -d --build

The image is based on node:24-bookworm-slim with Chromium + OS deps installed by Playwright, runs as an unprivileged vessel user, and docker-compose.yml applies the Playwright seccomp profile so Chromium's own sandbox is active. Prebuilt: ghcr.io/theshariq/vessel:latest (published by CI on every green main). Data (SQLite, screenshots, profiles) lives in the vessel-data volume at /data.

Put TLS in front and set VESSEL_PUBLIC_URL=https://vessel.yourdomain.com so connect URLs are correct. Caddy is two lines:

vessel.yourdomain.com {
  reverse_proxy vessel:7070
}

WebSockets (/live, /cdp) and SSE (/events/stream) work through any proxy that supports upgrades and doesn't buffer; for nginx add proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection "upgrade"; proxy_buffering off;.

Sizing#

Rule of thumb from measurement: a shared-tier session is ~60–130 MB (context + one page, depends on the site); an isolated session ~250–400 MB. The server process itself is ~80–120 MB.

Set VESSEL_MAX_SESSIONS explicitly; the auto default assumes ~350 MB per session, which is conservative for shared tier. CPU, not RAM, becomes the limit on JS-heavy sites: budget ~0.25 vCPU per active session.

Environment variables#

(m) => m + "| VESSEL_PUBLIC_HOME | same as VESSEL_SIGNUP | Serve the marketing site at / for visitors without a dashboard cookie; the dashboard is always at /app. The installer turns this on. | | VESSEL_PUBLIC_TOOLS | false | Free, rate-limited public tools (/tools/aio-report, /tools/llms-txt-checker) that need no API key. | | VESSEL_GOOGLE_CLIENT_ID / VESSEL_GOOGLE_CLIENT_SECRET | unset | Shows Continue with Google on /start and /login (OpenID Connect). Redirect URI to register in Google Cloud: <VESSEL_PUBLIC_URL>/auth/google/callback (Google requires HTTPS except on localhost). VESSEL_OIDC_AUTH_URL / _TOKEN_URL / _USERINFO_URL point the same flow at another OIDC provider. | | VESSEL_SIGNUP | false | Cloud mode: dashboard sign-up, organisations, API keys minted in the UI, per-org isolation. |\n| VESSEL_SESSION_SECRET | VESSEL_SECRET_KEY or random | Signs dashboard login cookies (set it so logins survive restarts). |\n| VESSEL_ORG_MAX_SESSIONS | 5 | Concurrent-session cap for each new organisation. |\n(m) => m + "| VESSEL_BACKUP_URL | unset | s3://KEY:SECRET@host/bucket/prefix — hourly encrypted DB backups to R2/S3/B2/MinIO. |\n| VESSEL_BACKUP_KEY | unset | Encrypts backups (AES-256-GCM). | | VESSEL_CONTACT_TO | hello@crawlgenie.com | Where website "Message us" submissions are e-mailed. Every message is also stored (GET /v1/contact, admin). | | VESSEL_MAIL_FROM | CrawlGenie <hello@crawlgenie.com> | Sender for those e-mails (the domain must be verified at the mail provider). | | VESSEL_RESEND_KEY | unset | Resend API key for outbound mail. Unset → messages are stored only and the server logs a warning. |\n| VESSEL_BACKUP_INTERVAL_MIN / VESSEL_BACKUP_KEEP | 60 / 48 | Backup cadence and retention count. |\n| VESSEL_PLANS | built-in Free/Pro/Team | Plan catalogue JSON (limits + prices). |\n(m) => m + "| VESSEL_LLM_API_KEY / VESSEL_LLM_PROVIDER / VESSEL_LLM_MODEL / VESSEL_LLM_BASE_URL | unset | Operator-level model for /v1/extract and the agent; cloud users can store llm_api_key in their vault instead. |\n| VESSEL_CRAWL_MAX_PAGES | 500 | Cap per crawl job. |\n" deploy/helm/vessel installs nodes as a StatefulSet (one PVC each for SQLite, recordings and profiles), an optional router Deployment that fronts them, and an optional Ingress:

helm install vessel deploy/helm/vessel \
  --set apiKeys="ops:$(openssl rand -hex 24):50" \
  --set publicUrl=https://crawlgenie.com \
  --set ingress.enabled=true --set ingress.host=crawlgenie.com --set ingress.className=nginx
# scale: 3 nodes behind the router
helm upgrade vessel deploy/helm/vessel --reuse-values --set nodes.replicas=3 --set router.enabled=true

Notes: each node needs ~2–6 GB RAM (see Sizing); /dev/shm is an in-memory emptyDir (nodes.shmSize); long-lived WebSockets/SSE need ingress read/send timeouts of 3600 s; put keys in an existing Secret with existingSecret. Routines and profiles are node-local — pin a routine's runs to one node or share the data dir (NFS) until the shared-store release.

How changes go live (no Vercel needed)#

CrawlGenie is a long-running server with Chromium, not a static site, so it does not deploy to Vercel. The flow is the same shape as "push → live", just for a server:

1. git push to main → GitHub Actions runs typecheck, tests, the Docker build and smoke test.
2. On green, CI publishes ghcr.io/theshariq/vessel:latest (multi-arch).
3. The VM runs watchtower (installed by deploy/install.sh), which checks the registry every 5 minutes and restarts CrawlGenie on the new image. /data is a volume, so session history, profiles, routines, accounts and keys survive every update. Live sessions on the old container end (agents reconnect; routines re-run).
4. The homepage (packages/server/public/home.html) is served by the same container and published to GitHub Pages on push, so marketing changes are live in about a minute.

Pin instead of auto-updating: set image: ghcr.io/theshariq/vessel:0.6.0 in /opt/vessel/docker-compose.yml and remove the watchtower service. Roll back: change the tag and docker compose up -d.

Kubernetes (Helm)#

deploy/helm/vessel installs nodes as a StatefulSet (one PVC each for SQLite, recordings and profiles), an optional router Deployment that fronts them, and an optional Ingress:

helm install vessel deploy/helm/vessel   --set apiKeys="ops:$(openssl rand -hex 24):50"   --set publicUrl=https://crawlgenie.com   --set ingress.enabled=true --set ingress.host=crawlgenie.com --set ingress.className=nginx
# scale: 3 nodes behind the router
helm upgrade vessel deploy/helm/vessel --reuse-values --set nodes.replicas=3 --set router.enabled=true

Each node needs ~2–6 GB RAM (see Sizing); /dev/shm is an in-memory emptyDir; WebSockets/SSE need ingress read/send timeouts of 3600 s; put keys in an existing Secret with existingSecret. Routines and profiles are node-local — pin a routine's runs to one node or share the data dir until the shared-store release.

Scaling out (router mode)#

A session lives in exactly one process. Past one box, run N CrawlGenie nodes and one router in front:

# node A and node B: normal CrawlGenie, same API key, not exposed publicly
docker run -d --name vessel-a -e VESSEL_API_KEYS=$KEY ... ghcr.io/theshariq/vessel:latest
docker run -d --name vessel-b -e VESSEL_API_KEYS=$KEY ... ghcr.io/theshariq/vessel:latest
# router: the public endpoint (dashboard, REST, MCP, WebSockets)
docker run -d --name vessel-router -p 7070:7070 \
  -e VESSEL_MODE=router -e VESSEL_NODES=http://vessel-a:7070,http://vessel-b:7070 \
  -e VESSEL_NODE_KEY=$KEY -e VESSEL_API_KEYS=$KEY -e VESSEL_PUBLIC_URL=https://crawlgenie.com \
  ghcr.io/theshariq/vessel:latest

The router places each new session on the node with the most free capacity, remembers session → node in its own SQLite file, proxies every /v1/sessions/:id/* call there (HTTP, SSE, live-view and CDP WebSockets), merges GET /v1/sessions, /v1/stats, /v1/stats/daily and /v1/approvals across nodes, and forwards routines, profiles, proxies, MCP and tools to the primary node (VESSEL_NODES[0]). Routines and profiles are therefore node-local in this version: keep them on the primary, or share the data directory between nodes on a network volume. Add nodes by restarting the router with a longer VESSEL_NODES list; drain a node by removing it (its running sessions keep working until they end).

Backups and retention#

Off-site (recommended on any VM): create a bucket on Cloudflare R2 (10 GB free), Backblaze B2 (10 GB free) or S3, then set VESSEL_BACKUP_URL=s3://ACCESS:SECRET@<account>.r2.cloudflarestorage.com/vessel-backups/prod?region=auto and VESSEL_BACKUP_KEY=$(openssl rand -hex 32). The server uploads an encrypted copy of the database every hour and keeps the last 48; GET /v1/admin/backups shows them. Recordings and profiles are not included — snapshot the VM disk for those.

• data/vessel.db (WAL mode: copy .db, .db-wal, .db-shm together, or use sqlite3 .backup).
• data/profiles/*.json are credentials — encrypt at rest.
• data/sessions/<id>/ grows ~10–30 KB per action; prune by age: find data/sessions -maxdepth 1 -mtime +14 -exec rm -rf {} +.

Health#

GET /healthz for liveness; GET /v1/stats for capacity (sessions.running/max, pool.capacity, memoryMb). Alert when running/max > 0.8 or when pool.sharedBrowsers drops to 0.

Section: Run · This page as markdown · Questions? hello@crawlgenie.com