# Deploying CrawlGenie

## Local (no Docker)

```bash
npm install
npx playwright install chromium     # once
npm run build
VESSEL_API_KEYS=dev-key npm start    # http://localhost:7070
```

## Docker / Compose (recommended)

```bash
VESSEL_API_KEYS=$(openssl rand -hex 24) docker compose up -d --build
```

The image is based on `node:24-bookworm-slim` with Chromium + OS deps installed by Playwright, runs as an
unprivileged `vessel` user, and `docker-compose.yml` applies the Playwright seccomp profile so Chromium's own
sandbox is active. Prebuilt: `ghcr.io/theshariq/vessel:latest` (published by CI on every green main).
Data (SQLite, screenshots, profiles) lives in the `vessel-data` volume at `/data`.

Put TLS in front and set `VESSEL_PUBLIC_URL=https://vessel.yourdomain.com` so connect URLs are correct.
Caddy is two lines:

```
vessel.yourdomain.com {
  reverse_proxy vessel:7070
}
```

WebSockets (`/live`, `/cdp`) and SSE (`/events/stream`) work through any proxy that supports upgrades
and doesn't buffer; for nginx add `proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade"; proxy_buffering off;`.

## Sizing

Rule of thumb from measurement: a shared-tier session is ~60–130 MB (context + one page, depends on the
site); an isolated session ~250–400 MB. The server process itself is ~80–120 MB.

| Box | Shared sessions | Isolated sessions | Approx cost/mo |
|---|---|---|---|
| 2 vCPU / 4 GB (Hetzner CX23) | 10–15 | 4–6 | ~$7 |
| 4 vCPU / 8 GB (Hetzner CX33) | 25–35 | 10–14 | ~$10 |
| 8 vCPU / 16 GB | 60–80 | 25–30 | ~$20–30 |

Set `VESSEL_MAX_SESSIONS` explicitly; the auto default assumes ~350 MB per session, which is
conservative for shared tier. CPU, not RAM, becomes the limit on JS-heavy sites: budget ~0.25 vCPU per
active session.

## Environment variables

| Variable | Default | Meaning |
|---|---|---|
| `VESSEL_HOST` / `VESSEL_PORT` | `0.0.0.0` / `7070` | bind address |
| `VESSEL_PUBLIC_URL` | `http://localhost:7070` | origin used in returned URLs |
| `VESSEL_API_KEYS` | *(empty = open!)* | comma-separated `key`, `name:key` or `name:key:maxSessions` |
| `VESSEL_DATA_DIR` | `./data` | SQLite + recordings + profiles |
| `VESSEL_WARM_BROWSERS` | `min(2, cpus)` | pre-launched Chromium processes |
| `VESSEL_CONTEXTS_PER_BROWSER` | `8` | shared sessions per process |
| `VESSEL_MAX_SESSIONS` | auto from RAM | hard cap |
| `VESSEL_DEFAULT_TIMEOUT_SEC` / `VESSEL_MAX_TIMEOUT_SEC` | `900` / `21600` | session lifetime |
| `VESSEL_IDLE_TIMEOUT_SEC` | `300` | close after inactivity |
| `VESSEL_HEADLESS` | `true` | set `false` on a desktop to watch |
| `VESSEL_CHROMIUM_ARGS` | | extra flags, space separated |
| `VESSEL_SSRF_GUARD` | `true` | |
| `VESSEL_RECORD_SCREENSHOTS` / `VESSEL_SCREENSHOT_QUALITY` | `true` / `45` | |
| `VESSEL_LOG_LEVEL` | `info` | `debug` enables request logs |
| `VESSEL_TOKEN_PRICE_PER_M` | `3` | USD per 1M input tokens, for the savings estimate |
| `VESSEL_PROXY_POOLS` | | JSON pools config (or put `proxies.json` in the data dir) |
| `VESSEL_WEBHOOK_URL` | | POST approval/session events here |
| `VESSEL_RETENTION_DAYS` | `14` | prune closed sessions' recordings after N days (0 = never) |
| `VESSEL_CORS_ORIGINS` | | `*` or comma list of allowed origins for the API |
| `VESSEL_AUTH_RATE_LIMIT` | `20` | failed auths per IP per minute before 429 |
| `VESSEL_SECRET_KEY` | unset | Encrypts vault secrets at rest (AES-256-GCM). Set it in production. |
(m) => m + "| `VESSEL_PUBLIC_HOME` | same as `VESSEL_SIGNUP` | Serve the marketing site at `/` for visitors without a dashboard cookie; the dashboard is always at `/app`. The installer turns this on. |
| `VESSEL_PUBLIC_TOOLS` | `false` | Free, rate-limited public tools (`/tools/aio-report`, `/tools/llms-txt-checker`) that need no API key. |
| `VESSEL_GOOGLE_CLIENT_ID` / `VESSEL_GOOGLE_CLIENT_SECRET` | unset | Shows **Continue with Google** on `/start` and `/login` (OpenID Connect). Redirect URI to register in Google Cloud: `<VESSEL_PUBLIC_URL>/auth/google/callback` (Google requires HTTPS except on localhost). `VESSEL_OIDC_AUTH_URL` / `_TOKEN_URL` / `_USERINFO_URL` point the same flow at another OIDC provider. |
| `VESSEL_SIGNUP` | `false` | Cloud mode: dashboard sign-up, organisations, API keys minted in the UI, per-org isolation. |\n| `VESSEL_SESSION_SECRET` | `VESSEL_SECRET_KEY` or random | Signs dashboard login cookies (set it so logins survive restarts). |\n| `VESSEL_ORG_MAX_SESSIONS` | `5` | Concurrent-session cap for each new organisation. |\n(m) => m + "| `VESSEL_BACKUP_URL` | unset | `s3://KEY:SECRET@host/bucket/prefix` — hourly encrypted DB backups to R2/S3/B2/MinIO. |\n| `VESSEL_BACKUP_KEY` | unset | Encrypts backups (AES-256-GCM). |
| `VESSEL_CONTACT_TO` | `hello@crawlgenie.com` | Where website "Message us" submissions are e-mailed. Every message is also stored (`GET /v1/contact`, admin). |
| `VESSEL_MAIL_FROM` | `CrawlGenie <hello@crawlgenie.com>` | Sender for those e-mails (the domain must be verified at the mail provider). |
| `VESSEL_RESEND_KEY` | unset | Resend API key for outbound mail. Unset → messages are stored only and the server logs a warning. |\n| `VESSEL_BACKUP_INTERVAL_MIN` / `VESSEL_BACKUP_KEEP` | `60` / `48` | Backup cadence and retention count. |\n| `VESSEL_PLANS` | built-in Free/Pro/Team | Plan catalogue JSON (limits + prices). |\n(m) => m + "| `VESSEL_LLM_API_KEY` / `VESSEL_LLM_PROVIDER` / `VESSEL_LLM_MODEL` / `VESSEL_LLM_BASE_URL` | unset | Operator-level model for `/v1/extract` and the agent; cloud users can store `llm_api_key` in their vault instead. |\n| `VESSEL_CRAWL_MAX_PAGES` | `500` | Cap per crawl job. |\n"
`deploy/helm/vessel` installs nodes as a StatefulSet (one PVC each for SQLite, recordings and profiles),
an optional router Deployment that fronts them, and an optional Ingress:

```bash
helm install vessel deploy/helm/vessel \
  --set apiKeys="ops:$(openssl rand -hex 24):50" \
  --set publicUrl=https://crawlgenie.com \
  --set ingress.enabled=true --set ingress.host=crawlgenie.com --set ingress.className=nginx
# scale: 3 nodes behind the router
helm upgrade vessel deploy/helm/vessel --reuse-values --set nodes.replicas=3 --set router.enabled=true
```

Notes: each node needs ~2–6 GB RAM (see Sizing); `/dev/shm` is an in-memory emptyDir (`nodes.shmSize`);
long-lived WebSockets/SSE need ingress read/send timeouts of 3600 s; put keys in an existing Secret with
`existingSecret`. Routines and profiles are node-local — pin a routine's runs to one node or share the data
dir (NFS) until the shared-store release.

## How changes go live (no Vercel needed)

CrawlGenie is a long-running server with Chromium, not a static site, so it does not deploy to Vercel. The flow is the
same shape as "push → live", just for a server:

1. `git push` to `main` → GitHub Actions runs typecheck, tests, the Docker build and smoke test.
2. On green, CI publishes `ghcr.io/theshariq/vessel:latest` (multi-arch).
3. The VM runs **watchtower** (installed by `deploy/install.sh`), which checks the registry every 5 minutes and
   restarts CrawlGenie on the new image. `/data` is a volume, so session history, profiles, routines, accounts and keys
   survive every update. Live sessions on the old container end (agents reconnect; routines re-run).
4. The homepage (`packages/server/public/home.html`) is served by the same container **and** published to GitHub
   Pages on push, so marketing changes are live in about a minute.

Pin instead of auto-updating: set `image: ghcr.io/theshariq/vessel:0.6.0` in `/opt/vessel/docker-compose.yml` and
remove the watchtower service. Roll back: change the tag and `docker compose up -d`.

## Kubernetes (Helm)

`deploy/helm/vessel` installs nodes as a StatefulSet (one PVC each for SQLite, recordings and profiles), an optional
router Deployment that fronts them, and an optional Ingress:

```bash
helm install vessel deploy/helm/vessel   --set apiKeys="ops:$(openssl rand -hex 24):50"   --set publicUrl=https://crawlgenie.com   --set ingress.enabled=true --set ingress.host=crawlgenie.com --set ingress.className=nginx
# scale: 3 nodes behind the router
helm upgrade vessel deploy/helm/vessel --reuse-values --set nodes.replicas=3 --set router.enabled=true
```

Each node needs ~2–6 GB RAM (see Sizing); `/dev/shm` is an in-memory emptyDir; WebSockets/SSE need ingress
read/send timeouts of 3600 s; put keys in an existing Secret with `existingSecret`. Routines and profiles are
node-local — pin a routine's runs to one node or share the data dir until the shared-store release.

## Scaling out (router mode)

A session lives in exactly one process. Past one box, run N CrawlGenie nodes and one router in front:

```bash
# node A and node B: normal CrawlGenie, same API key, not exposed publicly
docker run -d --name vessel-a -e VESSEL_API_KEYS=$KEY ... ghcr.io/theshariq/vessel:latest
docker run -d --name vessel-b -e VESSEL_API_KEYS=$KEY ... ghcr.io/theshariq/vessel:latest
# router: the public endpoint (dashboard, REST, MCP, WebSockets)
docker run -d --name vessel-router -p 7070:7070 \
  -e VESSEL_MODE=router -e VESSEL_NODES=http://vessel-a:7070,http://vessel-b:7070 \
  -e VESSEL_NODE_KEY=$KEY -e VESSEL_API_KEYS=$KEY -e VESSEL_PUBLIC_URL=https://crawlgenie.com \
  ghcr.io/theshariq/vessel:latest
```

The router places each new session on the node with the most free capacity, remembers session → node
in its own SQLite file, proxies every `/v1/sessions/:id/*` call there (HTTP, SSE, live-view and CDP
WebSockets), merges `GET /v1/sessions`, `/v1/stats`, `/v1/stats/daily` and `/v1/approvals` across nodes,
and forwards routines, profiles, proxies, MCP and tools to the primary node (`VESSEL_NODES[0]`).
Routines and profiles are therefore node-local in this version: keep them on the primary, or share
the data directory between nodes on a network volume. Add nodes by restarting the router with a longer
`VESSEL_NODES` list; drain a node by removing it (its running sessions keep working until they end).

## Backups and retention

**Off-site (recommended on any VM):** create a bucket on Cloudflare R2 (10 GB free), Backblaze B2 (10 GB free) or S3,
then set `VESSEL_BACKUP_URL=s3://ACCESS:SECRET@<account>.r2.cloudflarestorage.com/vessel-backups/prod?region=auto` and
`VESSEL_BACKUP_KEY=$(openssl rand -hex 32)`. The server uploads an encrypted copy of the database every hour and keeps
the last 48; `GET /v1/admin/backups` shows them. Recordings and profiles are not included — snapshot the VM disk for those.


* `data/vessel.db` (WAL mode: copy `.db`, `.db-wal`, `.db-shm` together, or use `sqlite3 .backup`).
* `data/profiles/*.json` are credentials — encrypt at rest.
* `data/sessions/<id>/` grows ~10–30 KB per action; prune by age:
  `find data/sessions -maxdepth 1 -mtime +14 -exec rm -rf {} +`.

## Health

`GET /healthz` for liveness; `GET /v1/stats` for capacity (`sessions.running/max`, `pool.capacity`,
`memoryMb`). Alert when `running/max > 0.8` or when `pool.sharedBrowsers` drops to 0.