# Changelog

## 0.11.0 — 2026-10-13
- Website: menus open on hover; one "why CrawlGenie exists" comparison (one platform instead of Browserbase + Firecrawl + TinyFish + Apify) with the detailed table folded underneath; chat bubble on every page (Ask CrawlGenie from the docs, or Message us → `POST /site/contact`, stored and e-mailed to `VESSEL_CONTACT_TO`); support address hello@crawlgenie.com in footer, legal pages and Organization schema; favicon, app icons, web manifest and social card from the new Spark G mark; meta descriptions within 160 characters.
- Dashboard: Token savings in the sidebar under Overview; the "+ Custom agent" button on Agents opens the create dialog again; Overview sessions table scrolls instead of cropping and shows Replay/Open without hovering; help-bot excerpts no longer split sentences at e-mail addresses.
- Public deployments: `/docs` and `/docs/<page>` are server-rendered HTML for visitors and crawlers (same markdown as the dashboard); app links opened while signed out redirect to `/login?next=…` and return after sign-in; unknown routes return a real 404 page; sign-out goes to the sign-in page; images and icons cache for a day.
- Plan ladder: monthly quotas for reads, searches and agent runs (REST and MCP), paid-only alerts/webhooks, scheduled routines, isolated tier, proxy pools, per-plan recording retention; Free resized to 120 browser-minutes with every product included.
- Plans: per-plan capacity (active watches, team members, memory documents, batch size) enforced with `402 plan_limit` + upgrade link; limits live in `public/pricing.json`.
- Dashboard: Get started checklist (separate from the onboarding wizard), behaviour-based recommendations, month strip on the Overview, session Pages/Console/Files tabs and expiry, tested setup prompt for coding agents, `/v1/insights`.
- Sign-in page: standalone `/start`, `/signup` and `/login` for public deployments (value slides, Google button), dashboard stays at `/app`.
- "Continue with Google" (OpenID Connect) with account linking by verified e-mail; `VESSEL_GOOGLE_CLIENT_ID/SECRET`.
- Vault: `{{totp:NAME}}` one-time codes (RFC 6238) for 2-factor logins.
- `VESSEL_PUBLIC_HOME`: marketing site at `/`; installer sets it with sign-up, public tools, log rotation and swap.

### Added
- **Web agents**: `schema` for structured results; `POST /v1/agent/batch` (one goal across many URLs or many goals) with a concurrency cap and
  `GET /v1/agent/batches/:id` for counts, results and usage.
- **Continuous reports for every watch**: `/v1/watches/:id/series` (metrics per run: counts, new rows, scores, column means) and
  `/v1/watches/:id/report` (7-day markdown or JSON: latest run, trend, alerts, new rows); trend sparklines on the watch page.
- **SEO + AIO report** agent (`seo_aio_report`), free public tools pages, in-app docs with the retrieval-only assistant.
- **Marketing site v2** on the "Graphite & Lime" theme (five theme directions in `/brand/themes.html`), repositioned on LLM token savings;
  web agents solution page; Enterprise private-deployment plan.

### Changed
- The product is no longer presented as open source; package licenses set to UNLICENSED and `private: true`. Code published before this
  commit remains under its original Apache-2.0 terms wherever it was copied.

## 0.10.0 — 2026-10-12

### Added
- **In-house documentation and help**: every server serves its docs at `/docs` (search, table of contents, raw `.md`, `/llms.txt`,
  `/llms-full.txt`) and an **Ask CrawlGenie** assistant that answers from the docs with retrieval only (`POST /help/ask`, public, no model).
  New pages: Concepts, Use cases by industry, Documents & media, Knowledge base, Troubleshooting, Signed agent identity; the Extractor
  library page is generated from the running catalogue. Dashboard links no longer point at GitHub.
- **Knowledge base** (`/v1/knowledge`, MCP `knowledge_add` / `knowledge_search`): collections of pages, documents, crawl output, watch rows
  and notes, chunked and indexed with SQLite FTS5 (BM25 + porter); questions return passages with sources and a token receipt. Dashboard page.
- **Signed agent identity (Web Bot Auth)**: Ed25519 key per deployment, `/.well-known/http-message-signatures-directory`, `GET /v1/identity`,
  `policy.signedAgent` / workspace default; navigations carry RFC 9421 signatures (tag `web-bot-auth`) so sites can verify the agent.
- **Documents**: PDF `pages` ranges and table-to-columns reconstruction; ODT/ODS/ODP and RTF parsing; Google Docs/Sheets/Slides, Drive,
  Dropbox and GitHub blob links normalised to export/raw URLs in `/v1/parse`, `/v1/scrape`, `web_answer`, `web_research`.
- **Reading less**: `granularity: "sentence"` (highlights: matching sentences, table rows, code lines), `redactPII`, `cache: prefer|only`
  on `/v1/scrape`; `POST /v1/answer {urls, q}` and `web_research {urls}` for one question across several sources.
- **Extractors**: `auto_list` (repeated blocks without CSS), `pricing_page`, `github_issues`, `search_watch` (web-scale monitor: new results
  for a query), `brand_assets`, `llms_txt`, `page_media` (every file/video/image on a page with the next step); `css_extract` gained `scroll`.
- SDKs: `knowledge.*`, `help.*`, `answer()`, `identity()`, parse options (TypeScript); `knowledge_*`, `answer`, `help_ask`, `identity` (Python).
- Docker image ships `docs/` and `CHANGELOG.md`.

### Fixed
- Page-range PDF parses no longer trigger OCR because of the per-page word heuristic; pdf.js synthetic space runs no longer defeat column detection.

## 0.9.0 — 2026-10-12

### Added
- **Savings by surface**: every session is tagged with the channel that created it (REST API, MCP, tools API, dashboard,
  built-in agents, or the OAuth connector that holds the key, e.g. `connector:ChatGPT`); `GET /v1/stats/savings` returns
  `byChannel` and the Token savings page shows a per-surface table.
- **Image OCR** (PNG/JPEG/WEBP/GIF/BMP via `/v1/parse`), **YouTube transcripts** (`youtube_transcript` extractor and
  `web_answer` on a video URL: caption track → timestamped paragraphs, no speech model), Playground **drop zone** for local files.
- Security: cross-site consent submissions are refused, OAuth endpoints are rate-limited per IP, the OCR document handoff is
  one-time and same-origin only, the Chrome extension no longer requests host permissions. `scripts/ab-connector.sh`
  compares cost and answers of Claude Code's built-in web tools vs the CrawlGenie connector on the same tasks.
- **OAuth 2.1 for MCP connectors**: CrawlGenie is an authorization server (`/.well-known/oauth-authorization-server`,
  `/.well-known/oauth-protected-resource`, dynamic client registration, PKCE, consent page). ChatGPT, claude.ai,
  Claude Desktop, Cursor and Gemini CLI add CrawlGenie as a custom connector; tokens are revocable workspace keys.
  Verified end to end with the official MCP client over a public HTTPS tunnel and with Claude Code (`claude mcp add`).
- **Cheapest-read tools**: `web_answer` (URL or document + question → only the matching passages) and `web_research`
  (search → read → focus, one call) with a **token receipt** on every result; `maxTokens` budgets on snapshot / text /
  scrape / browser_snapshot (exact BPE truncation).
- **Documents**: OCR for scanned PDFs (pages rendered by a CrawlGenie browser with pdf.js, read by tesseract.js; auto when
  the text layer is empty, `ocr: force|off`), XLSX (sheets → markdown tables) and PPTX (slides) parsing; all routed
  automatically from `/v1/scrape` and `web_answer`.
- **Scrape** gained `actions` (click/scroll/type before extracting) and `changeTracking` (new / same / changed + diff
  per owner and URL, so agents skip unchanged pages). **Crawl** gained `/answer` (passages across all pages with
  sources) and `/export?format=jsonl|md|csv|pdf`.
- **Outcome-first UI**: five-step onboarding (role → goal → connect your agent → try three runs → map), Overview
  "Start from a goal" + vocabulary with token cost per concept + zero-token-runs KPI, a "Do something with this page"
  panel on every session (ask, read, extract tables, links, download, instruct an agent, keep), next-step actions on
  crawl and watch pages, Playground tabs for monitor and agent plus **Get code** (curl / Python / TypeScript) on every form.
- Homepage ROI calculator, Chrome extension scaffold (`integrations/chrome-extension`), docs/INTEGRATIONS.md,
  `scripts/mcp-oauth-client.mjs`, SDK `tools.definitions/call/executable`, Python `tools()/call_tool()`.

## 0.8.0 — 2026-10-12

### Added
- **Extractor store**: 59 built-in agents across 14 verticals (54 extractors + 5 monitors) (jobs, e-commerce, local & leads, reviews, social, news,
  SEO, finance, real estate, travel, research, government/legal/health, events, any site) — a declarative recipe engine
  (`html` / `json` / `feed` sources) plus hand-written extractors (schema.org structured data, website contacts,
  tech-stack fingerprint, YouTube search/channel, SEC EDGAR, PubMed, sitemap monitor, HN hiring, JSON API watcher).
  `GET /v1/extractors` (grouped by vertical), `POST /v1/extractors/{id}/run` (sync rows, `?format=csv`), MCP/tools
  `vessel_extract`, `GET /v1/watches/{id}/dataset` (rows merged across runs with first/last seen). Live-tested from a
  consumer IP without a proxy: 50 of 54 extractors return rows; Yelp and eBay are labelled "needs proxy" and say so at run time.
- **Question-focused reading**: `q=` on `/v1/sessions/{id}/text`, `/v1/scrape`, `/v1/parse` and the
  `browser_extract_text` tool returns only the passages that answer the question (BM25-style, no model) and reports
  the measured token saving.
- **Document parsing** without a browser: `POST /v1/parse` for PDF (pdf.js text layer), DOCX (own zip + WordprocessingML
  reader: headings, lists, tables → markdown) and text/CSV/JSON; `/v1/scrape` routes document URLs there automatically.
- **Search + read**: `GET /v1/search?content=true` attaches markdown of the top hits in one call.
- **Agent onboarding**: `GET /skill.md` (also `/agent-onboarding/SKILL.md`), dashboard **Setup for agents** page with
  copy-ready MCP / SDK / CLI snippets, and a client CLI (`npx vessel-browser-sdk vessel-cli …`).
- **Settings → General** (server-side, admin): workspace name, default and idle session timeouts (apply to new
  sessions), default blocked resources, webhook URL; API key reveal / copy / regenerate; `GET|PUT /v1/settings`.
- Self-hosted admins mint revocable API keys from the dashboard (implicit Workspace org); accent picker (cobalt, coral,
  amber, sapphire, rose); `POST /v1/tick` so scale-to-zero hosts or an external cron can run due watches/schedules;
  `deploy/github-actions/vessel-extract.yml` runs extractors on GitHub Actions for free; HOSTING.md free-tier table.

### Changed
- `VESSEL_SSRF_GUARD=false` is now honoured by direct fetches (JSON/feed/document sources), not only by navigation.
- Watch auto-names use any subject field (query, domain, subreddit…), not only URLs.

## 0.7.0 — 2026-10-12

### Added
- **Extractors** (Apify-style actors, no LLM): `css_extract` (any page → rows via CSS, pagination), `web_search_results`
  (rank tracking), `google_maps_places`, `trustpilot_reviews`, `indeed_jobs`, `amazon_products` (price-change alerts),
  `reddit_posts` (feeds + search, no login), `linkedin_company` (public card), `producthunt_today`, `g2_reviews`
  (needs a residential pool — reports the wall honestly). Rows are deduplicated across runs ("12 new rows"), exported with
  `GET /v1/watches/:id/runs/:runId/export?format=csv|json&only=all|new`. Dashboard: Agents → Extractors, generic table
  with "New only" + CSV/JSON; watch sheet gets proxy-pool / login-profile options. Playbook catalogue now carries `category`.
- Web search falls back DuckDuckGo → **Brave** → Bing (Brave serves full snippets to a headless browser without a key).

### Changed
- **Interception-free policy enforcement.** Playwright request interception turned out to be a bot signal on its own
  (Reddit: CAPTCHA with a pass-through `route('**/*')`, normal page without it). Trackers, blocked domains,
  image/font/media/CSS blocks and literal private hosts are now enforced through Chromium's URL blocklist via CDP;
  interception is used only for allowlists, byte budgets, script/XHR blocks or the new `policy.strictNetwork: true`.
  Blocked-request metering unchanged (`usage.requestsBlocked`).
- Extractors retry navigation once on connection resets and survive hydration redirects (Trustpilot, LinkedIn).

## 0.6.0 — 2026-10-12

### Added
- **Crawl / map / batch-scrape / extract** (Firecrawl-shaped): `POST /v1/crawl` (site crawl with limits, include/exclude globs,
  robots.txt, N browser workers, job status + paginated pages), `POST /v1/batch-scrape`, `POST /v1/map` (sitemaps + links),
  `POST /v1/extract` (prompt + JSON Schema → structured data via the caller's own LLM key; OpenAI-compatible, Anthropic, Gemini),
  `GET /v1/llm`. Markdown reader upgraded with readability-style main-content selection, boilerplate removal and page metadata.
- Dependency-free LLM client (`src/llm/client.ts`) shared by extract and the upcoming agent runner.
- **Team invites** by link (`/v1/invites`, `/auth/join`, members list) — no email provider required; dashboard "API keys & team".
- **Web search** through CrawlGenie's own browser (`GET /v1/search`, tool `browser_search`), DuckDuckGo with Bing fallback, no API key.
- **Built-in agents (no API key)**: change monitor, new-posts digest, price & stock tracker, SEO audit, broken-link checker — stateful watches with diffs, cron schedules and per-watch webhooks (`/v1/playbooks`, `/v1/watches`); dashboard **Agents** gallery + watch pages with rendered diffs/reports.
- **Exact token metering**: every observation counted with the o200k tokenizer (read / full tree / HTML); `/v1/stats/savings`, `POST /v1/measure`; dashboard **Token savings** page with price-model selector and a live "Measure any URL" check.
- **Agent**: `POST /v1/agent/run` — natural-language tasks executed by a built-in observe→act loop with the caller's own model key (OpenAI-compatible/Anthropic/Gemini); per-step thoughts, usage and observation tokens; dashboard **Agent** page with live steps and result.
- **Auto-deploy**: `deploy/install.sh` now adds watchtower — every green push to main is live on the VM within ~5 minutes (docs/DEPLOY.md “How changes go live”).
- **Schedules**: cron-triggered routine runs with timezone support, per-schedule webhooks, run history and a preview endpoint; dashboard **Schedules** page with presets.
- Dashboard **Crawls & extract** pages (job list, live progress, per-page markdown viewer, JSON download, Extract dialog).
- `scripts/task-tokens.mjs`: whole-task benchmark with a real BPE tokenizer (gpt-tokenizer, dev dependency); results in docs/COMPARISON.md — 22.7× vs HTML, 4.0× vs full tree over 18 observations.
- Strict query booleans (`?delta=false` was treated as true before).

## 0.5.0 — 2026-10-12

### Added
- **Plans & billing**: Free / Pro / Team (override with `VESSEL_PLANS`), monthly browser-minute and concurrency limits enforced per workspace (`402 plan_minutes_exhausted` with an upgrade link), `GET /plans`, `GET /v1/billing`, admin `POST /v1/admin/orgs/:id/plan` and `GET /v1/admin/orgs`. **Stripe** via Payment Links (`VESSEL_STRIPE_LINKS`) + signed webhook `POST /billing/stripe/webhook` (`VESSEL_STRIPE_WEBHOOK_SECRET`): checkout upgrades the org, cancellation downgrades. Dashboard **Billing** page, plan badge and usage meter in the sidebar.
- **Off-site backups**: `VESSEL_BACKUP_URL=s3://KEY:SECRET@host/bucket/prefix` (R2/S3/B2/MinIO…), hourly `VACUUM INTO` → gzip → AES-256-GCM (`VESSEL_BACKUP_KEY`) → upload, retention `VESSEL_BACKUP_KEEP`; `GET /v1/admin/backups`, `POST /v1/admin/backup`, `POST /v1/admin/restore` (staged, applied at next start). Dependency-free SigV4 client.
- **Stealth** (`stealth: true` by default): webdriver flag, chrome.*, plugins, languages, permissions, WebGL vendor, hardware hints, outer size — closes the deterministic headless tells (not a CAPTCHA solver).
- **Playground** tab on the session page: go to URL, click/fill by ref, or run raw action JSON; results and the next snapshot inline.
- Bot-wall detection now samples visible text when the interactive tree is tiny (catches "Access is temporarily restricted"-style walls such as G2's).
- Bare domains accepted everywhere a URL is (`example.com` → `https://example.com`); validation errors name the field.
- Copy-session-id buttons, one-click 60-second demo on Get started, dashboard assets served `no-cache`.


## 0.4.0 — 2026-10-12

### Added
- **Cloud mode** (`VESSEL_SIGNUP=true`): sign-up/sign-in in the dashboard (`/auth/*`, scrypt passwords, signed cookies), organisations with a concurrent-session quota (`VESSEL_ORG_MAX_SESSIONS`), **API keys minted in the dashboard** (`/v1/keys`, shown once, SHA-256 at rest). Every session, profile, routine, secret and approval is tagged with its organisation and invisible to others — including over MCP, the tools API, SSE, live view and CDP. Env keys remain admins.
- **Server-side scripts**: `POST /v1/sessions/:id/script { code }` runs a Playwright snippet on the node with `page`/`context` in scope (logs, return value, timeout, recorded as a step). Gated by `VESSEL_ALLOW_SCRIPTS` (default off when sign-up is on).
- Docs: `docs/HOW-IT-WORKS.md` (owner walkthrough: inputs, flows, how to test end to end) and `docs/HOSTING.md` (which platforms can run CrawlGenie and why, with sources).
- Dashboard: auth card, **API keys** page, workspace + sign-out in the sidebar.


## 0.3.0 — 2026-10-11

### Added
- **Secrets vault**: `PUT/GET/DELETE /v1/secrets/:name`; agents write `{{secret:name}}` in `fill`/`type` (and `browser_type`), the server substitutes the value at execution. Key- and domain-scoped, AES-256-GCM at rest with `VESSEL_SECRET_KEY`. New MCP/HTTP tool `browser_list_secrets`.
- **Challenge detection**: `snapshot.challenge` (`cloudflare|captcha|access_denied|bot_wall` + hint), `challenge` recorder event, `session.challenge` webhook, ⚠ prefix on tool results.
- **One-shot endpoints**: `POST /v1/scrape` (markdown/text/html/snapshot/links, optional screenshot+pdf), `GET /v1/screenshot`, `GET /v1/pdf`.
- **File uploads**: `POST /v1/sessions/:id/files` (raw bytes, ≤64 MB) and `upload` actions by `{ id }`.
- **Extensions** on the isolated tier (`extensions: [...]` from `VESSEL_EXTENSIONS_DIR`).
- **Helm chart** (`deploy/helm/vessel`): StatefulSet nodes with PVCs, optional router Deployment and Ingress.
- `scripts/challenge.mjs` — honest "does it open?" sample across popular sites; 10-site token benchmark in docs.
- `/v1/stats` now reports `version` and `mode`.
- SDKs: `secrets`, `scrape`, `uploadFile`/`files` (TS + Python).
- Dashboard **Secrets** page (add / rotate / delete, domain scoping, copy placeholder) and a challenge banner on the session Snapshot panel.
- GitHub Pages landing site live at https://theshariq.github.io/CrawlGenie/.

### Changed
- **Dashboard design system v3**: ink-black primary actions, single cobalt accent reserved for meaning, cool neutral canvas, grouped sidebar with node status, page headers, Stripe-style stat strip and tables with hover actions, SVG area chart for tokens saved, browser-window frame around the live view, dark code blocks with copy, segmented controls and switches, redesigned new-session sheet, Get started and API & tools pages.

## 0.2.0 — 2026-10-11

### Added
- **Delta snapshots** (`snapshotOptions.delta`, session `snapshotDefaults`): only lines changed since the last observation of the same page; falls back to the full tree on large changes or ref renumbering.
- **Observation-token metering** per session: `observationTokens`, `observationTokensRaw`, `tokenSavingsPct`, `estimatedSavingsUsd` (`VESSEL_TOKEN_PRICE_PER_M`), shown on the dashboard.
- **Tools API** `GET /v1/tools?format=openai|anthropic|gemini|json` and `POST /v1/tools/:name` — the MCP tool surface for any function-calling model, no MCP client needed.
- **MCP 2026-07-28** via the v2 TypeScript SDK (per-request Streamable HTTP; 2025-era clients served statelessly). Tool results are stamped with `session_id`.
- **Proxy pools** (`VESSEL_PROXY_POOLS` / `data/proxies.json`): static rotation, template providers with `{session}`/`{country}` placeholders, sticky identities, failure cooldown, `GET /v1/proxies` health; per-session `proxy: { pool, country, sticky }`.
- **Webhooks** (`VESSEL_WEBHOOK_URL`): `approval.pending`, `approval.resolved`, `session.closed`.
- **Retention** (`VESSEL_RETENTION_DAYS`, `POST /v1/admin/prune`).
- **`assert` action** (text/notText/selector/url/title) so routines double as monitors.
- **OpenAPI 3.1** at `/openapi.json`, generated from the request schemas.
- **Python SDK** (`packages/python`, dependency-free) and TS SDK updates (`routines`, `proxies`, token usage).
- **Dashboard** rebuilt as a single-page app: overview KPIs (tokens/$ saved), unified session page (live + time-travel scrubber + snapshot/timeline/approvals/usage), routines with run/param forms and save-as-routine, approvals inbox, profiles, API & tools, settings; light theme by default, command palette, keyboard shortcuts, mobile layout.
- **Security**: CORS allowlist (`VESSEL_CORS_ORIGINS`), security headers, per-IP failed-auth rate limit (`VESSEL_AUTH_RATE_LIMIT`).
- **CI**: Linux typecheck/tests/bench, Docker build + non-root smoke test, GHCR publish on green main. Soak (`scripts/soak.mjs`) and token (`scripts/tokens.mjs`) benchmarks.

### Changed
- `approvals.mode: 'all'` gates only state-changing actions (not wait/scroll/hover/tab list).
- Snapshots bypass the action queue while an approval is pending, so reviewers can see the page.
- Interactive snapshot filter drops table/list wrappers, nameless clickables and separators; `[cursor=pointer]` only kept on generic nodes.
- Docker image now builds on `node:24-bookworm-slim` (node:sqlite requires ≥ 22.13).
- zod 4.

## 0.1.0 — 2026-10-11
Initial release: pooled/isolated sessions, interactive accessibility snapshots with refs, policy + SSRF guard + approval gates, flight recorder with fork/rewind, profiles with exclusive locks, routines with self-healing replay, live view with takeover, MCP, raw CDP relay, dashboard, TS SDK, Docker, docs, 30 tests.