Sign inStart free
Home/Docs/Changelog

Changelog

What changed in each release.

0.11.0 — 2026-10-13#

  • Website: menus open on hover; one "why CrawlGenie exists" comparison (one platform instead of Browserbase + Firecrawl + TinyFish + Apify) with the detailed table folded underneath; chat bubble on every page (Ask CrawlGenie from the docs, or Message us → POST /site/contact, stored and e-mailed to VESSEL_CONTACT_TO); support address hello@crawlgenie.com in footer, legal pages and Organization schema; favicon, app icons, web manifest and social card from the new Spark G mark; meta descriptions within 160 characters.
  • Dashboard: Token savings in the sidebar under Overview; the "+ Custom agent" button on Agents opens the create dialog again; Overview sessions table scrolls instead of cropping and shows Replay/Open without hovering; help-bot excerpts no longer split sentences at e-mail addresses.
  • Public deployments: /docs and /docs/<page> are server-rendered HTML for visitors and crawlers (same markdown as the dashboard); app links opened while signed out redirect to /login?next=… and return after sign-in; unknown routes return a real 404 page; sign-out goes to the sign-in page; images and icons cache for a day.
  • Plan ladder: monthly quotas for reads, searches and agent runs (REST and MCP), paid-only alerts/webhooks, scheduled routines, isolated tier, proxy pools, per-plan recording retention; Free resized to 120 browser-minutes with every product included.
  • Plans: per-plan capacity (active watches, team members, memory documents, batch size) enforced with 402 plan_limit + upgrade link; limits live in public/pricing.json.
  • Dashboard: Get started checklist (separate from the onboarding wizard), behaviour-based recommendations, month strip on the Overview, session Pages/Console/Files tabs and expiry, tested setup prompt for coding agents, /v1/insights.
  • Sign-in page: standalone /start, /signup and /login for public deployments (value slides, Google button), dashboard stays at /app.
  • "Continue with Google" (OpenID Connect) with account linking by verified e-mail; VESSEL_GOOGLE_CLIENT_ID/SECRET.
  • Vault: {{totp:NAME}} one-time codes (RFC 6238) for 2-factor logins.
  • VESSEL_PUBLIC_HOME: marketing site at /; installer sets it with sign-up, public tools, log rotation and swap.

Added#

  • Web agents: schema for structured results; POST /v1/agent/batch (one goal across many URLs or many goals) with a concurrency cap and GET /v1/agent/batches/:id for counts, results and usage.
  • Continuous reports for every watch: /v1/watches/:id/series (metrics per run: counts, new rows, scores, column means) and /v1/watches/:id/report (7-day markdown or JSON: latest run, trend, alerts, new rows); trend sparklines on the watch page.
  • SEO + AIO report agent (seo_aio_report), free public tools pages, in-app docs with the retrieval-only assistant.
  • Marketing site v2 on the "Graphite & Lime" theme (five theme directions in /brand/themes.html), repositioned on LLM token savings; web agents solution page; Enterprise private-deployment plan.

Changed#

  • The product is no longer presented as open source; package licenses set to UNLICENSED and private: true. Code published before this commit remains under its original Apache-2.0 terms wherever it was copied.

0.10.0 — 2026-10-12#

Added#

  • In-house documentation and help: every server serves its docs at /docs (search, table of contents, raw .md, /llms.txt, /llms-full.txt) and an Ask CrawlGenie assistant that answers from the docs with retrieval only (POST /help/ask, public, no model). New pages: Concepts, Use cases by industry, Documents & media, Knowledge base, Troubleshooting, Signed agent identity; the Extractor library page is generated from the running catalogue. Dashboard links no longer point at GitHub.
  • Knowledge base (/v1/knowledge, MCP knowledge_add / knowledge_search): collections of pages, documents, crawl output, watch rows and notes, chunked and indexed with SQLite FTS5 (BM25 + porter); questions return passages with sources and a token receipt. Dashboard page.
  • Signed agent identity (Web Bot Auth): Ed25519 key per deployment, /.well-known/http-message-signatures-directory, GET /v1/identity, policy.signedAgent / workspace default; navigations carry RFC 9421 signatures (tag web-bot-auth) so sites can verify the agent.
  • Documents: PDF pages ranges and table-to-columns reconstruction; ODT/ODS/ODP and RTF parsing; Google Docs/Sheets/Slides, Drive, Dropbox and GitHub blob links normalised to export/raw URLs in /v1/parse, /v1/scrape, web_answer, web_research.
  • Reading less: granularity: "sentence" (highlights: matching sentences, table rows, code lines), redactPII, cache: prefer|only on /v1/scrape; POST /v1/answer {urls, q} and web_research {urls} for one question across several sources.
  • Extractors: auto_list (repeated blocks without CSS), pricing_page, github_issues, search_watch (web-scale monitor: new results for a query), brand_assets, llms_txt, page_media (every file/video/image on a page with the next step); css_extract gained scroll.
  • SDKs: knowledge.*, help.*, answer(), identity(), parse options (TypeScript); knowledge_*, answer, help_ask, identity (Python).
  • Docker image ships docs/ and CHANGELOG.md.

Fixed#

  • Page-range PDF parses no longer trigger OCR because of the per-page word heuristic; pdf.js synthetic space runs no longer defeat column detection.

0.9.0 — 2026-10-12#

Added#

  • Savings by surface: every session is tagged with the channel that created it (REST API, MCP, tools API, dashboard, built-in agents, or the OAuth connector that holds the key, e.g. connector:ChatGPT); GET /v1/stats/savings returns byChannel and the Token savings page shows a per-surface table.
  • Image OCR (PNG/JPEG/WEBP/GIF/BMP via /v1/parse), YouTube transcripts (youtube_transcript extractor and web_answer on a video URL: caption track → timestamped paragraphs, no speech model), Playground drop zone for local files.
  • Security: cross-site consent submissions are refused, OAuth endpoints are rate-limited per IP, the OCR document handoff is one-time and same-origin only, the Chrome extension no longer requests host permissions. scripts/ab-connector.sh compares cost and answers of Claude Code's built-in web tools vs the CrawlGenie connector on the same tasks.
  • OAuth 2.1 for MCP connectors: CrawlGenie is an authorization server (/.well-known/oauth-authorization-server, /.well-known/oauth-protected-resource, dynamic client registration, PKCE, consent page). ChatGPT, claude.ai, Claude Desktop, Cursor and Gemini CLI add CrawlGenie as a custom connector; tokens are revocable workspace keys. Verified end to end with the official MCP client over a public HTTPS tunnel and with Claude Code (claude mcp add).
  • Cheapest-read tools: web_answer (URL or document + question → only the matching passages) and web_research (search → read → focus, one call) with a token receipt on every result; maxTokens budgets on snapshot / text / scrape / browser_snapshot (exact BPE truncation).
  • Documents: OCR for scanned PDFs (pages rendered by a CrawlGenie browser with pdf.js, read by tesseract.js; auto when the text layer is empty, ocr: force|off), XLSX (sheets → markdown tables) and PPTX (slides) parsing; all routed automatically from /v1/scrape and web_answer.
  • Scrape gained actions (click/scroll/type before extracting) and changeTracking (new / same / changed + diff per owner and URL, so agents skip unchanged pages). Crawl gained /answer (passages across all pages with sources) and /export?format=jsonl|md|csv|pdf.
  • Outcome-first UI: five-step onboarding (role → goal → connect your agent → try three runs → map), Overview "Start from a goal" + vocabulary with token cost per concept + zero-token-runs KPI, a "Do something with this page" panel on every session (ask, read, extract tables, links, download, instruct an agent, keep), next-step actions on crawl and watch pages, Playground tabs for monitor and agent plus Get code (curl / Python / TypeScript) on every form.
  • Homepage ROI calculator, Chrome extension scaffold (integrations/chrome-extension), docs/INTEGRATIONS.md, scripts/mcp-oauth-client.mjs, SDK tools.definitions/call/executable, Python tools()/call_tool().

0.8.0 — 2026-10-12#

Added#

  • Extractor store: 59 built-in agents across 14 verticals (54 extractors + 5 monitors) (jobs, e-commerce, local & leads, reviews, social, news, SEO, finance, real estate, travel, research, government/legal/health, events, any site) — a declarative recipe engine (html / json / feed sources) plus hand-written extractors (schema.org structured data, website contacts, tech-stack fingerprint, YouTube search/channel, SEC EDGAR, PubMed, sitemap monitor, HN hiring, JSON API watcher). GET /v1/extractors (grouped by vertical), POST /v1/extractors/{id}/run (sync rows, ?format=csv), MCP/tools vessel_extract, GET /v1/watches/{id}/dataset (rows merged across runs with first/last seen). Live-tested from a consumer IP without a proxy: 50 of 54 extractors return rows; Yelp and eBay are labelled "needs proxy" and say so at run time.
  • Question-focused reading: q= on /v1/sessions/{id}/text, /v1/scrape, /v1/parse and the browser_extract_text tool returns only the passages that answer the question (BM25-style, no model) and reports the measured token saving.
  • Document parsing without a browser: POST /v1/parse for PDF (pdf.js text layer), DOCX (own zip + WordprocessingML reader: headings, lists, tables → markdown) and text/CSV/JSON; /v1/scrape routes document URLs there automatically.
  • Search + read: GET /v1/search?content=true attaches markdown of the top hits in one call.
  • Agent onboarding: GET /skill.md (also /agent-onboarding/SKILL.md), dashboard Setup for agents page with copy-ready MCP / SDK / CLI snippets, and a client CLI (npx vessel-browser-sdk vessel-cli …).
  • Settings → General (server-side, admin): workspace name, default and idle session timeouts (apply to new sessions), default blocked resources, webhook URL; API key reveal / copy / regenerate; GET|PUT /v1/settings.
  • Self-hosted admins mint revocable API keys from the dashboard (implicit Workspace org); accent picker (cobalt, coral, amber, sapphire, rose); POST /v1/tick so scale-to-zero hosts or an external cron can run due watches/schedules; deploy/github-actions/vessel-extract.yml runs extractors on GitHub Actions for free; HOSTING.md free-tier table.

Changed#

  • VESSEL_SSRF_GUARD=false is now honoured by direct fetches (JSON/feed/document sources), not only by navigation.
  • Watch auto-names use any subject field (query, domain, subreddit…), not only URLs.

0.7.0 — 2026-10-12#

Added#

  • Extractors (Apify-style actors, no LLM): css_extract (any page → rows via CSS, pagination), web_search_results (rank tracking), google_maps_places, trustpilot_reviews, indeed_jobs, amazon_products (price-change alerts), reddit_posts (feeds + search, no login), linkedin_company (public card), producthunt_today, g2_reviews (needs a residential pool — reports the wall honestly). Rows are deduplicated across runs ("12 new rows"), exported with GET /v1/watches/:id/runs/:runId/export?format=csv|json&only=all|new. Dashboard: Agents → Extractors, generic table with "New only" + CSV/JSON; watch sheet gets proxy-pool / login-profile options. Playbook catalogue now carries category.
  • Web search falls back DuckDuckGo → Brave → Bing (Brave serves full snippets to a headless browser without a key).

Changed#

  • Interception-free policy enforcement. Playwright request interception turned out to be a bot signal on its own (Reddit: CAPTCHA with a pass-through route('**/*'), normal page without it). Trackers, blocked domains, image/font/media/CSS blocks and literal private hosts are now enforced through Chromium's URL blocklist via CDP; interception is used only for allowlists, byte budgets, script/XHR blocks or the new policy.strictNetwork: true. Blocked-request metering unchanged (usage.requestsBlocked).
  • Extractors retry navigation once on connection resets and survive hydration redirects (Trustpilot, LinkedIn).

0.6.0 — 2026-10-12#

Added#

  • Crawl / map / batch-scrape / extract (Firecrawl-shaped): POST /v1/crawl (site crawl with limits, include/exclude globs, robots.txt, N browser workers, job status + paginated pages), POST /v1/batch-scrape, POST /v1/map (sitemaps + links), POST /v1/extract (prompt + JSON Schema → structured data via the caller's own LLM key; OpenAI-compatible, Anthropic, Gemini), GET /v1/llm. Markdown reader upgraded with readability-style main-content selection, boilerplate removal and page metadata.
  • Dependency-free LLM client (src/llm/client.ts) shared by extract and the upcoming agent runner.
  • Team invites by link (/v1/invites, /auth/join, members list) — no email provider required; dashboard "API keys & team".
  • Web search through CrawlGenie's own browser (GET /v1/search, tool browser_search), DuckDuckGo with Bing fallback, no API key.
  • Built-in agents (no API key): change monitor, new-posts digest, price & stock tracker, SEO audit, broken-link checker — stateful watches with diffs, cron schedules and per-watch webhooks (/v1/playbooks, /v1/watches); dashboard Agents gallery + watch pages with rendered diffs/reports.
  • Exact token metering: every observation counted with the o200k tokenizer (read / full tree / HTML); /v1/stats/savings, POST /v1/measure; dashboard Token savings page with price-model selector and a live "Measure any URL" check.
  • Agent: POST /v1/agent/run — natural-language tasks executed by a built-in observe→act loop with the caller's own model key (OpenAI-compatible/Anthropic/Gemini); per-step thoughts, usage and observation tokens; dashboard Agent page with live steps and result.
  • Auto-deploy: deploy/install.sh now adds watchtower — every green push to main is live on the VM within ~5 minutes (docs/DEPLOY.md “How changes go live”).
  • Schedules: cron-triggered routine runs with timezone support, per-schedule webhooks, run history and a preview endpoint; dashboard Schedules page with presets.
  • Dashboard Crawls & extract pages (job list, live progress, per-page markdown viewer, JSON download, Extract dialog).
  • scripts/task-tokens.mjs: whole-task benchmark with a real BPE tokenizer (gpt-tokenizer, dev dependency); results in docs/COMPARISON.md — 22.7× vs HTML, 4.0× vs full tree over 18 observations.
  • Strict query booleans (?delta=false was treated as true before).

0.5.0 — 2026-10-12#

Added#

  • Plans & billing: Free / Pro / Team (override with VESSEL_PLANS), monthly browser-minute and concurrency limits enforced per workspace (402 plan_minutes_exhausted with an upgrade link), GET /plans, GET /v1/billing, admin POST /v1/admin/orgs/:id/plan and GET /v1/admin/orgs. Stripe via Payment Links (VESSEL_STRIPE_LINKS) + signed webhook POST /billing/stripe/webhook (VESSEL_STRIPE_WEBHOOK_SECRET): checkout upgrades the org, cancellation downgrades. Dashboard Billing page, plan badge and usage meter in the sidebar.
  • Off-site backups: VESSEL_BACKUP_URL=s3://KEY:SECRET@host/bucket/prefix (R2/S3/B2/MinIO…), hourly VACUUM INTO → gzip → AES-256-GCM (VESSEL_BACKUP_KEY) → upload, retention VESSEL_BACKUP_KEEP; GET /v1/admin/backups, POST /v1/admin/backup, POST /v1/admin/restore (staged, applied at next start). Dependency-free SigV4 client.
  • Stealth (stealth: true by default): webdriver flag, chrome.*, plugins, languages, permissions, WebGL vendor, hardware hints, outer size — closes the deterministic headless tells (not a CAPTCHA solver).
  • Playground tab on the session page: go to URL, click/fill by ref, or run raw action JSON; results and the next snapshot inline.
  • Bot-wall detection now samples visible text when the interactive tree is tiny (catches "Access is temporarily restricted"-style walls such as G2's).
  • Bare domains accepted everywhere a URL is (example.com → https://example.com); validation errors name the field.
  • Copy-session-id buttons, one-click 60-second demo on Get started, dashboard assets served no-cache.

0.4.0 — 2026-10-12#

Added#

  • Cloud mode (VESSEL_SIGNUP=true): sign-up/sign-in in the dashboard (/auth/*, scrypt passwords, signed cookies), organisations with a concurrent-session quota (VESSEL_ORG_MAX_SESSIONS), API keys minted in the dashboard (/v1/keys, shown once, SHA-256 at rest). Every session, profile, routine, secret and approval is tagged with its organisation and invisible to others — including over MCP, the tools API, SSE, live view and CDP. Env keys remain admins.
  • Server-side scripts: POST /v1/sessions/:id/script { code } runs a Playwright snippet on the node with page/context in scope (logs, return value, timeout, recorded as a step). Gated by VESSEL_ALLOW_SCRIPTS (default off when sign-up is on).
  • Docs: docs/HOW-IT-WORKS.md (owner walkthrough: inputs, flows, how to test end to end) and docs/HOSTING.md (which platforms can run CrawlGenie and why, with sources).
  • Dashboard: auth card, API keys page, workspace + sign-out in the sidebar.

0.3.0 — 2026-10-11#

Added#

  • Secrets vault: PUT/GET/DELETE /v1/secrets/:name; agents write {{secret:name}} in fill/type (and browser_type), the server substitutes the value at execution. Key- and domain-scoped, AES-256-GCM at rest with VESSEL_SECRET_KEY. New MCP/HTTP tool browser_list_secrets.
  • Challenge detection: snapshot.challenge (cloudflare|captcha|access_denied|bot_wall + hint), challenge recorder event, session.challenge webhook, ⚠ prefix on tool results.
  • One-shot endpoints: POST /v1/scrape (markdown/text/html/snapshot/links, optional screenshot+pdf), GET /v1/screenshot, GET /v1/pdf.
  • File uploads: POST /v1/sessions/:id/files (raw bytes, ≤64 MB) and upload actions by { id }.
  • Extensions on the isolated tier (extensions: [...] from VESSEL_EXTENSIONS_DIR).
  • Helm chart (deploy/helm/vessel): StatefulSet nodes with PVCs, optional router Deployment and Ingress.
  • scripts/challenge.mjs — honest "does it open?" sample across popular sites; 10-site token benchmark in docs.
  • /v1/stats now reports version and mode.
  • SDKs: secrets, scrape, uploadFile/files (TS + Python).
  • Dashboard Secrets page (add / rotate / delete, domain scoping, copy placeholder) and a challenge banner on the session Snapshot panel.
  • GitHub Pages landing site live at https://theshariq.github.io/CrawlGenie/.

Changed#

  • Dashboard design system v3: ink-black primary actions, single cobalt accent reserved for meaning, cool neutral canvas, grouped sidebar with node status, page headers, Stripe-style stat strip and tables with hover actions, SVG area chart for tokens saved, browser-window frame around the live view, dark code blocks with copy, segmented controls and switches, redesigned new-session sheet, Get started and API & tools pages.

0.2.0 — 2026-10-11#

Added#

  • Delta snapshots (snapshotOptions.delta, session snapshotDefaults): only lines changed since the last observation of the same page; falls back to the full tree on large changes or ref renumbering.
  • Observation-token metering per session: observationTokens, observationTokensRaw, tokenSavingsPct, estimatedSavingsUsd (VESSEL_TOKEN_PRICE_PER_M), shown on the dashboard.
  • Tools API GET /v1/tools?format=openai|anthropic|gemini|json and POST /v1/tools/:name — the MCP tool surface for any function-calling model, no MCP client needed.
  • MCP 2026-07-28 via the v2 TypeScript SDK (per-request Streamable HTTP; 2025-era clients served statelessly). Tool results are stamped with session_id.
  • Proxy pools (VESSEL_PROXY_POOLS / data/proxies.json): static rotation, template providers with {session}/{country} placeholders, sticky identities, failure cooldown, GET /v1/proxies health; per-session proxy: { pool, country, sticky }.
  • Webhooks (VESSEL_WEBHOOK_URL): approval.pending, approval.resolved, session.closed.
  • Retention (VESSEL_RETENTION_DAYS, POST /v1/admin/prune).
  • assert action (text/notText/selector/url/title) so routines double as monitors.
  • OpenAPI 3.1 at /openapi.json, generated from the request schemas.
  • Python SDK (packages/python, dependency-free) and TS SDK updates (routines, proxies, token usage).
  • Dashboard rebuilt as a single-page app: overview KPIs (tokens/$ saved), unified session page (live + time-travel scrubber + snapshot/timeline/approvals/usage), routines with run/param forms and save-as-routine, approvals inbox, profiles, API & tools, settings; light theme by default, command palette, keyboard shortcuts, mobile layout.
  • Security: CORS allowlist (VESSEL_CORS_ORIGINS), security headers, per-IP failed-auth rate limit (VESSEL_AUTH_RATE_LIMIT).
  • CI: Linux typecheck/tests/bench, Docker build + non-root smoke test, GHCR publish on green main. Soak (scripts/soak.mjs) and token (scripts/tokens.mjs) benchmarks.

Changed#

  • approvals.mode: 'all' gates only state-changing actions (not wait/scroll/hover/tab list).
  • Snapshots bypass the action queue while an approval is pending, so reviewers can see the page.
  • Interactive snapshot filter drops table/list wrappers, nameless clickables and separators; [cursor=pointer] only kept on generic nodes.
  • Docker image now builds on node:24-bookworm-slim (node:sqlite requires ≥ 22.13).
  • zod 4.

0.1.0 — 2026-10-11#

Initial release: pooled/isolated sessions, interactive accessibility snapshots with refs, policy + SSRF guard + approval gates, flight recorder with fork/rewind, profiles with exclusive locks, routines with self-healing replay, live view with takeover, MCP, raw CDP relay, dashboard, TS SDK, Docker, docs, 30 tests.

Section: About · This page as markdown · Questions? hello@crawlgenie.com