Strategy: why CrawlGenie, why it lasts, and what it honestly costs
Why this layer, why it lasts, what it costs.
1. The layer we own#
Every AI agent that touches the web needs three things Playwright does not provide: a cheap way for a model to see, guardrails the model cannot bypass, and operational memory (logins, flows, recordings). CrawlGenie is that layer. The model is a commodity that changes every quarter; the browser engine is a commodity (Chromium); the layer between them is where the operational knowledge of a business accumulates, and that is what people pay for.
2. Why it will not become redundant#
We bet on standards and on things that compound, not on a model or a vendor:
| Bet | Why it survives model/vendor churn |
|---|---|
| Accessibility tree (ARIA) as the observation format | A W3C standard every browser maintains; works for text models today, remains the cheapest grounding for vision models tomorrow. Refs make actions deterministic regardless of model. |
| CDP / WebDriver BiDi for control | Browser-vendor standards; Playwright is our current adapter, not our dependency. The Session interface hides the engine, so Firefox/WebKit via BiDi, or a lightweight non-Chromium engine for read-only work, can slot in. |
| MCP + plain JSON-Schema tools | Whatever protocol an agent framework speaks, the tool surface is defined once (tools/registry.ts) and emitted in each shape. New protocol = new emitter, not a rewrite. |
| Model-agnostic by design | CrawlGenie never calls an LLM. A better model makes routines easier to teach and approvals rarer; it never makes governance, recording or profiles less necessary. |
| Governance demand rises with autonomy | The more agents act unattended, the more required "who approved what, with what evidence" becomes (EU AI Act human-oversight and logging duties; SOC 2 change control). Approval gates + flight recorder map to that directly. |
| Customer data compounds | Routines, profiles, recordings and policies are the customer's operational memory. They get more valuable with use and are painful to leave. |
| OCI container deployment | Runs anywhere for the next decade; no proprietary runtime. |
Threats we watch: browsers shipping native agent APIs (WebMCP) → CrawlGenie should expose a site's declared tools through the same tool surface (roadmap); OS-level computer-use agents → the web remains the universal interface for business software, and the governance layer applies to any action surface.
3. Token economics (measured, scripts/tokens.mjs)#
| Observation style | Avg tokens / step (6 sites) | vs CrawlGenie |
|---|---|---|
| Raw HTML | 57,528 | 15.7× more |
| Full accessibility tree (Playwright MCP default) | 14,168 | 3.9× more |
| Screenshot (vision) | ~1,500 + extra steps for coordinates | comparable per step, more steps |
| CrawlGenie interactive | 3,670 | — |
| CrawlGenie delta (same page, small change) | 80–300 | 12–45× less than interactive |
| CrawlGenie routine replay | 0 | no model call |
Claims we make: "4–16× fewer observation tokens than DOM/a11y agents", "delta steps cut repeat observations by 60–95%", "routines cost zero tokens per step", and every session shows its own tokenSavingsPct and estimatedSavingsUsd. We do not claim a universal "10×": it depends on the site and the agent's loop, which is why the meter is in the product.
4. Cost: what "$10/month" really means#
The VM is $7–10/month (Hetzner CX23/CX33). Total cost of ownership for a small team is more like $10–40/month: add residential proxy bandwidth if you need it ($1–8/GB on the open market, bought direct), recording storage, and an hour or two of operator time. Density depends on workload: 25–35 concurrent shared sessions on 4 vCPU/8 GB holds for agent-paced browsing; sustained JS-heavy pages drop that to 8–12. The hosted alternatives land at $100–300/month for the same usage, so the honest multiple is 5–10×, not 30×. A future hosted CrawlGenie tier would need ~$0.02–0.04/browser-hour to cover operations and margin, still at or below the cheapest incumbent.
5. Adoption without a sales network#
- Zero-friction start:
docker runornpx, dashboard in 60 seconds, MCP in one command. - The number: every session shows tokens and dollars saved; screenshots of that are the marketing.
- Distribution surfaces that need no network: GitHub, Docker Hub, npm, PyPI, MCP directories (Claude, Cursor, Smithery, PulseMCP), Show HN, the Playwright/MCP communities.
- Framework reach: MCP + OpenAI/Anthropic/Gemini tool schemas + TS/Python SDKs means every tutorial can include CrawlGenie in five lines.
- Honesty as positioning: published benchmarks with code, limits stated up front (no anti-bot promises), commercial licence. The category's top complaints are about vendors hiding exactly these things.
6. Definition of "best" we hold ourselves to#
- Fastest honest observe→act loop per token (measured, published).
- The only open platform with fork/rewind, approval gates and self-healing routines.
- Secure by default: auth on, SSRF guard on, non-root container, approvals available.
- Works with any model, any framework, any language, on any Linux box.
- No feature exists without a test; no claim exists without a script that reproduces it.
Section: About · This page as markdown · Questions? hello@crawlgenie.com