# CrawlGenie

> The web stack for AI agents that cuts LLM token costs: browser, search, documents with OCR, 61 agents, web agents, memory, connectors for ChatGPT and Claude. Everything measured in real tokens.

Base URL: https://crawlgenie.com. Agents: read https://crawlgenie.com/skill.md first. API: https://crawlgenie.com/openapi.json

## Start here

- [Quickstart](https://crawlgenie.com/docs/quickstart.md): Run CrawlGenie and do something useful in ten minutes.
- [Concepts: what to use for what](https://crawlgenie.com/docs/concepts.md): Session, crawl, extractor, watch, routine, schedule, knowledge — the one-page map.
- [Use cases by problem and industry](https://crawlgenie.com/docs/use-cases.md): What people actually need from the web and which CrawlGenie call solves it.

## Read the web for less

- [Documents, files and media](https://crawlgenie.com/docs/documents.md): PDF (text and OCR), Office files, Google Docs/Sheets/Slides, images, YouTube transcripts, page ranges, tables.
- [Knowledge base (retrieval without a model)](https://crawlgenie.com/docs/knowledge.md): Store pages, documents and crawl output; ask questions across them for a few hundred tokens.
- [Connect ChatGPT, Claude, agents](https://crawlgenie.com/docs/integrations.md): Connectors (OAuth), MCP, function calling, SDKs, Chrome extension, where the savings show.

## Build

- [HTTP API reference](https://crawlgenie.com/docs/api.md): Every endpoint with request and response shapes.
- [Extractor library](https://crawlgenie.com/docs/extractors.md): Every ready-made extractor and monitor, by vertical, with inputs — generated from this server.
- [How it works (for the owner)](https://crawlgenie.com/docs/how-it-works.md): Modes, a session end to end, routines, profiles, secrets, how to test.
- [Architecture](https://crawlgenie.com/docs/architecture.md): Tiers, the agent interface, policy, recorder, data.

## Run

- [Deploy and sizing](https://crawlgenie.com/docs/deploy.md): Docker, Compose, Kubernetes, env vars, scaling, backups.
- [Where to run it (free options)](https://crawlgenie.com/docs/hosting.md): VM vs functions, free tiers verified Oct 2026, GitHub Actions, Cloud Run.
- [Security model](https://crawlgenie.com/docs/security.md): Threat model, controls, known limits, hardening checklist.
- [Signed agent identity (Web Bot Auth)](https://crawlgenie.com/docs/agent-identity.md): Let sites verify your agent cryptographically instead of challenging it.
- [Troubleshooting](https://crawlgenie.com/docs/troubleshooting.md): Blocked pages, empty extractors, OCR, OAuth, timeouts — symptom → cause → fix.
- [FAQ](https://crawlgenie.com/docs/faq.md): Short answers to the questions every evaluator asks.

## About

- [Compared with Firecrawl, Browserbase, Apify, Steel](https://crawlgenie.com/docs/comparison.md): Measured numbers, gap analysis, where CrawlGenie is weaker.
- [Strategy and economics](https://crawlgenie.com/docs/strategy.md): Why this layer, why it lasts, what it costs.
- [Changelog](https://crawlgenie.com/docs/changelog.md): What changed in each release.