Web scraping for LLMs that sends the model only what matters
CrawlGenie is LLM web scraping that reads less: it scrapes pages into clean markdown and, when you ask a question, returns just the passages or sentences that answer it, with a receipt of tokens kept versus avoided. No model is used to read, so it is deterministic, fast and free to run.
What is LLM web scraping and why is it expensive?
LLM web scraping means fetching web content so a language model can reason over it. The cost is the tokens the model reads: a typical article is 20–30 thousand tokens as HTML, 5–8 thousand as clean markdown, and a few hundred as the passages that answer one question. Agents multiply this on every step, every retry and every day. CrawlGenie attacks all three.
How does question-focused scraping work?
POST /v1/scrape with a url and a q. The page is rendered in a real browser, boilerplate is removed, the text is split into passages, and a BM25-style ranker with phrase and heading bonuses keeps the passages that match. granularity: "sentence" returns only matching sentences, table rows and code lines. The response carries focus.tokensFull, focus.tokensFocused and the percentage saved, counted with the real tokenizer.
For developers: the exact call
POST /v1/scrape
{"url": "https://acme.com/terms", "q": "refund policy", "granularity": "sentence", "maxTokens": 800, "redactPII": true}
→ content: "- Customers may request a full refund within 14 days…"
focus: { tokensFull: 27611, tokensFocused: 412, tokenSavingsPct: 98.5 } pii: { redacted: 2 }What about the same pages every day?
Use cache: "prefer" to serve your own last copy with zero outbound requests, changeTracking to get new / same / changed with a line diff, or turn the page into a watch so it runs on a schedule and alerts only when something changed. Rows from the 61 extractors (jobs, prices, reviews, filings, papers, places, posts) cost zero model tokens because no model reads the page at all.
How does this compare with Firecrawl, Jina Reader or Crawl4AI?
Jina Reader and Crawl4AI give you markdown; Firecrawl adds model-written answers and hosted indexes. CrawlGenie keeps the reading deterministic (you can audit which passages were kept and why), adds exact receipts, documents with OCR, a knowledge base, browser sessions with approvals, and signed agent identity, all behind one key with no model gateway. See the Firecrawl comparison.
How do agents call it?
From ChatGPT or Claude: the web_answer tool through the connector. From Claude Code, Cursor or Codex: the MCP server. From code: the REST API, the TypeScript or Python SDK, or the tool schemas for the OpenAI, Anthropic and Gemini SDKs. The model reads /skill.md to learn which call is cheapest for which job.
Related: PDF to markdown for LLMs · knowledge base · extractor library · API reference.
What would this save you?
Move the sliders. The saving is computed from the measured ratios above and the model price you pick.
Assumes 10% of the page is kept per read (measured range 5–15%). Receipts in the product show the real number for your traffic.
Questions people ask
Does it use an LLM to decide which passages to keep?
No. Ranking is BM25 with phrase and heading bonuses, sub-millisecond and deterministic. Your model does the reasoning on the kept passages, so the saving is real and auditable.
Can I get the full markdown instead of passages?
Yes: omit q and you get readability-grade markdown with boilerplate removed; add maxTokens for an exact budget.
How do I scrape pages that need a login?
Create a session with a profile (saved cookies) or type a secret the model never sees, then scrape inside that session or save a routine for the clicks.
Is scraping with CrawlGenie legal?
You are responsible for the sites you read and the data you keep. CrawlGenie helps: it respects robots.txt in crawls by default, redacts PII on request, keeps retention settings, and identifies itself with a signed key instead of pretending to be a person.
Spend tokens on answers, not on pages.
Create a workspace in a minute. Connect ChatGPT, Claude, Cursor or your own agents. Every answer comes with a token receipt.