llms.txt checker: what does a site tell agents about itself?
llms.txt is a plain-text index a site publishes for language models (and llms-full.txt is the whole documentation as one file). This checker fetches both for any domain, validates the format, lists the linked pages with their summaries, and counts the tokens so you know whether reading it beats crawling.
What should a good llms.txt contain?
An H1 with the site name, a one-paragraph blockquote summary, H2 sections, and bullet links in the form "[Title](URL): one-line summary". Optional sections can be marked so agents skip them. Keep it current: a stale index sends agents to pages that moved. For scale: Anthropic's documentation index lists 760 links and about 22,000 tokens, while its llms-full.txt runs to roughly 401,000 tokens; adoption studies in 2026 found between 10% and 28% of domains with a valid file.
Do AI crawlers actually read llms.txt?
Coding agents and some assistants do; the big search crawlers mostly do not yet (one 2026 log study counted seven fetches of llms.txt against thousands of robots.txt). It is cheap, it helps agents that use it a lot, and it does not replace crawler access rules or structured data. CrawlGenie publishes its own at /llms.txt and reads others' with the llms_txt extractor.
How do agents use this inside CrawlGenie?
For developers: the exact call
POST /v1/extractors/llms_txt/run {"input": {"url": "https://docs.stripe.com"}}
→ hasIndex, hasFull, indexTokens, fullTokens, rows[] {section, title, url, summary}Related: SEO + AIO report · documents guide · knowledge base.
Questions people ask
What is the difference between llms.txt and robots.txt?
robots.txt tells crawlers where they may go; llms.txt tells language models what the site is and which pages matter. One is permission, the other is a map.
Can this tool generate an llms.txt for my site?
Not yet; it checks and validates. A crawl plus the docs index in CrawlGenie produces the material you need in a minute.
Spend tokens on answers, not on pages.
Create a workspace in a minute. Connect ChatGPT, Claude, Cursor or your own agents. Every answer comes with a token receipt.