Everything we know about Answer Engine Optimization.
Primers, technical write-ups, and the full rubric behind the AEO Site Checker. Each article corresponds to one or more checks the auditor runs — read first, fix second, re-audit third.
Cloudflare's September 15 AI crawler defaults: what changes and what to check this week
From September 15, 2026, new Cloudflare zones block Training and Agent crawlers by default on pages that show ads, 'Verified' no longer means 'allowed', and robots.txt gains a use= signal. What each change does to your citations, and the three settings to confirm.
Where AI answers get their sources: what 680 million citations show
A synthesis of the largest 2025–2026 citation datasets — Profound, Goodie, Surfer, Semrush, Peec, Ahrefs — on which domains ChatGPT, Perplexity, and Google AI Overviews actually cite, how little the platforms overlap, and what a single site can do about it.
llms.txt, one year on: what the server logs say
Two large studies in 2025 and 2026 checked whether AI crawlers actually request /llms.txt and whether having one correlates with being cited. The answers are 'almost never' and 'no'. What the file is still good for, and why the audit keeps checking it.
Google AI Overviews and AI Mode: what actually decides whether you're in
Googlebot, not Google-Extended, controls AI Overviews and AI Mode. What Google's own documentation says about inclusion, the snippet controls that apply, how query fan-out changes what gets retrieved, and the 2026 numbers on how much search now runs through these surfaces.
Training, index, or live fetch: the three kinds of AI crawler
Every AI user agent does one of three jobs — model training, search indexing, or fetching a page on a user's behalf — and each job has different rules for robots.txt, verification, and what blocking it costs you. The 2026 reference, from the operators' own documentation.
The AEO scoring rubric, check by check
Every one of the 27 always-on checks and the 1 type-conditional check behind the 0–100 score: what each one measures, how many points it carries, why the weights are what they are, and how to read a result.
Cloudflare bot protection and AEO — the silent killer
Cloudflare's default bot challenge blocks the AI crawlers you want to be cited by. How to detect it, and the WAF rules to selectively unblock the major AI bots.
The Princeton GEO paper, summarized
What Aggarwal et al. (2024) actually measured about citation rates in generative search. The +41% statistics lift, the +28% quotation lift, and what the paper got wrong.
Front-loaded answers: the single highest-leverage AEO content fix
AI engines read the first ~200 words and stop. If your answer is buried in paragraph 7, you're invisible. How to rewrite so the model finds the answer immediately.
Why AI crawlers don't run JavaScript (and what to do about it)
AI search crawlers fetch raw HTML and don't render JavaScript. Why, how to tell if your site is invisible to them, and the fastest fixes for React, Vue, and SPA sites.
JSON-LD structured data for AI search
Which schema.org types AI engines actually read, where to put the JSON-LD, and the seven schemas that earn the most AEO points — copy-pasteable templates included.
How ChatGPT, Claude, and Perplexity actually crawl your site
A technical breakdown of how each major AI search engine fetches, parses, and selects sources to cite. From request headers to retrieval pipelines.
robots.txt for AI crawlers — the complete guide
Which AI crawlers exist, which ones actually matter, and the exact robots.txt allowlist to make sure ChatGPT, Claude, Perplexity, and Google AI Overviews can read your site.
What is /llms.txt and how to write one
/llms.txt is a markdown file at your site root that gives AI agents a curated index of your most important pages. The format, rules, and a copy-pasteable template.
What is Answer Engine Optimization (AEO)?
AEO is structuring a website so AI search engines — ChatGPT, Claude, Perplexity, Google AI Overviews — can crawl it and cite it. The four pillars, and how to start today.
AEO vs SEO: what's actually different
Answer Engine Optimization is not just SEO with a new name. The side-by-side: what's the same, what's different, and which old SEO tactics now hurt you.