/evaluators

Technical SEO · seo-ai-crawler-access

AI Bot Access Evaluator

Checks whether robots.txt and page-level robots directives let major AI crawlers access public content while keeping private app routes blocked.

What it checks

If AI crawlers cannot reach public pages, the site may be invisible in answer engines even when the content is strong.

Pass criteria

  • Public marketing, docs, tools, and research pages are crawlable.
  • Private routes such as /dashboard and /api are blocked where appropriate.
  • Robots policy distinguishes public content access from private app surfaces.

Example

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Disallow: /dashboard/
Disallow: /api/

Common failures

  • - Blocking every AI crawler globally.
  • - Allowing private dashboard or API routes into public crawl paths.
  • - Using page metadata that contradicts robots.txt.

Related surfaces

MCP aeo_scan/tools/robots-generator/docs/api

FAQ

Should AI crawlers be allowed on every route?
No. Public content pages should be crawlable; authenticated dashboards, private APIs, and internal tools should remain blocked.