---
title: "AI crawler policy (allow/deny matrix from robots.txt) (geo/robots-ai-policy)"
description: "geo/robots-ai-policy: Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot)…"
canonical: https://void-design.vercel.app/rules/geo/robots-ai-policy
lastModified: 2026-09-16
---

# AI crawler policy (allow/deny matrix from robots.txt)

`geo/robots-ai-policy` · severity **info** · category AI search · detected by `void geo` and `void audit`

## Why it matters

Training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot…) and retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) are separate tokens; an explicit policy controls each independently. A bot named in its own group ignores the * group.

## How to fix it

State the policy explicitly in app/robots.ts. Allow retrieval bots to stay visible in AI search, and decide training separately.

## Example

```ts
// app/robots.ts
const RETRIEVAL = ['OAI-SearchBot', 'ChatGPT-User', 'Claude-SearchBot', 'Claude-User', 'PerplexityBot', 'Perplexity-User']
const TRAINING = ['GPTBot', 'ClaudeBot', 'Google-Extended', 'Applebot-Extended', 'CCBot', 'meta-externalagent', 'Amazonbot', 'MistralAI-Training', 'Bytespider']
const PRIVATE = ['/api/', '/account/']

export default function robots(): MetadataRoute.Robots {
  const allowTraining = process.env.SEO_ALLOW_AI_TRAINING !== 'false'
  return {
    rules: [
      { userAgent: '*', allow: '/', disallow: PRIVATE },
      { userAgent: RETRIEVAL, allow: '/', disallow: PRIVATE }, // repeat disallows: named groups ignore '*'
      ...(allowTraining ? [] : [{ userAgent: TRAINING, disallow: '/' }]),
    ],
    sitemap: 'https://acme.example/sitemap.xml',
  }
}
```

## References

- https://developers.openai.com/api/docs/bots
- https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- https://docs.perplexity.ai/guides/bots
- https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

## More ai search rules

`void geo` reports 41 rules in this category. Generative-engine optimisation: whether AI crawlers that don't run JavaScript see the same content, valid and visible JSON-LD, an explicit AI robots policy, llms.txt, Markdown mirrors and answer-first writing.

- `llms-txt-missing` No /llms.txt — [geo/llms-txt-missing](https://void-design.vercel.app/rules/geo/llms-txt-missing)
- `llms-txt-error` /llms.txt returns a server error — [geo/llms-txt-error](https://void-design.vercel.app/rules/geo/llms-txt-error)
- `llms-txt-invalid` /llms.txt doesn't follow the llms.txt format — [geo/llms-txt-invalid](https://void-design.vercel.app/rules/geo/llms-txt-invalid)
- `llms-txt-broken-links` /llms.txt links to URLs that fail — [geo/llms-txt-broken-links](https://void-design.vercel.app/rules/geo/llms-txt-broken-links)
- `llms-txt-not-linked` llms.txt exists but isn't linked from the home page — [geo/llms-txt-not-linked](https://void-design.vercel.app/rules/geo/llms-txt-not-linked)
- `search-bots-blocked` robots.txt blocks AI search or user-triggered fetchers — [geo/search-bots-blocked](https://void-design.vercel.app/rules/geo/search-bots-blocked)
- `training-bots-blocked` robots.txt blocks AI training crawlers — [geo/training-bots-blocked](https://void-design.vercel.app/rules/geo/training-bots-blocked)
- `content-signal` No Content-Signal usage preferences — [geo/content-signal](https://void-design.vercel.app/rules/geo/content-signal)

Detected by `void geo` and `void audit`. Explain it in a terminal: `void rules geo/robots-ai-policy`