--- title: "No robots.txt (seo/robots-txt-missing)" description: "seo/robots-txt-missing: A 404 means \"crawl everything\", but you lose the Sitemap directive and any explicit AI crawler policy." canonical: https://void-design.vercel.app/rules/seo/robots-txt-missing lastModified: 2026-09-16 --- # No robots.txt `seo/robots-txt-missing` · severity **warn** · category SEO · detected by `void seo` and `void audit` ## Why it matters A 404 means "crawl everything", but you lose the Sitemap directive and any explicit AI crawler policy. ## How to fix it Add app/robots.ts with a Sitemap line and an explicit policy for AI crawlers. ## Example ```ts // app/robots.ts import type { MetadataRoute } from 'next' export default function robots(): MetadataRoute.Robots { if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== 'production') { return { rules: [{ userAgent: '*', disallow: '/' }] } // previews never indexable } return { rules: [{ userAgent: '*', allow: '/', disallow: ['/api/', '/account/'] }], sitemap: 'https://acme.example/sitemap.xml', } } ``` ## References - https://nextjs.org/docs/app/api-reference/file-conventions/metadata/robots - https://developers.google.com/crawling/docs/robots-txt/robots-txt-spec ## More seo rules `void seo` reports 60 rules in this category. What search engines need from the raw response: status codes, titles and descriptions in , self-referencing canonicals, one h1, crawlable links, Open Graph, icons, robots.txt and a sitemap with real dates. - `apple-touch-icon` apple-touch-icon missing or not a 180×180 PNG — [seo/apple-touch-icon](https://void-design.vercel.app/rules/seo/apple-touch-icon) - `manifest` Web app manifest missing or incomplete — [seo/manifest](https://void-design.vercel.app/rules/seo/manifest) - `theme-color` No theme-color meta — [seo/theme-color](https://void-design.vercel.app/rules/seo/theme-color) - `robots-txt-unreachable` robots.txt returns 5xx/429 or times out — [seo/robots-txt-unreachable](https://void-design.vercel.app/rules/seo/robots-txt-unreachable) - `robots-txt-invalid` robots.txt serves HTML, exceeds 500 KiB, or has unparseable lines — [seo/robots-txt-invalid](https://void-design.vercel.app/rules/seo/robots-txt-invalid) - `robots-blocks-all` robots.txt disallows the site for * / Googlebot / Bingbot — [seo/robots-blocks-all](https://void-design.vercel.app/rules/seo/robots-blocks-all) - `robots-blocks-resources` robots.txt blocks JS/CSS needed to render — [seo/robots-blocks-resources](https://void-design.vercel.app/rules/seo/robots-blocks-resources) - `robots-sitemap-directive` robots.txt has no absolute Sitemap: line — [seo/robots-sitemap-directive](https://void-design.vercel.app/rules/seo/robots-sitemap-directive) Detected by `void seo` and `void audit`. Explain it in a terminal: `void rules seo/robots-txt-missing`