Skip to content

Rule · SEO

No robots.txt

seo/robots-txt-missingwarnvoid seoupdated

Why it matters

A 404 means "crawl everything", but you lose the Sitemap directive and any explicit AI crawler policy.

How to fix it

Add app/robots.ts with a Sitemap line and an explicit policy for AI crawlers.

Example

ts
// app/robots.ts
import type { MetadataRoute } from 'next'
export default function robots(): MetadataRoute.Robots {
  if (process.env.VERCEL_ENV && process.env.VERCEL_ENV !== 'production') {
    return { rules: [{ userAgent: '*', disallow: '/' }] } // previews never indexable
  }
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: ['/api/', '/account/'] }],
    sitemap: 'https://acme.example/sitemap.xml',
  }
}

References

More seo rules

void seo reports 60 rules in this category. What search engines need from the raw response: status codes, titles and descriptions in <head>, self-referencing canonicals, one h1, crawlable links, Open Graph, icons, robots.txt and a sitemap with real dates.