---
title: "llms.txt and Markdown mirrors (seo)"
description: "Status (2026-09): llmstxt.org spec v2 (August 2026) adds and per-page rel=\"alternate\" type=\"text/markdown\" discovery."
canonical: https://void-design.vercel.app/docs/seo/llms-txt-markdown
lastModified: 2026-09-16
---
# llms.txt and Markdown mirrors
Status (2026-09): llmstxt.org spec **v2** (August 2026) adds `` and per-page `rel="alternate" type="text/markdown"` discovery. Lighthouse 13 has an experimental `llms-txt` audit (a 5xx fails; a 404 is N/A).
**Evidence of value is thin. Ship it because it's cheap, and score it `info`.** In Ahrefs' 137K-domain log study (2026-06), 97% of llms.txt files got zero requests. The rest came mostly from SEO tools and coding agents (GPTBot, Claude-Code), and retrieval bots made 1.1%. Google's AI guide says llms.txt and Markdown are **not used** by Google Search. The main beneficiaries are coding agents reading your docs.
Canonical code: `templates/next/src/app/llms.txt/route.ts` and `src/app/md/[...slug]/route.ts`, both generated from the page registry `src/content/pages.ts` (path, title, description, lastModified, section, `markdown()`).
## 1. `/llms.txt`
```ts
// src/app/llms.txt/route.ts
import { docs, posts } from "@/lib/content";
import { site } from "@/site.config";
export const dynamic = "force-static";
export function GET() {
const body = [
`# ${site.name}`, // required H1: the exact brand name (entity consistency)
"",
`> ${site.description}`, // one-sentence summary blockquote
"",
`${site.name} is open source (MIT). This file lists the latest stable docs. Pricing: free for OSS, Team $12/seat/month.`,
"",
"## Docs",
"",
...docs.map((d) => `- [${d.title}](${site.url}${d.path}.md): ${d.description}`),
"",
"## Blog",
"",
...posts.slice(0, 20).map((p) => `- [${p.title}](${site.url}/blog/${p.slug}.md): ${p.description}`),
"",
"## Optional", // agents may skip this section
"",
`- [Changelog](${site.url}/changelog.md)`,
"",
].join("\n");
return new Response(body, { headers: { "Content-Type": "text/markdown; charset=utf-8" } });
}
```
Rules:
- Curated, not a sitemap dump. Under ~100 KB. It starts with `# H1`, then `> summary`, then `## Section` lists of `- [name](url): note`.
- Link targets are `.md` mirrors or clean server-rendered HTML, never login walls or client-only apps. Every link returns 200.
- Only list what exists. Facts come from `site.config.ts` so llms.txt, JSON-LD and page copy never disagree (a real site once claimed the wrong license in llms.txt).
- Leave out noindex, auth and app routes.
- Optional `llms-full.txt` (all Markdown concatenated) is a convention (nextjs.org ships one), not part of the spec.
- Advertise it site-wide from the root layout: `` (React 19 hoists it into ``).
## 2. Markdown mirrors: explicit `.md` URLs (preferred)
```ts
// src/app/md/[...slug]/route.ts
import { getPost, posts } from "@/lib/content";
import { site } from "@/site.config";
export const dynamic = "force-static";
export const dynamicParams = false;
export function generateStaticParams() {
return posts.map((p) => ({ slug: ["blog", p.slug] }));
}
export async function GET(_req: Request, { params }: { params: Promise<{ slug: string[] }> }) {
const { slug } = await params;
const post = slug[0] === "blog" ? getPost(slug[1] ?? "") : undefined;
if (!post) return new Response("Not found", { status: 404 });
const html = `${site.url}/blog/${post.slug}`;
const md = `---\ntitle: ${JSON.stringify(post.title)}\ncanonical: ${html}\ndateModified: ${post.updated}\n---\n\n# ${post.title}\n\n${post.markdown}\n`;
return new Response(md, {
headers: {
"Content-Type": "text/markdown; charset=utf-8",
"X-Robots-Tag": "noindex", // don't compete with the HTML page in search
Link: `<${html}>; rel="canonical"`,
},
});
}
```
Route `.md` URLs to that handler with rewrites (the template's approach, no proxy needed):
```ts
// next.config.ts
async rewrites() {
return [
{ source: "/index.md", destination: "/md/index" },
{ source: "/:path+.md", destination: "/md/:path+" },
];
},
```
Or with a proxy (needed anyway if you also negotiate on `Accept`, §3):
```ts
// src/proxy.ts (Next 16 renamed middleware.ts → proxy.ts, export name `proxy`)
import { NextResponse, type NextRequest } from "next/server";
export function proxy(request: NextRequest) {
const { pathname } = request.nextUrl;
if (pathname.endsWith(".md")) {
return NextResponse.rewrite(new URL(`/md${pathname.slice(0, -3)}`, request.url)); // /blog/x.md → /md/blog/x
}
return NextResponse.next();
}
export const config = { matcher: ["/blog/:path*", "/docs/:path*"] };
```
Page metadata advertises the mirror with `pageMetadata({ …, markdown: true })`, which emits ``.
Verified: `GET /blog/hello-world.md` returns 200 `text/markdown` with `x-robots-tag: noindex` and a canonical `Link` header.
Mirror rules:
- The same facts as the HTML (main-text similarity ≥ 0.8). Materially different content for bots risks being treated as cloaking.
- Not in `sitemap.xml`.
- Strip nav, footer and cookie text. Keep headings, tables, code blocks and links (absolute URLs).
## 3. `Accept: text/markdown` negotiation (only behind a cache-aware edge)
```ts
// inside proxy(): same URL, Markdown for agents that ask
const accept = request.headers.get("accept") ?? "";
if (accept.includes("text/markdown")) {
const res = NextResponse.rewrite(new URL(`/md${pathname}`, request.url));
res.headers.set("Vary", "Accept");
return res;
}
```
**Caveat (verified on 16.3.5):** on the *HTML* response for prerendered routes, Next overwrote a custom `Vary: Accept` (set via proxy or `headers()`). A shared CDN can then serve Markdown to browsers. Use negotiation only where the proxy runs before the cache (e.g. Vercel), or add `Vary: Accept` at the CDN. Otherwise use explicit `.md` URLs only.
Alternative: Cloudflare "Markdown for Agents" converts at the edge (paid zones). It adds `vary: accept` and a default `Content-Signal`.
## 4. Other stacks
- Astro: `src/pages/llms.txt.ts` exporting `GET` with `prerender = true`; `.md` mirrors via `src/pages/[...slug].md.ts` from content collections.
- Static hosts: generate `llms.txt` and `*.md` at build time into the output dir. Set `X-Robots-Tag: noindex` for `*.md` in host headers config (`_headers` on Netlify/Cloudflare Pages).
## Sources
- https://llmstxt.org/ · https://llmstxt.org/changes.md
- https://developer.chrome.com/docs/lighthouse/agentic-browsing/llms-txt
- https://ahrefs.com/blog/llmstxt-study/
- https://developers.google.com/search/docs/fundamentals/ai-optimization-guide
- https://nextjs.org/docs/app/guides/ai-agents · https://nextjs.org/docs/app/api-reference/file-conventions/proxy
- https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/