Skip to content

Reference · seo

llms.txt and Markdown mirrors

seo/references/llms-txt-markdown.md147 linesupdated 16 Sept 2026

Status (2026-09): llmstxt.org spec v2 (August 2026) adds and per-page rel="alternate" type="text/markdown" discovery. Lighthouse 13 has an experimental llms-txt audit (a 5xx fails; a 404 is N/A).

Evidence of value is thin. Ship it because it's cheap, and score it info. In Ahrefs' 137K-domain log study (2026-06), 97% of llms.txt files got zero requests. The rest came mostly from SEO tools and coding agents (GPTBot, Claude-Code), and retrieval bots made 1.1%. Google's AI guide says llms.txt and Markdown are not used by Google Search. The main beneficiaries are coding agents reading your docs.

Canonical code: templates/next/src/app/llms.txt/route.ts and src/app/md/[...slug]/route.ts, both generated from the page registry src/content/pages.ts (path, title, description, lastModified, section, markdown()).

1. /llms.txt

ts
// src/app/llms.txt/route.ts
import { docs, posts } from "@/lib/content";
import { site } from "@/site.config";

export const dynamic = "force-static";

export function GET() {
  const body = [
    `# ${site.name}`,                                    // required H1: the exact brand name (entity consistency)
    "",
    `> ${site.description}`,                             // one-sentence summary blockquote
    "",
    `${site.name} is open source (MIT). This file lists the latest stable docs. Pricing: free for OSS, Team $12/seat/month.`,
    "",
    "## Docs",
    "",
    ...docs.map((d) => `- [${d.title}](${site.url}${d.path}.md): ${d.description}`),
    "",
    "## Blog",
    "",
    ...posts.slice(0, 20).map((p) => `- [${p.title}](${site.url}/blog/${p.slug}.md): ${p.description}`),
    "",
    "## Optional",                                        // agents may skip this section
    "",
    `- [Changelog](${site.url}/changelog.md)`,
    "",
  ].join("\n");
  return new Response(body, { headers: { "Content-Type": "text/markdown; charset=utf-8" } });
}

Rules:

  • Curated, not a sitemap dump. Under ~100 KB. It starts with # H1, then > summary, then ## Section lists of - [name](url): note.
  • Link targets are .md mirrors or clean server-rendered HTML, never login walls or client-only apps. Every link returns 200.
  • Only list what exists. Facts come from site.config.ts so llms.txt, JSON-LD and page copy never disagree (a real site once claimed the wrong license in llms.txt).
  • Leave out noindex, auth and app routes.
  • Optional llms-full.txt (all Markdown concatenated) is a convention (nextjs.org ships one), not part of the spec.
  • Advertise it site-wide from the root layout: (React 19 hoists it into ).

2. Markdown mirrors: explicit .md URLs (preferred)

ts
// src/app/md/[...slug]/route.ts
import { getPost, posts } from "@/lib/content";
import { site } from "@/site.config";

export const dynamic = "force-static";
export const dynamicParams = false;

export function generateStaticParams() {
  return posts.map((p) => ({ slug: ["blog", p.slug] }));
}

export async function GET(_req: Request, { params }: { params: Promise<{ slug: string[] }> }) {
  const { slug } = await params;
  const post = slug[0] === "blog" ? getPost(slug[1] ?? "") : undefined;
  if (!post) return new Response("Not found", { status: 404 });
  const html = `${site.url}/blog/${post.slug}`;
  const md = `---\ntitle: ${JSON.stringify(post.title)}\ncanonical: ${html}\ndateModified: ${post.updated}\n---\n\n# ${post.title}\n\n${post.markdown}\n`;
  return new Response(md, {
    headers: {
      "Content-Type": "text/markdown; charset=utf-8",
      "X-Robots-Tag": "noindex",                 // don't compete with the HTML page in search
      Link: `<${html}>; rel="canonical"`,
    },
  });
}

Route .md URLs to that handler with rewrites (the template's approach, no proxy needed):

ts
// next.config.ts
async rewrites() {
  return [
    { source: "/index.md", destination: "/md/index" },
    { source: "/:path+.md", destination: "/md/:path+" },
  ];
},

Or with a proxy (needed anyway if you also negotiate on Accept, §3):

ts
// src/proxy.ts (Next 16 renamed middleware.ts → proxy.ts, export name `proxy`)
import { NextResponse, type NextRequest } from "next/server";

export function proxy(request: NextRequest) {
  const { pathname } = request.nextUrl;
  if (pathname.endsWith(".md")) {
    return NextResponse.rewrite(new URL(`/md${pathname.slice(0, -3)}`, request.url)); // /blog/x.md → /md/blog/x
  }
  return NextResponse.next();
}
export const config = { matcher: ["/blog/:path*", "/docs/:path*"] };

Page metadata advertises the mirror with pageMetadata({ …, markdown: true }), which emits .

Verified: GET /blog/hello-world.md returns 200 text/markdown with x-robots-tag: noindex and a canonical Link header.

Mirror rules:

  • The same facts as the HTML (main-text similarity ≥ 0.8). Materially different content for bots risks being treated as cloaking.
  • Not in sitemap.xml.
  • Strip nav, footer and cookie text. Keep headings, tables, code blocks and links (absolute URLs).

3. Accept: text/markdown negotiation (only behind a cache-aware edge)

ts
// inside proxy(): same URL, Markdown for agents that ask
const accept = request.headers.get("accept") ?? "";
if (accept.includes("text/markdown")) {
  const res = NextResponse.rewrite(new URL(`/md${pathname}`, request.url));
  res.headers.set("Vary", "Accept");
  return res;
}

Caveat (verified on 16.3.5): on the HTML response for prerendered routes, Next overwrote a custom Vary: Accept (set via proxy or headers()). A shared CDN can then serve Markdown to browsers. Use negotiation only where the proxy runs before the cache (e.g. Vercel), or add Vary: Accept at the CDN. Otherwise use explicit .md URLs only.

Alternative: Cloudflare "Markdown for Agents" converts at the edge (paid zones). It adds vary: accept and a default Content-Signal.

4. Other stacks

  • Astro: src/pages/llms.txt.ts exporting GET with prerender = true; .md mirrors via src/pages/[...slug].md.ts from content collections.
  • Static hosts: generate llms.txt and *.md at build time into the output dir. Set X-Robots-Tag: noindex for *.md in host headers config (_headers on Netlify/Cloudflare Pages).

Sources