robots.txt for Astro Sites

There are two ways to add robots.txt to an Astro site. The simplest is a static file: put robots.txt in public/ and Astro copies it to the root of dist/. The more flexible one is an endpoint at src/pages/robots.txt.ts that builds the file from your site config. That way the sitemap URL always matches your domain, and you can serve different rules on staging.

Key Takeaways

• Static: public/robots.txt. Dynamic: src/pages/robots.txt.ts returning a Response.
• Always include a Sitemap: line with an absolute URL.
• robots.txt controls crawling, not indexing. Use noindex to keep a page out of results.
• Block staging with an environment check, not by remembering to edit a file.

Option 1: a static file

# public/robots.txt
User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap-index.xml

Files in public/ are served as-is at the same path, so this ends up at /robots.txt. Good enough for most sites.

Option 2: an endpoint

// src/pages/robots.txt.ts
import type { APIRoute } from 'astro';

const allowAll = (sitemap: string) => `User-agent: *
Allow: /

Sitemap: ${sitemap}
`;

export const GET: APIRoute = ({ site }) => {
  const sitemap = new URL('sitemap-index.xml', site).href;
  return new Response(allowAll(sitemap), {
    headers: { 'Content-Type': 'text/plain; charset=utf-8' },
  });
};

In a static build, Astro runs this once and writes dist/robots.txt. The site value comes from astro.config.mjs, so changing domains updates the sitemap line automatically. Delete public/robots.txt if you switch to this; two sources for the same path causes a conflict.

Block staging and previews

Staging sites shouldn’t be crawled. Decide from an environment variable at build time:

// src/pages/robots.txt.ts
import type { APIRoute } from 'astro';

export const GET: APIRoute = ({ site }) => {
  const isProduction = import.meta.env.PUBLIC_DEPLOY_ENV === 'production';

  const body = isProduction
    ? `User-agent: *\nAllow: /\n\nSitemap: ${new URL('sitemap-index.xml', site).href}\n`
    : `User-agent: *\nDisallow: /\n`;

  return new Response(body);
};

Set PUBLIC_DEPLOY_ENV=production only in your production environment on the host. Every other build blocks crawlers. For stronger protection on staging, put the site behind authentication or add noindex headers; robots.txt is a request, not access control.

Blocking AI crawlers

To opt out of AI training crawlers while staying in search, list their user agents:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap-index.xml

Consider the trade-off first. Blocking these crawlers can reduce how often AI assistants cite your pages. If you want your content quoted in AI answers, keep them allowed. User agent names change over time, so check each provider’s documentation.

Disallowing specific paths

User-agent: *
Disallow: /admin/
Disallow: /search/
Allow: /

Disallowed pages can still appear in results if other sites link to them, just without a description. To keep a page out of the index, allow crawling and add a noindex meta tag instead. Search engines can only see the tag on pages they’re allowed to crawl.

Check it

npm run build && cat dist/robots.txt

After deploying, open https://your-domain/robots.txt and use the robots.txt report in Google Search Console to confirm Google can read it. Google’s robots.txt introduction covers the syntax rules.

Generating the file from site keeps it consistent with your canonical URLs. The Sitemap: line should point to the file the sitemap integration creates; see add a sitemap to Astro. Both are covered in the Astro SEO checklist.

FAQ

Where do I put robots.txt in an Astro project?

In public/robots.txt for a static file, or src/pages/robots.txt.ts for a generated one. Use one or the other, not both.

Does Astro generate robots.txt automatically?

No. Neither Astro nor the sitemap integration creates it; add one yourself.