SEO for the AI Era: How to Use llms.txt to Guide ChatGPT, Gemini, and Perplexity

Illustration of AI crawlers connecting to a website

Beyond Google: SEO Meets AI Assistants

Readers now discover content through ChatGPT, Gemini, Perplexity, and other AI surfaces. To control how these models crawl and quote your site, add an llms.txt file--think of it as robots.txt for large language models.

What Is llms.txt?

Place a plain-text file at https://example.com/llms.txt. Inside, list permissions for AI crawlers. They will check this file before training on, summarizing, or citing your pages.

Allow Everything

User-Agent: *
Allow: /

Block a Folder

User-Agent: *
Disallow: /members-only/

Block a Specific LLM Crawler

User-Agent: gpt-4
Disallow: /

Why It Matters

AI search experiences reward transparent sites. When you allow access, assistants can cite you, link back, and surface your brand in Discover-style carousels. Denying access keeps your content out of their summaries. llms.txt declares those boundaries explicitly.

Publish an AI Use Policy

Pair the file with an /ai-policy page that states:

Whether AI-generated summaries are permitted.
How attribution should appear.
Contact details for correction or takedown requests.

This makes expectations clear for both bots and human editors.

Make Your Pages AI-Friendly

Permissions alone aren't enough. Optimize signals that help LLMs understand and quote you accurately:

Rich OGP metadata (title, description, social image).
JSON-LD BlogPosting or relevant schema.
Author bios and publisher information.
Clean heading hierarchy with descriptive subheads.
Internal links that clarify topic clusters.

Set It Up Now

Future SEO means serving humans and AI in tandem. Configure llms.txt, document your policies, and polish your metadata so assistants share--and credit--your insights everywhere your audience searches.