What is llms.txt?

llms.txt is a proposed convention for giving AI agents a concise, structured overview of a website.

It is a Markdown-formatted text file normally available at:

/llms.txt

Instead of forcing an agent to understand an entire website from navigation, JavaScript, advertisements, templates, sidebars, and other page elements, the file provides a curated starting point.

An llms.txt file can tell an agent:

• What the website or project is.
• What information is most important.
• Where documentation can be found.
• Where product information can be found.
• Which guides or reference material should be consulted.
• Which resources are secondary or optional.

The file acts more like a guide to useful content than a complete copy of the website.

Why does llms.txt exist?

Websites are primarily designed for people.

A typical webpage may contain:

• Navigation menus
• Cookie banners
• Advertising
• JavaScript
• Footers
• Repeated templates
• Related-content modules
• Tracking scripts
• Interactive elements

AI agents usually need the useful information inside the page rather than all of that surrounding interface.

llms.txt provides a lightweight way for a site owner to say:

This is what this website is about, and these are the resources you should read if you need more information.

The file is intentionally concise. Detailed information should generally remain on the pages linked from it.

Does llms.txt improve AI rankings?

There is currently no reliable basis for promising that publishing llms.txt will improve rankings, citations, traffic, recommendations, or visibility in any particular AI assistant.

llms.txt should therefore not be treated as an AI ranking trick.

Its practical purpose is narrower:

Make your website easier to navigate and interpret when an AI agent chooses to use the file.

Different AI systems may:

• Read llms.txt.
• Ignore llms.txt.
• Discover your content through search instead.
• Crawl individual webpages.
• Use their own indexes.
• Combine several discovery methods.

Publishing the file makes useful structured information available. It does not control what an AI system chooses to retrieve or cite.

Where should llms.txt be located?

For a whole website, the conventional location is:

https://example.com/llms.txt

An llms.txt file can also exist inside a subdirectory.

For example:

https://example.com/docs/llms.txt

A file placed under /docs/ can describe the content beneath /docs/.

This allows large websites to maintain more focused files for different sections.

For example:

/llms.txt
/docs/llms.txt
/developers/llms.txt

When several files could apply, an agent can use the file located closest to the content it is examining.

What does an llms.txt file contain?

The llmstxt.org format is intentionally simple.

A conforming file can contain:

1. An H1 title.
2. An optional blockquote summary.
3. Optional explanatory content.
4. H2 sections containing lists of useful resources.

The H1 is the core required element.

A well-developed file will normally contain considerably more context.

llms.txt format reference

The typical structure is:

# Site or Project Name

> A short description containing the most important context an agent
> needs to understand the website.

Additional information or instructions can appear here.

## Documentation

- [Documentation title](URL): Short explanation of the resource
- [Another document](URL): Explanation of what is available here

## Products

- [Product](URL): Product description

## Guides

- [Guide](URL): Description of the guide

## Optional

- [Secondary resource](URL): Additional information

H1 title

Example:

# Acme Analytics

The H1 identifies the website, company, project, product, or section covered by the file.

There should be a clear H1 near the beginning of the document.

Summary blockquote

Example:

> Acme Analytics helps ecommerce teams understand customer acquisition,
> conversion and retention across their marketing channels.

The summary should contain enough context for an agent to understand what the site represents.

Keep it:

• Short
• Specific
• Factual
• Understandable without marketing jargon

Additional context

After the summary, you can add information that changes how the linked resources should be interpreted.

For example:

The documentation applies to version 4 of the Acme API.

For current pricing and plan limits, use the pricing page rather than
older blog articles.

Use this area for meaningful instructions rather than promotional copy.

H2 sections

Resources are grouped beneath H2 headings:

## Documentation

Possible section names include:

Documentation
API
Products
Services
Guides
Reference
Policies
Blog
About
Support
Optional

Choose categories that make sense for your website.

File lists

Resources beneath an H2 should be presented as Markdown links.

For example:

- [API authentication](https://example.com/docs/auth): Authentication methods and token requirements

A useful entry contains:

1. A descriptive link title.
2. A destination.
3. A concise description of what the agent will find there.

Avoid entries such as:

- [Click here](...)

Prefer:

- [API authentication guide](...): How applications obtain and refresh API access tokens

Optional section

## Optional conventionally contains resources that are useful but not essential.

For example:

## Optional

- [Company history](...): Background on the company's founding and major milestones
- [Engineering blog](...): Technical articles from the engineering team

An agent operating with a limited context budget can skip this material.

Example llms.txt file

Here is a simple example for a local software company:

# Acme CRM

> Acme CRM is customer relationship management software for small
> professional-services businesses.

Pricing and product capabilities change periodically. Use the current
pricing and product pages for plan-specific information.

## Product

- [CRM overview](https://example.com/product): Overview of the CRM, contact management and sales features
- [Pricing](https://example.com/pricing): Current plans, prices and feature limits
- [Integrations](https://example.com/integrations): Supported third-party integrations

## Documentation

- [Getting started](https://example.com/docs/getting-started): Initial account setup and configuration
- [Contact management](https://example.com/docs/contacts): Creating, importing and organizing contacts
- [API documentation](https://example.com/docs/api): API endpoints, authentication and examples

## Support

- [Help center](https://example.com/help): Troubleshooting and account help
- [System status](https://status.example.com): Current service availability

## About

- [About Acme](https://example.com/about): Company background and contact information

## Optional

- [Blog](https://example.com/blog): Articles about CRM, sales operations and customer management

The purpose is not to list every URL on the website.

The purpose is to identify the URLs that best explain the site.

How to generate an llms.txt file

Open llms.txt Studio and enter your website domain into the generator.

The free generator can inspect your publicly accessible site and construct an llms.txt file using information already published on your pages.

No AI model is required to write the generated copy.

The generator uses existing information such as:

• Page titles
• Headings
• Meta descriptions
• Website structure
• Sitemap information

After generation:

1. Review the title.
2. Review the summary.
3. Check the selected URLs.
4. Make sure descriptions are accurate.
5. Remove irrelevant URLs.
6. Add important resources that were missed.
7. Publish the result as /llms.txt.
8. Run the validator against the live file.

You remain responsible for deciding which information best represents your website.

How llms.txt Studio generates your file

The Studio generator begins by examining your site's crawl instructions and discoverable structure.

A typical generation pass checks:

robots.txt
    ↓
sitemap.xml
    ↓
homepage
    ↓
important internal pages
    ↓
categorization
    ↓
llms.txt

The crawler can inspect up to 60 HTML pages during a generation job.

Pages may be organized into useful families such as:

• Documentation
• API
• Product
• Pricing
• Blog
• About

The generated titles and descriptions are derived from information already published by the website.

This makes generation deterministic rather than asking an AI model to invent summaries about your company.

If an important page has a poor title or no useful description, improving the original page may also improve the generated file.

How to validate an llms.txt file

Studio provides two validation methods.

Validate a live file

Enter a domain and the validator attempts to retrieve:

https://example.com/llms.txt

It then checks the file's structure and samples linked resources.

Validate pasted text

You can also paste a draft before publishing it.

This is useful for checking the structure while developing or editing a file.

What validation checks

Validation can detect issues such as:

• Missing llms.txt
• HTML returned instead of text
• Missing H1
• Missing summary
• Missing sections
• Malformed resource entries
• Dead links
• Duplicate URLs
• Excessive numbers of links
• Off-domain links
• Relevant robots.txt directives

A validation grade measures the quality and usability of the file according to Studio's validator.

It is not an AI visibility score and does not predict whether an AI assistant will cite the site.

Validator rules

llms.txt Studio currently uses the following rule families.

Errors identify serious structural or delivery problems.

Warnings identify conditions that may make the file less useful.

Information findings provide context and do not reduce the grade.

Studio samples up to 20 linked resources when checking link availability rather than attempting to crawl every destination in a large file.

What is llms-full.txt?

llms-full.txt is an expanded representation that can contain page content directly rather than acting primarily as a directory of links.

Think of the difference as:

llms.txt
    → concise map of important information

llms-full.txt
    → larger collection containing more of the information itself

In Studio, generation of llms-full.txt is optional.

The Studio generator currently limits its generated full file to 500 KB.

Is llms-full.txt part of the core llmstxt.org format?

Do not confuse the two.

The core llmstxt.org proposal focuses on llms.txt as a compact Markdown guide linking agents to useful resources.

llms-full.txt is an additional convention used by some tools and websites.

You should generally publish a good /llms.txt even if you do not publish /llms-full.txt.

llms.txt vs robots.txt

These files solve different problems.

robots.txt

robots.txt communicates crawling preferences or restrictions to automated clients.

It answers questions such as:

May this crawler access these URLs?

llms.txt

llms.txt provides context and recommended resources.

It answers:

If you need to understand this website, where should you look?

A simple comparison:

An llms.txt file does not override robots.txt.

llms.txt vs sitemap.xml

A sitemap and an llms.txt file can coexist.

They are not substitutes.

sitemap.xml

A sitemap generally attempts to enumerate pages that search engines may want to discover.

It can contain hundreds, thousands, or millions of URLs.

llms.txt

llms.txt should be selective.

Instead of saying:

Here are all 4,000 pages on our website.

It says:

Here are the resources that best explain what this website does.

That difference matters because an AI agent often operates with a limited amount of context.

A concise, curated resource list is easier to work with than an exhaustive URL inventory.

Markdown versions of webpages

The llmstxt.org proposal also recommends making important pages available as clean Markdown.

For example, an HTML documentation page could have a Markdown equivalent:

/docs/authentication
/docs/authentication.md

or:

/docs/authentication.html
/docs/authentication.html.md

The exact implementation depends on the site's URL structure.

The purpose is to give an agent access to the meaningful text without requiring it to reconstruct content from complex HTML.

Your llms.txt can then link directly to these Markdown-friendly resources.

Advanced discovery with Link headers

Websites can make Markdown alternatives easier for clients to discover using standard link relations.

Conceptually:

Link: </docs/page.md>; rel="alternate"; type="text/markdown"

A page can also indicate which llms.txt describes it using a describedby relationship.

Conceptually:

Link: </docs/llms.txt>; rel="describedby"

These declarations may be exposed through:

• HTML <link> elements
• HTTP Link response headers

This is an advanced implementation and is not required to get started with a basic root-level /llms.txt.

How monitoring works

Creating a correct file once does not guarantee that it stays correct.

Websites change.

For example:

Product page changes
        ↓
Old llms.txt remains unchanged
        ↓
Link eventually redirects or disappears
        ↓
Agent receives outdated guidance

Studio Monitor is designed to detect this type of drift.

Monitoring can periodically check:

• Whether /llms.txt still exists
• Whether it remains valid
• Whether the grade changes
• Whether important links have died
• Whether the site's structure has changed
• Whether relevant content appears to be missing
• Whether the file itself changed

The current monitoring service performs weekly checks.

Alerts are intended to tell you when maintenance may be required rather than claiming that a change affected AI rankings.

Crawler fetch logs and canary links

Studio Monitor can optionally place a special resource link in your llms.txt.

This acts as a canary.

If a crawler requests that resource, Studio can record information such as:

• Timestamp
• User-Agent string

This provides evidence that something presenting that User-Agent followed the link.

What a canary can prove

It can show:

A request identifying itself with this User-Agent fetched the canary URL.

What a canary cannot prove

It cannot establish with certainty:

• Who operated the client
• Whether the User-Agent string was genuine
• Why the resource was fetched
• Whether content was used in an answer
• Whether the website was added to an AI index
• Whether a future citation resulted from the fetch

User-Agent strings are self-declared and can be spoofed.

A lack of canary requests also does not prove that no AI system has seen your llms.txt. A system could retrieve or process the file without following that particular link.

Treat fetch logs as evidence, not attribution.

About the llms.txt Studio crawler

Studio identifies its crawler rather than crawling anonymously.

The crawler User-Agent is:

llmstxt-studio/1.0

During generation, Studio may request:

• /robots.txt
• Declared sitemap files
• /sitemap.xml when appropriate
• The homepage
• Up to 60 selected HTML pages

During validation it may request:

• /llms.txt
• /llms-full.txt
• /robots.txt
• A sample of URLs listed inside the file

The crawler:

• Stays on the same registrable domain during page discovery.
• Does not execute JavaScript.
• Does not submit forms.
• Does not intentionally crawl external links during site discovery.
• Limits page requests to protect the target website.
• Honors applicable robots.txt page-crawling directives.

The current crawler limit is at most two requests per second to a host during a job.

How to block the Studio crawler

If you do not want Studio to crawl site pages, add an appropriate rule to your robots.txt.

Example:

User-agent: llmstxt-studio
Disallow: /

Blocking the crawler may prevent the generator from examining the pages required to build a useful file.

The public metadata resources used for crawler discovery and validation may still be checked where appropriate.

Best practices

Keep the file concise

llms.txt is a guide, not a database dump.

Prioritize the pages an agent would genuinely need to understand your organization, product, documentation, policies, or services.

Start with authoritative resources

Prefer primary pages such as:

Documentation
Product pages
Service pages
Pricing
Policies
API references
Support documentation
About information

Use secondary material only where it adds meaningful context.

Write useful descriptions

Poor:

- [Pricing](...): Pricing

Better:

- [Pricing](...): Current plans, billing options, feature limits and subscription terms

Avoid marketing superlatives

Prefer factual descriptions.

Poor:

The world's greatest revolutionary AI platform.

Better:

A platform for monitoring customer reviews and measuring brand visibility in AI-generated answers.

Keep links alive

A beautifully formatted file becomes much less useful when its links return errors.

Validate after:

• Site migrations
• Documentation reorganizations
• Product launches
• Major redesigns
• URL changes
• CMS migrations

Use canonical information

Where possible, link to the authoritative source.

If pricing is defined on /pricing, do not send an agent to a two-year-old article discussing prices.

Keep important resources out of Optional

The Optional section should contain secondary information.

Do not put critical documentation there merely to make the main file look shorter.

Update the file when the site changes

Treat llms.txt as maintained website infrastructure rather than a one-time SEO deliverable.

Common mistakes

Treating llms.txt like a sitemap

Do not automatically put every indexable URL into the file.

A smaller list of high-value resources is usually more useful.

Stuffing keywords into descriptions

llms.txt is not a meta-keyword file.

Descriptions should explain content, not repeat target search terms.

Making unsupported AI visibility claims

Avoid statements such as:

Adding llms.txt guarantees ChatGPT citations.

or:

llms.txt increases your AI ranking.

Those claims cannot be reliably established.

Linking only to the homepage

A file that contains nothing except the homepage gives an agent little additional guidance.

Link directly to useful resources.

Copying the sitemap

A sitemap may contain pages that are irrelevant to understanding the site.

Curate rather than copy.

Using vague link titles

Avoid:

- [Learn more](...)
- [Click here](...)
- [Page](...)

Use:

- [API authentication reference](...)
- [Enterprise pricing](...)
- [Returns and refund policy](...)

Publishing HTML at /llms.txt

If /llms.txt redirects to or returns a normal HTML webpage, automated clients may not receive the expected resource.

Check the live response after deployment.

Forgetting maintenance

URLs change.

Products disappear.

Documentation moves.

A stale llms.txt can become misleading even if its syntax remains valid.

Troubleshooting

My validator says /llms.txt is missing

Confirm that this URL is publicly accessible:

https://yourdomain.com/llms.txt

Common causes include:

• File was uploaded to the wrong folder.
• Framework routing intercepts the request.
• CDN has not updated.
• Deployment excluded .txt files.
• File requires authentication.

/llms.txt returns my homepage

Your web server may be configured to route unknown paths to the application's HTML entry point.

Create an explicit route or static file for:

/llms.txt

Then test the URL again.

My H1 is missing

An H1 uses one #:

# Example Company

Do not start the file with only:

## Example Company

My summary is missing

Use a Markdown blockquote beneath the H1:

# Example Company

> Example Company provides inventory management software for independent retailers.

My links are being reported as dead

Check the destination directly.

A dead-link result may be caused by:

• HTTP 404
• HTTP 410
• Server error
• Connection failure
• Bot protection
• Authentication requirements
• Firewall rules

A redirect that successfully reaches a live destination is normally less serious than a URL that fails completely.

Why is an external link being flagged?

A curated llms.txt will generally be easier to trust and maintain when most resources point to the domain it describes.

External resources are not inherently forbidden by the llmstxt.org proposal, and sometimes they are useful.

Studio may warn about them so you can verify that the external dependency is intentional.

The generator missed an important page

Possible causes include:

• Page is absent from the sitemap.
• Page is difficult to discover.
• robots.txt prevents crawling it.
• Important content requires JavaScript.
• Page metadata is weak.
• Crawl limit was reached.
• Page appears less important than competing pages.

You can edit the generated file manually before publishing it.

The generator is a starting point, not the final authority on your website.

The generator description is weak

Studio relies on information your website already provides rather than asking an AI model to invent marketing copy.

Review the affected page's:

<title>
meta description
H1
page content

Improving those elements can make the source page clearer for humans, search engines, crawlers, and future generated files.

Frequently asked questions

Is llms.txt an official internet standard?

llms.txt is a proposal and convention, not a universal requirement that every AI provider has committed to support.

Adoption does not mean every crawler or assistant will use it.

Is llms.txt the same as robots.txt?

No.

robots.txt communicates crawling rules.

llms.txt provides context and useful content destinations.

They complement rather than replace each other.

Is llms.txt the same as sitemap.xml?

No.

A sitemap is generally an inventory of discoverable URLs.

llms.txt is a curated guide to resources that are particularly useful for understanding the site.

Does every website need llms.txt?

No.

It may be especially useful for sites with substantial structured information, including:

• Documentation sites
• Software products
• APIs
• SaaS platforms
• Ecommerce sites
• Service businesses
• Educational sites
• Knowledge bases
• Organizations with detailed policies or reference material

Whether it provides value depends on your website and how agents interact with its information.

Can I create llms.txt manually?

Yes.

The format is intentionally simple enough to write manually.

A generator mainly saves time discovering, organizing, and checking suitable pages.

Does llms.txt need XML?

No.

The file uses Markdown-style text rather than XML.

Should I include every page?

Usually not.

Think:

Which resources would I give an intelligent assistant if it had only a few minutes to understand this site?

Those are strong candidates for llms.txt.

How many links should I include?

There is no universal number that fits every site.

The objective is useful curation.

Studio warns when a file exceeds 200 links because extremely large files begin to work against the purpose of providing a concise guide.

A small company may need only a few dozen resources.

A large documentation platform may need more.

Should blog posts be included?

Include blog posts when they provide authoritative information that helps explain important topics.

Avoid automatically listing an entire blog archive.

Evergreen guides are usually stronger candidates than:

• News announcements
• Short-lived promotions
• Thin posts
• Duplicate topics

Should pricing be included?

If pricing is public and relevant to understanding your product or service, the current pricing page is usually a useful resource.

Link to the canonical current page rather than copying potentially changing prices into the description.

Can llms.txt contain external links?

Yes, when they genuinely help explain the website or project.

Examples might include:

• Official source repositories
• External documentation platforms
• Standards documentation
• Authoritative dependencies

Use external links deliberately.

Can llms.txt contain instructions for AI agents?

It can contain explanatory context about how to interpret the site's resources.

Keep those instructions factual and relevant.

Do not assume that arbitrary instructions inside an llms.txt file will override an agent's own rules or system instructions.

Should the file contain my entire website copy?

No.

That defeats the main purpose of a concise llms.txt.

Keep the primary file focused on orientation and resource discovery.

Is llms-full.txt required?

No.

It is optional.

The primary resource is /llms.txt.

Will ChatGPT read my llms.txt?

It may or may not.

Publishing a file makes it available to clients that choose to use it. It does not force any particular assistant to retrieve it.

Will Perplexity read my llms.txt?

The same principle applies.

Availability does not guarantee retrieval or use by any specific service.

Will Gemini read my llms.txt?

Again, support and retrieval behavior are controlled by the individual system.

Do not equate publishing the file with guaranteed ingestion.

Can I see which AI crawler read my file?

Normal server logs may contain request information including User-Agent strings.

Studio Monitor can additionally use a canary resource to record requests that follow a unique link.

Remember that User-Agent strings can be spoofed.

Treat them as evidence of the reported client identity, not cryptographic proof.

Does blocking an AI crawler in robots.txt make llms.txt useless?

That depends on the client and your objective.

robots.txt and llms.txt communicate different things.

If you block a crawler from content it needs, you should not assume that publishing an llms.txt file gives that crawler permission to ignore your access rules.

How often should llms.txt be updated?

Update it whenever meaningful changes make the current file inaccurate.

Typical triggers include:

• New products
• New documentation
• Removed services
• URL migrations
• Pricing-page changes
• Restructured help centers
• Rebranding
• Major policy changes

Stable websites may require few changes.

Frequently changing websites may benefit from automated monitoring.

What grade should I aim for?

Aim for a file with:

• Valid structure
• A clear title
• A useful summary
• Logical sections
• Descriptive links
• Live URLs
• No unnecessary duplication

A high validation grade means the file satisfies the validator's quality checks.

It does not mean the site will receive more AI citations.

Glossary

AI agent

Software that uses an AI model to perform tasks and may retrieve external information as part of those tasks.

AI crawler

An automated client that requests pages or resources from websites for purposes related to an AI system.

Different crawlers may serve different purposes, including indexing, retrieval, training, or user-requested browsing.

llms.txt

A Markdown-formatted file that gives agents concise context about a website and links to important resources.

llms-full.txt

An optional expanded text resource containing substantially more page content than a standard llms.txt.

It should not be confused with the core llms.txt proposal.

robots.txt

A conventional website file containing rules intended for automated crawlers.

Normally available at:

/robots.txt

sitemap.xml

An XML resource containing URLs a website wants search engines or other clients to discover.

Normally available at a location such as:

/sitemap.xml

Markdown

A lightweight text-formatting syntax using characters such as:

# Heading

## Subheading

- List item

[Link](URL)

> Blockquote

llms.txt uses Markdown because it is easy for humans, language models, and conventional software to process.

H1

A top-level Markdown heading:

# Website Name

The llmstxt.org format requires an H1.

H2

A second-level Markdown heading:

## Documentation

H2 headings are commonly used to organize linked resources in llms.txt.

Blockquote

Markdown text beginning with >.

In an llms.txt file, it is commonly used for the concise site summary.

Example:

> Example Corp builds accounting software for independent businesses.

User-Agent

A text identifier sent by an HTTP client when requesting a website resource.

Example conceptually:

User-Agent: ExampleCrawler/1.0

User-Agent values are self-declared and should not be treated as verified identity.

Canary link

A unique URL placed in a resource primarily to detect whether a client follows it.

A request provides evidence that someone or something accessed the URL, but it does not prove the real-world identity of the requester.

Crawlability

Whether automated clients can access and retrieve website resources.

Crawlability can be affected by:

• robots.txt
• Authentication
• Firewalls
• Bot protection
• HTTP errors
• Server configuration

Context window

The amount of information an AI model can work with during a particular interaction.

One reason for keeping llms.txt concise is to avoid spending context on hundreds of irrelevant pages.

Recommended workflow

A practical workflow for maintaining llms.txt is:

1. Inspect website
        ↓
2. Generate llms.txt
        ↓
3. Review important resources
        ↓
4. Validate structure
        ↓
5. Publish at /llms.txt
        ↓
6. Validate live URL
        ↓
7. Update when website changes
        ↓
8. Monitor for broken links and drift

The objective throughout the process is the same:

Give an agent a small, accurate and useful map of your website without pretending that the file controls whether the agent ultimately uses, ranks or cites your content.