About our bot — llms.txt Studio

If you saw this user-agent in your logs, someone ran your domain through llms.txt Studio. Here is exactly what it did.

What it fetches. On a generate: /robots.txt, the sitemap(s) it declares (or /sitemap.xml), the home page, then up to 60 HTML pages on the same registrable domain, chosen by family (docs, API, product, pricing, blog, about). On a validate: /llms.txt, /llms-full.txt (HEAD), /robots.txt, and a HEAD or GET on up to 20 links listed in the file. On a weekly monitor check: the same as a validate, plus a 15-page discovery pass that reads only the sitemap and the home page.

Limits, per job. 60 pages, 2 MB per response, 8 second timeout, at most 2 requests per second to any one host, 3 minutes wall-clock, 5 redirects. It does not execute JavaScript, submit forms, or follow links to other domains.

robots.txt. Honoured for page crawling. Rules for llmstxt-studio apply first, then *. Fetching /llms.txt, /robots.txt and sitemaps is always allowed, since those files exist to be read by agents.

Who triggers it. A person typing a domain into the free tool (limited to one generate and one validate per day per address), or a paying customer's weekly monitor for a domain they have proved they control via DNS TXT or a meta tag.

What it stores. Page titles, meta descriptions, a link list, and, only when the customer asked for llms-full.txt, extracted page text. Nothing is used to train anything. Results are reachable by an unlisted link only.

To block it, add this to your robots.txt. The generator will then produce a file with your home page only, and the validator will still read /llms.txt.

Questions or abuse reports: [email protected]. Include the timestamp and your domain.