llms.txt, robots.txt and JSON-LD generator
Four fields in, four shippable files out: llms.txt, llms-full.txt, the correct AI-crawler block for robots.txt, and JSON-LD. Every output is copy-ready, and every one has a one-line command below to prove it went live.
Your files
Upload this to the root of your domain so it resolves at /llms.txt. Most hosts: drop it next to index.html.
llms-full.txt holds the actual page text, not just links. We can't read your pages from here, so this is the skeleton with slots marked — paste each page's opening paragraphs into its slot.
If you already have a robots.txt, append the AI-crawler block rather than replacing the file. Keep any existing Sitemap: line.
Paste this before </head> on your home page. It declares your organisation, your site, and the site-wide search action — the three nodes AI assistants look for first.
What each file answers, and how to prove it went live
Each file maps to one row in the readability check. Upload it, run the command under it, and you know whether it took — no dashboard, no waiting for a report to change colour.
- llms.txt answers “which of my pages should an AI read first”. Check it with
curl -s https://yoursite.com/llms.txt | head -1— you should get a #, not <!DOCTYPE. The usual mistake is leaving a single-page-app fallback at that path: it returns 200 with HTML in it, the check calls it a red, and the check is right. - llms-full.txt carries the same list with the page text attached, so the assistant does not have to fetch each page again. Check the size with
curl -s https://yoursite.com/llms-full.txt | wc -c. It should be several times llms.txt. If the two are about the same size, the text did not make it in. - The robots.txt block answers “can a retrieval crawler get in at all”. Check it with
curl -s https://yoursite.com/robots.txt | grep -A2 OAI-SearchBotand you should see Allow: /. The usual mistake is replacing the whole file instead of appending — that drops your Sitemap line and every rule you already had. - JSON-LD answers “can a machine tell who you are without guessing”. Check with
curl -s https://yoursite.com/ | grep -c 'application/ld+json'— anything above zero is present. For more than presence, paste the URL into Google's Rich Results Test; it names the property that is missing.
Only one of the four has cleared our significance bar. JSON-LD sits at ρ 0.391 against a 0.285 threshold; llms.txt is 0.168, under it. We generate llms.txt anyway because it is cheap and cannot hurt you, but we are not going to tell you it moves citations. The robots.txt block is not a bonus at all — it is the thing that stops you being locked out of the pool before anything else gets a chance to matter.
Questions people ask about these four files
Which AI crawlers should I allow?
The retrieval ones: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-SearchBot, Claude-User. These decide what enters the answer pool, so blocking them removes you from consideration entirely. GPTBot, ClaudeBot and CCBot are training crawlers — a different matter, and blocking those costs you nothing.
Is llms.txt an official standard?
No. It is a proposed convention with uneven adoption. It is cheap to add and cannot hurt you, which is why we generate it — but we do not claim it is a ranking factor, and we would be lying if we said the evidence was strong.
Does any of this actually move citations?
Two of the four have a citation-rate comparison behind them. Pages carrying JSON-LD were cited 36% of the time against 10% for pages without it, and pages that allowed retrieval crawlers were cited 32% against 7%. A third, llms.txt, we have a correlation for but no comparison: 0.168, under the 0.285 our sample needs. All of it comes from one 48-source experiment, so it is correlation, not causal proof — the full caveat list is on the methodology page.
How long until the files take effect?
The files themselves are live the moment you upload them — that is what the curl commands above check. Whether anything reads them is a different question. A crawler picks up robots.txt on its next visit and there is no published schedule for when that is. We are not going to promise 24 hours, because that is not ours to promise.
Can I ship just one of them?
Yes, and if you are picking one, pick JSON-LD. It is the only file here that clears the significance threshold (0.391 against 0.285). The robots.txt block is second in importance for a different reason: it is not an improvement, it is a prerequisite — a blocked retrieval crawler never sees any of the other three.
Will this overwrite my existing robots.txt?
No, and it should not. The generator gives you a block to append, not a file to replace. Replacing it wholesale is the most common way to break this: you lose your Sitemap line, your crawl-delay rules and anything else already in there, and nothing warns you.
Done with the easy part?
The files that go in your root
- llms.txt and llms-full.txt
- The AI-crawler block for robots.txt
- Site-wide JSON-LD
The parts a generator cannot write
- Full fix list, ranked by how hard each one is
- Per-page JSON-LD on every template
- FAQ and summary blocks for your key pages
- Step-by-step upload instructions
$199 one-time · not open yet · you upload it yourself