Let the Machines Read You (Except the Books)
I opened my CDN’s AI dashboard for the first time recently and found a number I could not explain. In seven days, AI crawlers had made about 1,500 requests to my author site, and 793 of them had received an HTTP 403, the web’s way of saying no.
My robots.txt said everyone was welcome. It always had. Something between the crawlers and my pages had been refusing them on its own authority, and the crawler that fires when a reader asks an AI about me directly was among the ones refused.
If you have been following the zero-click shelf argument, this is the plumbing underneath it. An answer engine can only recommend a book it was allowed to read.
Two Kinds of Crawler
AI crawlers look alike in a server log and do opposite jobs.
Answer and search crawlers fetch your page live, usually because a reader just asked a question. OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User, Applebot, Bingbot and Googlebot belong here. They quote you, name you, and often link to you. They are the shelf a reader browses now.
Training crawlers read your pages in bulk so a future model can learn from them. GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended and Meta-ExternalAgent belong here. They send nobody to your site. What they give you is memory: a model that has read your blog may know your name when someone asks it about dark fantasy.
These are two separate decisions. Plenty of authors block training and welcome answers, and the web gives you the controls to do exactly that.
My Call
I let both read the site at large. The blog, the codex, the home page and the page about the game are open to everyone, training crawlers included, because a model that knows the name Alejandro Ashes is worth more to a solo author than a closed door.
The books and the games are a different case. The text of the novels, the playable games and their code, the music and the card art are the work itself, and I ask training crawlers to stay out of them.
Your line may land somewhere else. The point is to draw it on purpose, because the default is whatever your host or plugin picked for you.
The Whole File
Here is the core of my robots.txt:
User-agent: *
Content-Signal: search=yes, ai-input=yes
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
(and the rest of the training crawlers)
Allow: /games
Disallow: /read/
Disallow: /game
Disallow: /music/
Disallow: /audio/The first group is for everyone. The Content-Signal line is a newer, machine-readable way to state what you allow: search results, AI answers, and (when you add it) training.
The second group names the training crawlers one by one. One detail cost me a test run. Disallow: /game also matches /games, my page about the game, because rules match by prefix. A single Allow: /games line wins that conflict, since the longest matching rule takes priority.
The Trap Under the File
robots.txt is the part you can read. The part that surprised me lived in the dashboard of my CDN, which sits in front of the site and has its own AI policy.
Two settings mattered. The first was a policy that blocked “training” crawlers. It sorted the crawler that serves a reader’s live question, Claude-User, into the same bucket as the bulk trainers, and refused it. Answer traffic got blocked by a rule written for training. The second was a sync option that writes the CDN’s own rules above yours in robots.txt, which can contradict the file you wrote.
The fix took five minutes once I knew where to look. The crawler list shows a switch per bot (on means blocked), so I turned every one off. The policy dialog set training to allow, and I turned the sync off so my file would be the only rule. Then I opened my own address, /robots.txt, in a browser. It showed exactly my file and nothing above it.
That last step is the one to copy. Your file, the host’s policy and any plugin can each speak for your site. Read what the public sees.
What the File Cannot Do
robots.txt is a request. Well-behaved crawlers honor it, and the big AI companies publish their crawler names and say they do. A scraper that ignores it can still copy any page you serve to the public.
So treat the file as a signal to the honest majority. What must stay private belongs behind something stronger than a request: a login, a paywall, or simply a place the public web does not reach. Your real protection for a book is copyright and where the full text lives.
What I Expect, and What I Do Not Know Yet
I changed these settings only just now, so I have no results to report. In that same seven-day window Perplexity’s crawler made 273 requests and sent me zero visitors, a good reminder that a crawl proves nothing about a citation, and a citation proves nothing about a click. I expect the refusals to drop to near zero within days. Whether any of it turns into readers is something I will report in a month, with numbers.
Your Checklist
- Decide on answer crawlers and training crawlers separately.
- Write one robots.txt with both, and name the crawlers you mean.
- Open your host’s or CDN’s AI settings and look for any policy that blocks by category, plus any option that edits robots.txt for you.
- Read your live
/robots.txtin a browser. - Check your server’s refusal count a week later.
Most authors will never look at any of this, and the machines will keep deciding about them anyway. Ten minutes in a dashboard puts the decision back in your hands.
