Real human. Fast replies.
💬Text me(386) 569-1227
📞Call me(386) 569-1227
✉️Email mechenzo@honesty.im
📝Send a messageI reply same day
Somewhere on your website there’s a plain text file called robots.txt, and odds are nobody has opened it since the day the site launched. Lately it’s become the place where business owners go to “stop AI from stealing my stuff,” usually by pasting in a few lines they found in a forum thread.
This week Google put out a short video that politely explains why a lot of those lines do absolutely nothing. I watched it so you don’t have to (you should, though, it’s short).
The short version
Google said robots.txt is about access, nothing more. In the October 21 episode of its Search Central series, What can you actually put in a robots.txt?, Gary Illyes and Cherry Prommawin walk through it: a disallow rule means “don’t come in here.” It doesn’t dictate what a system does with data it already crawled.
That sounds like a technicality. It isn’t. It’s the difference between a “Please Knock” sign and a deadbolt, and plenty of people have been treating the sign like a deadbolt.
They also named the only instructions the file understands: user-agent, allow, disallow and sitemap. Everything else is ignored without a peep. No error, no warning, no little red flag. Just a line of text sitting there feeling important.
No. If you wrote something like “NoAI: true” or “AI-training: disallow” into your robots.txt, the parser skipped right over it. It’s not in the rulebook, so it doesn’t exist.
I see this all the time: a site with a robots.txt that reads like a strongly worded letter to the internet. Custom rules, comments shouting in all caps, a line that says “This means you, robots.” Very satisfying to write. Zero effect.
If you want to keep a specific AI crawler out, you have to name that crawler’s user-agent and give it a disallow. That’s it. And remember the access point from above: that only stops future visits. Whatever got collected before you added the rule is already gone, and robots.txt can’t call it back.
Google also has a separate token called Google-Extended that controls whether your content can be used for training and grounding its Gemini models. That one’s a use preference, not a crawling rule, which is exactly why it lives in its own lane. Blocking Googlebot to “stop Google AI” is like cancelling your mail to stop junk flyers. Technically works. Also kills your search traffic.
For most small businesses, probably not, or at least not all of them. Here’s the part that gets lost in the panic: the same AI tools people want to block are the ones customers now ask “who’s a good plumber near me?”
If you slam the door on every AI bot, you might be fine with that. Just know you’re also opting out of a lot of the places people find businesses now. A bakery that blocks everything and then wonders why ChatGPT recommends the bakery across town has kind of done this to itself.
My take: a local service business wants to be found, quoted and recommended. Your hours, services, prices and reviews are not trade secrets. They’re the whole point. Protect the stuff that’s actually yours to protect, like paid courses, member areas or original research you sell, and let the rest work for you.
If you want to go the other direction and help AI tools understand you, that’s a different file entirely. I wrote about it here: llms.txt for small business.
Type your domain followed by /robots.txt into a browser. That’s the whole trick. If it loads, read it like a stranger would.
Things I’d look for, roughly in order of how much damage they cause:
One more thing, straight from Google’s own robots.txt documentation: blocking a page in robots.txt isn’t a reliable way to keep it out of search results. If a page shouldn’t show up in Google, it needs a noindex tag or a password, not a disallow line and a prayer.
It can stop OpenAI’s crawlers from visiting pages going forward if you name them correctly. It doesn’t erase anything already collected, and it doesn’t stop other people from quoting or linking to you.
Yes, and it’s one of the most common half-assed launch mistakes I clean up. A leftover “Disallow: /” can keep Google from crawling your pages at all, so new content and updates never get picked up.
No file means crawlers assume they can visit everything, which is fine for a lot of small sites. A short, clean one with a sitemap line is still nice to have. Short and clean is the goal. Nobody gives out awards for the longest robots.txt.
Robots.txt is a doormat with instructions written on it. Polite bots read it. It was never a lock, and Google just said so out loud. The damn thing does one job, and it does it fine, as long as you don’t ask it to do six others.
So decide what you want first: found by AI, hidden from it, or somewhere in between. Then set up the file, your pages and your AI search setup to match. If you’d rather not spend a Saturday reading crawler documentation, I get it. Tell me what you’re working with and I’ll take a look.
