AI crawlers are the exploration robots of artificial intelligence companies: they roam the web, read pages and feed assistant answers. Each identifies itself with a recognisable agent name: GPTBot and OAI-SearchBot for OpenAI (ChatGPT), ClaudeBot for Anthropic (Claude), Google-Extended for training Gemini models, PerplexityBot for Perplexity, plus Bingbot which feeds Copilot. On top of these come "on-demand" readers that visit a page at the exact moment a user asks their assistant a question.
Their access is controlled in the robots.txt file, the web's historic gatekeeper: each robot can be allowed or blocked individually. And this is where a strategic choice arises. For a publisher that lives off its content, blocking AI crawlers can be a defensible position. For a local business, it is almost always a mistake: a blocked robot is an assistant that cannot recommend you. If ChatGPT cannot read your menu, it will suggest the restaurant across the street. AI visibility starts with an open door.
Opening the door is not enough — the visit also has to be worthwhile. AI crawlers have short exploration budgets and often limited JavaScript execution: a slow site, or one whose content only appears after heavy scripts run, partly escapes them. That is a strong technical argument for static websites — all the content is in the page from the first read, no detours. Speed, clean HTML, structured data and llms.txt form the complete welcome kit.
Worth knowing: server logs show which robots visit your site, and let you distinguish training exploration (the model is learning, occasional visits) from live reading (a customer is looking for you, right now). On the sites I manage, AI crawler access is verified and documented — it is the "technical foundations" step of the AI search method.
Common mistakes
- Blocking them by default : disallowing GPTBot or ClaudeBot "just in case" removes any chance of being cited by ChatGPT or Claude.
- Confusing crawlers with malicious scrapers : official AI bots respect robots.txt; blocking them protects nothing useful for a local business.
- Serving JavaScript-only content : many AI crawlers read script-injected content poorly. The essentials must live in the HTML.