AI search. Reviewed October 29, 2026
AI crawler
Also called: GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot
An AI crawler is a bot such as GPTBot, ClaudeBot or PerplexityBot that reads web pages so an AI assistant can learn from them or quote them, and a site allows or blocks each one in its robots.txt file.
In practice
Some hosting platforms and security services block these bots by default, which keeps the site out of AI answers entirely. Check robots.txt and the firewall rules before assuming the assistants can read the site.
A local business has no reason to block them. The downside of being quoted is small and the downside of being invisible is the phone not ringing.
Related terms
- llms.txtllms.txt is a plain text file at the root of a website, like robots.txt, that gives AI crawlers a summary of the business and a list of its important pages with one-line descriptions.
- AI searchAI search is any search where an assistant such as ChatGPT, Perplexity, Gemini or Google's AI Mode writes the answer and names a few businesses or sources, instead of returning ten links.
- Server-side renderingServer-side rendering means the server sends a finished HTML page instead of a blank shell that JavaScript fills in later, so search engines and AI crawlers see the content on the first request.
Go deeper
Want this handled, not explained?
Skill Mammoth builds the site you own, gets you found on Google and ChatGPT, and texts every lead back in under 60 seconds. Month to month, every price published.