Short answer: Give every AI crawler its own User-agent group in robots.txt and write Allow and Disallow rules under that group. A healthy setup keeps search and citation bots (OAI-SearchBot, PerplexityBot, Googlebot, Bingbot) open and limits training bots such as GPTBot or ClaudeBot according to your content policy.
- A crawler that finds a group with its own name ignores the general “*” group, so shared restrictions must be repeated in every bot-specific group.
- The longest matching path wins when Allow and Disallow overlap.
- Google-Extended and Applebot-Extended are control tokens, not separate crawlers; blocking Googlebot removes a site from Google Search, AI Overviews and AI Mode.
- CDN or firewall settings can block AI bots even when robots.txt allows them.
robots.txt is a request, not a security measure, and it will not hide private pages. Test the file after every change and review it whenever platforms announce new crawler names. Racoun manages 14 AI crawlers as one of the seven components of its GEO service.
Talk to Racoun: we are based in Istanbul and work with brands across Türkiye. Message us on WhatsApp or email ekip@racoun.com.
