Can AI fetch it?
robots.txt plus OpenAI, Claude, and Perplexity bots, training, search, and live answers.
35/35
robots.txt
PassA real robots.txt is live, crawlers have an explicit map.
Site is crawlable
PassUser-agent * can fetch the homepage.
OpenAI bots (training, search, live fetch)
PassGPTBot allowed · OAI-SearchBot allowed · ChatGPT-User allowed
Claude & Perplexity
PassClaudeBot allowed · PerplexityBot allowed