AI bot index · #1426 most visited · Checked 2026-11-17
Does toyhou.se block ChatGPT, Claude and Perplexity?
Only for training. toyhou.se blocks GPTBot and ClaudeBot (model training) but lets ChatGPT, Perplexity and Claude search read its home page.
Crawler by crawler
- Google Search Googlebotallowed
- Bing (also feeds ChatGPT search and Copilot) Bingbotallowed
- ChatGPT search OAI-SearchBotallowed
- ChatGPT reading a link ChatGPT-Userallowed
- Perplexity PerplexityBotallowed
- Claude search Claude-SearchBotallowed
- Gemini (Google AI) Google-Extendedblocked
- OpenAI model training GPTBot training onlyblocked
- Anthropic model training ClaudeBot training onlyblocked
From toyhou.se/robots.txt, home page path. Re-check it live
History
- 2026-10-30started blocking Google-Extended, GPTBot, ClaudeBot
Same as shown above since 2026-10-30. We re-check it every few days.
We read each site's robots.txt as an ordinary visitor sees it and test the home page path (/) for each crawler, using the same rules Google documents (most specific group, longest matching rule). Some sites serve a different robots.txt to verified crawlers, and a site can allow the home page but block other sections, so treat this as what the public file says, not a guarantee.
Now check your own site
Free, no sign-up. See which AI crawlers can read it, and the 3 things to fix first.
Nearby in the index
jino.ru · blic.rs · jutarnji.hr · aljazeera.com · 24h.com.vn · sozcu.com.tr · index.hr · 3bmeteo.com