AI bot index · #1711 most visited · Checked 2026-09-28
Does usnews.com block ChatGPT, Claude and Perplexity?
Yes. usnews.com's robots.txt blocks OAI-SearchBot, ChatGPT-User, PerplexityBot, so ChatGPT and Perplexity cannot read its home page, and it also blocks AI training.
Crawler by crawler
- Google Search Googlebotallowed
- Bing (also feeds ChatGPT search and Copilot) Bingbotallowed
- ChatGPT search OAI-SearchBotblocked
- ChatGPT reading a link ChatGPT-Userblocked
- Perplexity PerplexityBotblocked
- Claude search Claude-SearchBotallowed
- Gemini (Google AI) Google-Extendedblocked
- OpenAI model training GPTBot training onlyblocked
- Anthropic model training ClaudeBot training onlyblocked
From usnews.com/robots.txt, home page path. Re-check it live
History
Same as shown above since 2026-09-28. We re-check it every few days.
We read each site's robots.txt as an ordinary visitor sees it and test the home page path (/) for each crawler, using the same rules Google documents (most specific group, longest matching rule). Some sites serve a different robots.txt to verified crawlers, and a site can allow the home page but block other sections, so treat this as what the public file says, not a guarantee.
Now check your own site
Free, no sign-up. See which AI crawlers can read it, and the 3 things to fix first.
Nearby in the index
xsmn.mobi · choiceadvantage.com · fabguys.com · minhngoc.net.vn · thscore99.com · xoso.com.vn · packzy.com · mobiledokan.com