Guide: Is our website readable by AI?
The seven reasons an assistant cannot read a website, how to check each, and the fix.
The short answer
In two paragraphs.
Often it is not, and the business does not know. Before an assistant can recommend a website it has to open it and read words on it. Our question map opens every website it is given the way an AI crawler does, and the same seven reasons come up again and again: the site keeps every robot out, it turns the reader away, its pages hold no words until a browser runs a script, it does not answer, it answers too slowly, it blocks the AI companies' crawlers by name, or it opens fine and never says what the business sells.
Each one has a check you can do yourself and a fix that is usually small. The quickest check is the question map on this site: type your website and the first thing it tells you is whether it could be read, and why not.
The seven reasons
What stops the reading, how to see it, and what to change.
| Reason | How to check | The fix |
|---|---|---|
| The rules keep every robot out | Open yoursite.com/robots.txt. A line Disallow: / under User-agent: * closes the whole site | Open the site to robots, and close only the pages that should be private |
| The site turns the reader away | A security service or a firewall answers robots with an error page (403) or a challenge | Allow the search and AI crawlers through the security service's settings |
| The pages hold no words until a script runs | View the page's source: if the text you see on screen is missing there, a script builds it | Render the words on the server, so the page arrives with its text |
| The site does not answer | The address fails, or a certificate error stops the page | Fix the hosting or the certificate; test from outside your network |
| The site answers too slowly | The home page takes more than five seconds to arrive. Crawlers give up long before a person does | Smaller pages, fewer scripts, a faster host |
| The AI crawlers are blocked by name | robots.txt names GPTBot, ClaudeBot, PerplexityBot, Google-Extended or the others with Disallow | Decide which to let in. The crawlers that answer a person's question are the ones that recommend you |
| The home page does not say what you sell | Read the home page as a stranger. If it does not say what is sold, for whom and where, in words, the assistant cannot either | Write it on the home page, in plain words, with the places |
The crawlers
The AI companies' robots, by name.
Two kinds of AI crawler read websites. One answers a person's question at that moment, and it is the one whose reading becomes a recommendation. The other gathers text to train a model. A site can let in one kind and keep out the other, by name, in its robots.txt. Blocking the first kind blocks the recommendations.
| Company | Answers a person's question | Gathers text for training |
|---|---|---|
| OpenAI (ChatGPT) | OAI-SearchBot, ChatGPT-User | GPTBot |
| Anthropic (Claude) | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | |
| Google (Gemini, AI Overviews) | Googlebot, as for search | Google-Extended |
| Apple | Applebot | Applebot-Extended |
| Common Crawl, used by many | CCBot |
This site lets all of them in, and says so in its own robots.txt. Which to let in is a decision for each business; the point is to make it knowingly.
No website
A business on Instagram alone has nothing an assistant can read.
The assistants recommend what they can read, and an Instagram or Facebook page is one they cannot open the way they open a website. Our question map lets a business in by its name when it has no website, and shows what its customers ask; the way to being recommended, for that business, is a website made for AI search, which is work we do.
Does a slow website really stop AI from reading it?
Yes. A crawler waits a few seconds and moves on. Our reader gives a home page fifteen seconds on a second try; a page that takes longer is recorded as unread, and the business is told the seconds we measured.
Should we block GPTBot and the training crawlers?
That is a choice about your text being used for training, and businesses decide it either way. Keep the two kinds apart: blocking OAI-SearchBot, ChatGPT-User, PerplexityBot or Claude-SearchBot blocks the crawlers that answer customers' questions, which is where recommendations come from.
Our site is built with a JavaScript framework. Is it unreadable?
Only if the words arrive by script alone. Most frameworks can render pages on the server, so the text is in the page when it is fetched. View the source of a page: if the sentences you see on screen are there, the assistant sees them too.
How do we check all seven at once?
Type your website into the question map on this site. It opens the site the way a crawler does, reads its robots.txt, measures how long the home page takes, reads what the home page says you sell, and tells you the first thing it found. Senpai.ai fixes the basics with you, and then the rest.
The question map
What are your customers asking AI?
They ask ChatGPT what to buy and who to trust before they ever reach you. Type your website and see the questions they ask, and who AI names instead of you.
People ask AI things like
- Where can I try on running shoes in Dubai today?
- Which running shoes stay cool in very hot weather?
- How do I know my running shoe size?
Find what's worth changing next.
Bring the market, the product or the question. We'll start there.
Or start with the question map: see what your customers ask AI.