# AI crawler check

Free tool: see whether ChatGPT, Claude, Perplexity and Google's crawlers can read your site. robots.txt, text without JavaScript, noindex, sitemap.

By Dasha. Updated November 1 2026. https://heardofyou.io/tools/ai-crawler-check

Paste your website and see in a few seconds whether AI assistants can read it at all. Does robots.txt block their crawlers? Does your homepage return real text without JavaScript? Did you accidentally tell engines not to index or quote you? Is there a sitemap? Free, nothing stored, no sign-up.

<form id="crawler-form" class="pane flow-form cc-form" novalidate>
  <label for="cc-url">Your website</label>
  <div class="cc-row">
    <input id="cc-url" type="text" name="url" placeholder="example.com" autocomplete="url" inputmode="url" maxlength="200" required>
    <button class="btn btn-primary" type="submit">Check</button>
  </div>
  <p class="form-msg" aria-live="polite"></p>
  <p class="form-note">Reads robots.txt and your homepage as a crawler would. Nothing is stored.</p>
</form>

<div id="crawler-results" class="cc-results" aria-live="polite" hidden></div>

This checks robots.txt and the first HTML response only. It does not run JavaScript. It cannot see firewalls that block crawlers by IP address rather than by name. A green report means the rules let crawlers in, not that every crawler gets through. For that, read the server logs. Or ask Perplexity a buyer question and see whether your page appears in its sources.

Readable is the floor, not the ceiling. Passing every check means an assistant can read you. It does not mean it will recommend you. That depends mostly on what other pages say about you. Pass and still missing? Read [why AI doesn't recommend your product](/guides/why-ai-doesnt-recommend-you).

## What it checks

### 1. robots.txt rules for AI crawlers

The checker fetches `yoursite.com/robots.txt`. For each named crawler, it reports whether your homepage and a sample deep page (`/pricing`) are allowed or blocked. Rules match the way the crawlers match them: named groups beat the `*` group, the longest matching path wins, `*` and `$` work as wildcards.

| Crawler | Who | What it feeds |
|---|---|---|
| `OAI-SearchBot` | OpenAI | ChatGPT's live search answers and citations |
| `ChatGPT-User` | OpenAI | Fetches a page when a ChatGPT user asks about it |
| `GPTBot` | OpenAI | Training of future OpenAI models |
| `Claude-SearchBot` | Anthropic | Claude's live search answers |
| `Claude-User` | Anthropic | Fetches a page when a Claude user asks about it |
| `ClaudeBot`, `anthropic-ai` | Anthropic | Training of future Claude models |
| `PerplexityBot` | Perplexity | Perplexity's search index and citations |
| `Perplexity-User` | Perplexity | Fetches a page when a Perplexity user asks about it |
| `Googlebot` | Google | Google Search, AI Overviews and AI Mode |
| `Google-Extended` | Google | Training of Gemini models |
| `Bingbot` | Microsoft | Bing Search and Copilot |
| `CCBot` | Common Crawl | A public web archive many models are trained on |
| `Applebot-Extended` | Apple | Training of Apple models |
| `Bytespider` | ByteDance | ByteDance's crawler |

Search and training are separate decisions. OpenAI, Anthropic and Perplexity each run separate crawlers for them. So you can allow live search and block training if you want.

What you cannot do is block the search crawler and still appear in that assistant's live answers. Brands that block Claude's crawler get almost no Claude citations (Muck Rack). We allow all of them on heardofyou.io, on purpose, and say so in our own [robots.txt](/robots.txt).

### 2. Readable text without JavaScript

The checker fetches your homepage the way a crawler does, with no JavaScript, and counts the words of real text that come back. If your product name, what it does and who it is for only appear after a script runs, many crawlers see an empty shell.

It also flags a bot challenge, consent screen or firewall page served instead of your content. That is a common silent failure on sites behind aggressive bot protection. And it checks whether the raw HTML carries a title tag and a meta description.

### 3. noindex and nosnippet

A `noindex` robots tag or `X-Robots-Tag: noindex` header tells search engines not to index a page. Google will not use it in AI Overviews or AI Mode either. `nosnippet` and `max-snippet:0` limit what can be quoted from a page, which cuts exactly what an assistant needs.

These get left on by accident when a site moves from staging. The checker reads the homepage's HTML and headers and flags any of them.

### 4. A sitemap

The checker looks for a `Sitemap:` line in robots.txt and for `/sitemap.xml`. A sitemap is not an AI feature. It is how search engines find your pages, and live AI answers are built from search results.

### 5. llms.txt (nice to have, not a ranking factor)

The checker says whether `/llms.txt` exists and marks it as optional. Studies found no correlation between having one and being cited (SE Ranking, about 300,000 domains). In one server-log study, 97% of the files got zero requests (Ahrefs). We have one. It is a courtesy, not a lever.

## How to read the result

- **All green:** assistants can read you. Move on to what actually decides recommendations: third-party mentions, reviews, and your homepage's first screen.
- **A search crawler is blocked:** fix it today. This is the one check that can remove you from an assistant's answers all by itself.
- **A training crawler is blocked:** your choice. Being in model memory helps a known brand win ties. So if you want to be recommended, we suggest allowing it. Just make it a deliberate decision.
- **Homepage returned 403 or a challenge page:** your firewall is answering instead of your site. Allow the named crawlers, then re-check.
- **Little or no text without JavaScript:** ask whoever built your site for server-side rendering or static HTML on the public pages. Until then, make your homepage title and description tags carry the facts.
- **noindex or nosnippet present:** almost always a mistake. Remove it from public pages.
- **No sitemap:** add one and submit it to Google Search Console and Bing Webmaster Tools.
- **No llms.txt:** fine. Add one if you have an hour. Expect nothing from it.

## Do it by hand

The checker calls itself `HeardOfYouBot/1.0`. To see exactly what a specific crawler gets, run the same checks yourself. Open a terminal and replace `example.com` with your site.

**Check robots.txt rules for each crawler:**

```
curl -s https://example.com/robots.txt
```

Read the output. A block under `User-agent: *` applies to every crawler, unless a named section overrides it. Look for `Disallow: /` under any crawler name in the table above. No `Disallow` line for a crawler, or `Allow: /`, means allowed.

**Fetch your homepage as a crawler and count the words:**

```
curl -s -A "OAI-SearchBot" https://example.com/ | sed 's/<[^>]*>/ /g' | wc -w
```

Under a hundred words for a real homepage usually means the content loads with JavaScript. To see what came back, drop the `wc -w` and read it. "Just a moment", "Checking your browser" or a consent form means a firewall or bot challenge is answering instead of your page.

Repeat with `-A "ClaudeBot"`, `-A "PerplexityBot"` and `-A "Googlebot"`. Some firewalls treat each one differently. That is the one thing this page's checker cannot see for you.

**Check for noindex and snippet limits in the HTML:**

```
curl -s https://example.com/ | grep -i -E 'noindex|nosnippet|max-snippet'
```

Empty output is good. Any line showing a `robots` meta tag with those values is a problem on a public page.

**Check for noindex in the headers:**

```
curl -s -I https://example.com/ | grep -i x-robots-tag
```

Empty output is good.

**Check for a sitemap and llms.txt:**

```
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/sitemap.xml
curl -s -o /dev/null -w "%{http_code}\n" https://example.com/llms.txt
```

`200` means the file exists. `404` means it does not. The first matters. The second is optional.

**See what the assistant actually retrieves:** ask Perplexity your buyer question and open its sources. If your page appears, you are readable and retrievable for that question. If not, readability is probably not your problem.

## About the tool

The checker runs on our server and fetches four files from your site: `/robots.txt`, `/`, `/sitemap.xml`, `/llms.txt`. It stores nothing, not the address, not the result. It reads at most one megabyte of your homepage and gives up after eight seconds per file. It will not fetch private or local addresses.

To link to a result, add `?url=yoursite.com` to this page's address. A command-line version is on the list. Its release will go in the [changelog](/changelog).

## What to do next

Readable and still not named? The next question is which pages the assistants read instead of yours. The [free check](/check) answers that: 5 buyer questions, 3 runs, 5 assistants. You get who was named instead and the sources cited, within 2 working days.

<script src="/tools.js" defer></script>

## Questions

**Does the checker store my site or my results?**

No. It fetches robots.txt, your homepage, /sitemap.xml and /llms.txt, builds the report, and keeps nothing. The report lives only in your browser tab.

**Should I block GPTBot but allow OAI-SearchBot?**

That is a legitimate choice. OAI-SearchBot feeds ChatGPT's live answers. GPTBot feeds training. If you sell to people who ask ChatGPT, allow at least the search crawlers. We allow both on our own site, on purpose.

**Does having an llms.txt file count?**

The checker lists it as nice to have. It is not a ranking factor. No correlation with citations across about 300,000 domains, and 97% of files got zero requests in one log study.

**The checker says my homepage returned 403 or a challenge page. What does that mean?**

A firewall or bot protection answered instead of your site. The checker calls itself HeardOfYouBot. If it is refused, the AI crawlers it stands in for often are too. Ask whoever runs the firewall to allow the named crawlers. Then confirm with the curl commands on this page, once per crawler name.

**My site passes all of this and ChatGPT still ignores me. Why?**

Readable is the floor, not the ceiling. The usual cause: no third-party page mentions you. See the guide on why AI doesn't recommend your product.


## Sources

- OpenAI, bots and crawlers documentation (OAI-SearchBot, GPTBot, ChatGPT-User) (https://platform.openai.com/docs/bots)
- Google, AI features and your website (https://developers.google.com/search/docs/appearance/ai-features)
- Google, robots meta tag and X-Robots-Tag (noindex, nosnippet) (https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag)
- RFC 9309, Robots Exclusion Protocol (how robots.txt rules are matched) (https://www.rfc-editor.org/rfc/rfc9309.html)
- Muck Rack, Generative Pulse, October 2026 (blocking Claude's crawler and Claude citations) (https://www.globenewswire.com/news-release/2026/10/28/3390134/0/en/generative-pulse-earned-media-still-drives-more-than-84-of-ai-citations.html)
- SE Ranking, llms.txt across about 300,000 domains (vendor) (https://seranking.com/blog/llms-txt/)
- Ahrefs, llms.txt study: 97% of files got zero requests (vendor) (https://ahrefs.com/blog/llmstxt-study/)