Is Your Website Accidentally Blocking the AI Crawlers That Recommend You?

You have a good website, your business is real, and your service is solid. Yet when a customer asks ChatGPT or Perplexity to recommend a company like yours in Guyana, your name never comes up. Before you assume you need more content or more reviews, check something simpler and far more common: your own website may be turning away the exact automated visitors that feed those AI answers. Most owners who are blocking AI crawlers never chose to. A security plugin, an SEO preset, a hosting default, or a single Cloudflare toggle made the decision for them.

This is the quiet failure behind a lot of AI invisibility. Every other step in getting recommended by AI assumes the crawlers can actually read your site. If they are locked out, nothing else you do matters. This guide names the crawlers that count, shows you how to check whether you are blocking them, and shows you how to fix it. It fits inside the full method for getting recommended by AI search in Guyana, which is where to go once the door is open.

How AI answers actually reach your website

When you ask an AI assistant to recommend a business, it draws on two things: what it learned during training, and what it can look up live at the moment you ask. Both routes depend on automated programs called crawlers (also called bots or user-agents) visiting your pages and reading them. Each major AI company runs its own crawlers, and each one has a name your website can see. If your site tells a crawler by name that it is not welcome, that crawler leaves, and whatever it would have fed into the AI answer never gets collected.

The blocking is almost never malicious or even intentional. Bots also include the bad kind: scrapers that steal content, tools that hammer a server, and spammers. So security products ship with settings that block bots aggressively by default, and they cannot always tell a helpful AI crawler from a harmful scraper. The result is that a setting meant to protect your site shuts out the visitors you most want.

The crawlers that matter and what each one feeds

You do not need to memorize dozens of names. A handful of crawlers cover the AI assistants your customers actually use. Here is who they are and what each one feeds.

• OAI-SearchBot and ChatGPT-User feed ChatGPT. OAI-SearchBot is how OpenAI builds the search index behind ChatGPT's answers, and ChatGPT-User is what fetches a page live when a user's question triggers a lookup.
• PerplexityBot feeds Perplexity, the AI answer engine that cites its sources directly, so being readable to it means your site can appear as a named reference.
• Google-Extended feeds Google's AI features, including AI Overviews and Gemini. It is separate from Google's normal search crawler, so you can be indexed for regular search and still be excluded from Google's AI answers.
• ClaudeBot feeds Claude, Anthropic's assistant, used both on its own and inside other products.
• Applebot-Extended feeds Apple Intelligence, which matters as AI features spread across iPhones in the region.

How to check whether you are blocking them

There are three places a block usually hides. Check them in order, because the first one takes two minutes and catches the majority of cases.

1. Read your robots.txt

Every website can have a small text file at the root that tells crawlers what they may and may not visit. Open a browser and go to your own address followed by /robots.txt, for example yourbusiness.com/robots.txt. It loads as plain text. You are looking for any line that names one of the crawlers above next to the word Disallow. A block looks like a line naming a crawler as the User-agent, followed by "Disallow: /", which means "do not read any of this site."

• Scan for the crawler names: OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-Extended, ClaudeBot, Applebot-Extended.
• For each one you find, look at the Disallow line beneath it. "Disallow: /" is a full block. An empty Disallow, or no mention at all, means the crawler is allowed.
• Watch for a catch-all block. A section that starts with "User-agent: *" and then "Disallow: /" blocks every crawler that is not given its own exception, which can include the AI ones.

If the file names these crawlers and disallows them, you have found your problem, and it is fixable in minutes. If robots.txt looks clean, move to the next two checks, because a block can live in a setting that never touches this file.

2. Check your Cloudflare bot settings

If your site runs behind Cloudflare (a very common setup for speed and security), Cloudflare can block bots before they ever reach your robots.txt. Two settings do this. "Bot Fight Mode" challenges or blocks automated traffic broadly, and can catch AI crawlers in the net. Separately, Cloudflare added a dedicated control for AI scrapers and crawlers, and in some accounts blocking AI bots is turned on by default. Log in to your Cloudflare dashboard, open the Security or Bots section, and see whether Bot Fight Mode is on and whether the AI crawler setting is set to block. If a developer or host set up Cloudflare for you, ask them to confirm these two settings.

3. Check your WordPress security and SEO plugins

On WordPress, which runs a large share of small-business sites, the block often lives inside a plugin. Security plugins commonly have a toggle to block AI bots or scrapers, sometimes switched on by default and sold as a feature. SEO plugins can also write rules into your robots.txt on your behalf, and some now include an option to disallow AI crawlers. Open your security plugin and your SEO plugin, look for any setting mentioning AI, bots, scrapers, or crawlers, and read what it is currently doing. A setting labelled something like "Block AI bots" being switched on is exactly the accidental block we are describing.

How to fix it

The fix matches wherever you found the block. In every case the goal is the same: let the AI crawlers read your site.

1. In robots.txt: remove the Disallow rules aimed at the AI crawlers, or add an explicit allow. For each crawler you want in, a section that names it as the User-agent with an empty Disallow line tells it the whole site is open. If a developer manages the file, send them the crawler names and ask for them to be allowed.
2. In Cloudflare: turn off Bot Fight Mode if it is the cause, or use Cloudflare's controls to allow the specific AI crawlers rather than blocking the category. Cloudflare lets you allow named bots while still blocking the harmful ones, so you keep the protection and open the door.
3. In a plugin: switch off the "block AI bots" or equivalent option in your security plugin, and in your SEO plugin remove any rule that disallows AI crawlers. If the SEO plugin is what generates your robots.txt, edit it there so your change is not overwritten later.

After you change anything, wait a short while and reload yoursite.com/robots.txt to confirm it now reflects what you want. Changes take effect for future crawls, so this is not instant in AI answers, but you have removed the barrier that was stopping the process before it started.

Blocking AI on purpose is a legitimate choice

None of this means every business should let every AI crawler in. Some owners block AI deliberately, and for good reasons. A publisher whose content is the product may not want it absorbed into an AI model for free. A business with proprietary guides, pricing, or research might decide the trade is not worth it. If that is a considered decision, blocking these crawlers is a perfectly reasonable stance, and the settings above are how you keep them out on purpose.

The point is that it should be a decision, not an accident. For most local businesses in Guyana that live on referrals (the salon, the hardware store, the tour operator, the accountant), the calculation runs the other way. You want to be the name an assistant offers when someone nearby asks for a recommendation, and that is worth far more than protecting a services page from being read. For those businesses, being in the AI answers is the whole point, and an accidental block is pure lost opportunity.

Why this is the first thing to check

When a business is completely absent from AI answers, an accidental crawler block is the single most common reason. It is also the cheapest to fix, because it costs nothing but a few minutes of checking and a setting change. Everything else in AI visibility (clear content, consistent business details, reviews, being cited elsewhere) builds on the assumption that the crawlers can reach you. Open the door first. Then the full method for getting recommended by AI has somewhere to work.