Technical checklist to avoid blocking search engines and ai crawlers

The real issue: a site open to humans can be closed to machines

Most teams check a page by opening it in a browser. If it loads, they assume everything is fine. For SEO and ai visibility, that is not enough. A crawler may receive a different response because of user agent, location, IP range, request frequency or a security rule.

The common failure mode is a service page that returns 200 to a normal visitor but 403 or 429 to Google-Extended, gptbot, OAI-SearchBot, perplexitybot or claudebot. In that case the copy is not the first problem: access is.

Robots.txt: crawl control is not index control

Google describes robots.txt as a way to tell crawlers which URLs they may access and to manage server load. It should not be used as a privacy mechanism or as a substitute for noindex. Private content needs real access control.

For ai visibility, the practical decision is to separate general search crawlers, AI search or answer crawlers, and training crawlers. Some may be useful to allow and others may be limited, but the choice should be intentional.

• Check that robots.txt returns 200 and is not serving an old cached version.
• Include the sitemap line with the correct XML sitemap URL.
• Avoid broad disallow rules on folders that contain service pages, article pages, key images or CSS/JS resources.
• Document any explicit AI bot block so marketing, SEO and systems teams know why it exists.

HTTP status codes that change how a site is read

HTTP codes are basic crawlability signals. A 200 delivers content. A 301 can consolidate a migration. A 404 or 410 communicates absence. A 403 means blocked. A 429 means too many requests. A 5xx means the server failed to respond correctly.

The audit must test these codes with several user agents. Browser-only validation misses many problems because the failure often appears only when a request looks like a crawler.

• 200 for public indexable pages.
• 301 for permanent redirects from old URLs.
• 404 or 410 for content that has genuinely been removed.
• 403 only when there is a clear reason to block.
• 429 with care: useful against abuse, risky when applied to legitimate crawlers on public pages.

Cloudflare and WAF: protect without hiding the site

Cloudflare supports custom rules, managed challenges, rate limiting and exceptions. Those tools are useful, but an aggressive setup can make a website technically published and practically invisible to crawlers.

The right answer is rarely to allow every bot everywhere. A better setup protects login paths, forms, internal APIs and abusive parameters while keeping important public pages accessible when the request pattern fits legitimate crawling.

• Separate public pages, forms, admin areas and dynamic endpoints.
• Identify which rule causes each 403 or 429 before changing the WAF.
• Test googlebot, OAI-SearchBot, gptbot, ChatGPT-User, claudebot, perplexitybot and Applebot user agents.
• Prefer precise exceptions over disabling protections globally.

Sitemap, canonical and hreflang: structure signals

A sitemap does not force indexing, but it helps discovery and communicates modified URLs. This matters when content is new, multilingual or not yet strongly linked internally.

Canonical and hreflang complete the structure. Canonical points to the preferred version when duplicates or variants exist. Hreflang connects equivalent Spanish, English and Catalan pages so they do not compete blindly.

• Every indexable URL should have an absolute canonical pointing to itself or its main version.
• Every multilingual group should connect ES, EN and CA equivalents.
• The sitemap should update lastmod when a page is substantially rewritten.
• Sitemap URLs should return 200, not long redirect chains.

AI bots to review separately

Not all AI bots have the same purpose. OpenAI separates crawlers for search, training and user-triggered requests. Anthropic documents different agents for different uses. Perplexity documents its crawler for search results.

If a company wants assistant visibility, blocking everything related to AI can be counterproductive. If it needs to limit content use, allowing everything without policy can also be wrong. The decision should follow the use case.

• OAI-SearchBot and ChatGPT-User for OpenAI search or user-request scenarios.
• GPTBot when the company policy allows or limits training use.
• ClaudeBot and Claude-User according to the desired Anthropic access policy.
• PerplexityBot when the company wants to appear as a source in Perplexity.
• Googlebot for Google Search and Google-Extended as a separate AI-related control.

Operational checklist before publishing an important page

Before a landing page, service page or article is considered finished, it should pass a basic technical checklist. This is not decorative work: it prevents content from existing only for users who already know the URL.

The review must happen in production because real issues often come from CDN, WAF, cache, redirects, SSL or hosting rules rather than the local HTML.

• Final URL returns 200 for browsers and relevant bots.
• No accidental robots.txt block.
• Canonical points to the final URL.
• The URL is in the sitemap and robots.txt links the sitemap.
• If languages exist, hreflang connects all equivalent versions.
• Main content is present in HTML and does not depend only on JavaScript interaction.
• The page has title, description, H1, structured data and coherent internal links.
• Cloudflare does not return a challenge, 403 or 429 to allowed bots.

Turning the audit into decisions

A useful output is not a huge list of errors. It is a decision per URL: index, redirect, remove with 410, keep blocked or improve. Mixing those cases creates noise.

For ai visibility, priority pages are the ones that explain what the company does, who it serves, where it operates and why it is trustworthy. Those pages must be accessible, clear, linked and supported by verifiable signals.