GPTBot and Other AI Crawlers: Allow or Block Them?

GPTBot collects content for OpenAI’s model training, so most stores can block it while allowing the separate crawlers that make products discoverable in AI search. Choose access by the bot’s job. A blanket AI block can stop a shopper’s assistant from reading your product pages as well as stop training crawlers.

What GPTBot Does

OpenAI documents GPTBot as its training crawler and OAI-SearchBot as its search crawler. Blocking GPTBot leaves the separate search permission intact. Blocking OAI-SearchBot excludes your pages from ChatGPT search answers, although navigational links can remain.

ChatGPT-User fetches pages when a person asks ChatGPT to visit them. It does not decide search inclusion, and robots.txt rules may not apply to those user-directed requests. Allowing search access makes your pages eligible for discovery. It does not guarantee that ChatGPT will recommend a product.

Other AI Crawlers to Know

AI crawler names identify different jobs, and some robots.txt tokens control data use without identifying a separate visiting bot.

Token	Purpose	What the control affects
Google-Extended	Controls use of Google-crawled content for Gemini model training and grounding	Gemini training and certain Gemini answers. It does not control Google Search inclusion or rankings.
ClaudeBot	Collects public web content for Anthropic’s model development	Future training collection. Anthropic uses Claude-SearchBot and Claude-User separately for search and user requests.
PerplexityBot	Finds pages for Perplexity search results	Search discovery. Perplexity documents it as separate from foundation-model training.
Applebot-Extended	Controls training use of content collected by Applebot	Apple foundation-model training. This token does not crawl pages or prevent Apple search inclusion.
bingbot	Crawls pages for Microsoft’s search index	Bing discovery. Blocking it can remove access to product information used in search experiences.

Google-Extended is broader than a training-only switch: Google says it also controls grounding in Gemini Apps and relevant Vertex AI services. Google Search has its own crawler and page controls. Apple likewise separates Applebot’s search crawl from Applebot-Extended’s training permission.

The table follows each operator’s crawler documentation. Treat the token as a policy choice, then inspect the actual request identity before creating a firewall exception.

Should Your Store Allow or Block AI Crawlers

For most stores, allow verified search crawlers to read public products and buying guides, and block training-only crawlers when you want to reserve training use. Search access gives assistants information they can cite or use to find your products. Training collection has no direct promise of a visit or sale in return.

Depart from that default when your catalogue has contractual access restrictions or crawling creates a measurable load problem. Reduce request rates or limit particular routes before excluding every search service. Keep customer accounts and private order information behind authentication, regardless of crawler preferences.

Decide separately whether to allow Google-Extended, because blocking it also limits certain Gemini grounding uses. If AI search visibility is your goal, crawler access is only the first step. Generative engine optimisation explains how useful product answers can earn citations.

How to Control AI Crawlers in robots.txt

Use named robots.txt groups to express separate search and training permissions. The following example is for a hypothetical retailer with public product pages. Add it to the existing file while preserving the store’s sitemap and route restrictions.

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

A specific bot group can take precedence over the general User-agent: * group. Repeat any relevant path restrictions inside the specific groups. The example allows public discovery only. It does not replace your existing private-route controls.

OpenAI publishes these request-header strings. The version numbers in its crawler examples can change.

GPTBot:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Copy this matching robots.txt line.

User-agent: GPTBot

OAI-SearchBot:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot

ChatGPT-User:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Use the short token in robots.txt, not the complete browser-style header. A robots.txt request can include an additional marker in the header.

On Shopify, use the published theme’s robots.txt.liquid template. Shopify advises working with a Partner or having code-editing and SEO expertise.

1. Open Online Store in the Shopify admin and select Edit code for the relevant theme.
2. If the template is absent, choose Add a new template, select robots and create it.
3. Add the bot groups while retaining the Liquid that outputs Shopify’s default rules.
4. Save the published theme, then open your store’s /robots.txt and confirm the new groups appear beside the retained defaults.

Shopify warns against replacing the template with a plain-text file because that loses automatic rule updates. Its Shopify Catalog syndication to agentic storefronts is also separate from direct robots.txt access.

On WooCommerce, edit the WordPress site’s robots.txt output. If the host serves a physical file, back it up and edit it through the hosting file manager or SFTP. If WordPress generates the file, use the existing SEO plugin’s robots editor or have a developer use WordPress’s robots_txt filter. Avoid installing a second tool that competes to control the same file.

After saving, fetch /robots.txt without signing in. It must return the intended text successfully. Test a public product URL with a robots parser for OAI-SearchBot and GPTBot: the first must be allowed and the second disallowed. Then inspect real verified requests for firewall blocks. A successful request with a copied user-agent header alone does not prove that a genuine bot can enter.

The llms.txt proposal provides a separate site summary for AI tools. It does not enforce access permissions.

AI Scraping and Price Scraping

AI scraping copies content for another party’s use, while search crawling collects information to answer queries and direct people to sources. The same product page can be visited for either purpose. A bot copying the entire catalogue without referrals or purchases can consume resources while giving your store little back.

Competitors and resellers can automate price scraping to monitor stock or undercut your offers. Look for rapid sequential requests across product pages, repeated catalogue sweeps and requests that collect every variant. Those patterns are investigation signals, not proof of who operates the bot. Search crawlers can also visit many pages.

Compare request volume with verified identity and actual downstream visits. Apply rate limits to excessive traffic and stronger restrictions to repeated evasion. Keep ordinary shoppers’ product comparisons working while you investigate.

Bot Management: Useful Agents vs Abusive Bots

Bot management combines traffic behaviour with identity checks so a store can admit useful automation and restrict abuse. Cloudflare Bot Management scores requests from 1 to 99, with lower scores indicating automation. Granular scores require Enterprise Bot Management. A low score does not make a search crawler malicious, so verified identity and route-specific rules must inform the decision. Cloudflare explains its scores and detection methods.

Cloudflare’s documented defaults for new domains allow Search and block Training and Agent traffic on pages displaying ads. Do not assume every new store blocks every AI crawler. A Training block can also block mixed-purpose search crawlers. Inspect your selected policy, because an edge block takes effect even when robots.txt allows access. Cloudflare documents the AI bot controls.

Bot attacks include card testing at payment forms, credential stuffing at login and stock scalping around limited releases. Use firewall rules and rate limits matched to those routes, with verified shopping agents admitted only for authorised actions. Engage a security specialist for payment abuse or account protection beyond basic firewall configuration.

Bot Verification and AI Agent Identity

Web Bot Auth is a proposal for automated clients to sign HTTP requests so a store can verify the sender cryptographically. The store checks the signature against a registered public key. Time limits and replay checks help reject copied requests. Cloudflare supports Web Bot Auth verification.

Cloudflare documents work with Visa’s Trusted Agent Protocol and Mastercard’s Agent Pay, which build on Web Bot Auth for shopping-agent authentication. Its commerce documentation also names American Express’s intended use of the proposal. These are specific protocol relationships, not evidence that every agent or merchant supports them. Cloudflare describes the payment-network work.

Know your agent checks connect a request to its operator and the permissions it has. At checkout, agent identity must sit beside customer consent and payment authorisation. A verified operator can help a store distinguish an authorised buyer from an impersonator, but a valid signature alone does not approve a charge. Require the purchase to stay within the customer’s instructions and keep the usual fraud controls.

Seeing Which Bots Visit Your Store

Find AI bots in server access logs or your CDN’s traffic dashboard by filtering the user-agent field. Record the requested path and response status, then compare source addresses with the operator’s published ranges. Anyone can copy a bot name into a header, so the name alone is insufficient evidence of identity.

In Cloudflare, inspect AI Crawl Control and security events for allowed or blocked requests. Compare product-page requests with edge responses, because a blocked request may never reach your hosting logs. A 403 suggests denial, a 429 suggests rate limiting and a challenge can prevent an automated client from reading the page.

Some bots ignore robots.txt. Perplexity says its user-directed fetcher generally ignores those rules, and OpenAI says they may not apply to ChatGPT-User. For enforced restrictions, use firewall rules based on verified identity or behaviour, with rate limits where full blocking is unnecessary. Keep robots.txt reachable so cooperative crawlers can read your preferences.

After a policy change, review genuine crawler requests and confirm public product pages remain accessible to the search services you chose. Investigate unexpected blocks before relaxing security across the whole store.