GPTBot, OAI-SearchBot, Bytespider: which does what

GPTBot collects pages that may be used to train OpenAI's models. OAI-SearchBot fetches pages for ChatGPT search, and ChatGPT-User opens a page when someone asks ChatGPT about it. Bytespider is ByteDance's crawler, and we found no ByteDance documentation for it. Because each has its own name in robots.txt, a site can refuse training and still be readable by AI search.

OpenAI states the split on its crawler page:

Each setting is independent of the others – for example, a webmaster can allow OAI-SearchBot in order to appear in search results while disallowing GPTBot to indicate that crawled content should not be used for training OpenAI’s generative AI foundation models.

Three jobs, one file

OpenAI, Anthropic and Perplexity each split their crawlers by job, and the job decides what a block costs you.

• Training. The crawler collects pages that may go into a future model. A block is a training opt-out, kept separate from search.
• Search index. The crawler fetches pages ahead of time so an AI search product can find and link them. A block is a search opt-out for that product.
• User fetch. A fetcher opens one page because a person asked an assistant about it. It does not crawl, and two of those three vendors say robots.txt may not stop theirs.

Two names do none of these. Google-Extended and Applebot-Extended send no requests at all; they are tokens your robots.txt uses to say how pages another crawler fetched may be used.

Crawler by crawler

Each line below quotes the operator's own page, read on 31 October 2026. Tokens and wording change, so check the page before you copy a rule from it.

OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User

• GPTBot is for training: "Disallowing GPTBot indicates a site’s content should not be used in training generative AI foundation models."
• OAI-SearchBot is for ChatGPT search: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links."
• ChatGPT-User fetches during a conversation: "ChatGPT-User is not used for crawling the web in an automatic fashion. Because these actions are initiated by a user, robots.txt rules may not apply."

Anthropic: ClaudeBot, Claude-SearchBot, Claude-User

Anthropic's help page uses the same three jobs under its own names:

• ClaudeBot is for training: "When a site restricts ClaudeBot access, it signals that the site's future materials should be excluded from our AI model training datasets."
• Claude-SearchBot indexes for search. Blocking it "prevents our system from indexing your content for search optimization".
• Claude-User fetches for a person's question. Blocking it "prevents our system from retrieving your content in response to a user query".

Anthropic says its bots "respect “do not crawl” signals by honoring industry standard directives in robots.txt", and asks for the rule on "every subdomain that you wish to opt out from."

Perplexity: PerplexityBot, Perplexity-User

Perplexity's crawler page lists two agents and no training crawler:

• PerplexityBot is search only. It "is designed to surface and link websites in search results on Perplexity. It is not used to crawl content for AI foundation models."
• Perplexity-User fetches for a user's question: "Since a user requested the fetch, this fetcher generally ignores robots.txt rules."

Google-Extended and Applebot-Extended: rules, not crawlers

Google's crawler list says it outright: "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings; the robots.txt user-agent token is used in a control capacity." The token decides whether pages Google already crawled may train Gemini models and be used for grounding (handing a model fresh content while it writes an answer, so the answer rests on it) in Gemini Apps and Vertex AI. It does not take a page out of AI Overviews; the nosnippet guide covers what does.

Apple's token has the same shape. About Applebot says: "Applebot-Extended does not crawl webpages." Applebot fetches for Spotlight, Siri and Safari, and Applebot-Extended decides whether that copy may train Apple's models. "Webpages that disallow Applebot-Extended can still be included in search results."

Common Crawl: CCBot

CCBot crawls for a public dataset, not for one AI company. Common Crawl's CCBot page describes "an open repository of web crawl data that is universally accessible and analyzable by anyone", and its FAQ says CCBot checks robots.txt before it fetches a page. Model builders have used it: OpenAI's 2020 GPT-3 paper took 60% of its training mix from filtered Common Crawl. Blocking CCBot means, in the FAQ's words, "our crawler will stop crawling your website". The FAQ says nothing about crawls that were already published.

ByteDance: Bytespider

We found no page from ByteDance about Bytespider on 31 October 2026: no stated purpose, no user-agent string, nothing on robots.txt. Pages that rank for its name often say it ignores robots.txt. We couldn't trace that to anything ByteDance published or to a study we could read in full, so we don't repeat it. geo.new files Bytespider with the training crawlers, which is our classification, not ByteDance's.

What a robots.txt block does not do

A disallow is a request. The robots standard, RFC 9309, says so in one line: "These rules are not a form of access authorization."

• It may not stop a user fetch. OpenAI says robots.txt "may not apply" to ChatGPT-User and Perplexity says its fetcher "generally ignores" it; Anthropic counts Claude-User among the bots that honour it.
• It does not reach back. Anthropic's wording is "future materials". None of the pages quoted here promises to remove what was collected before the block.
• It is not instant. OpenAI says a change "can take ~24 hours" to reach ChatGPT search; Perplexity says "up to 24 hours".
• It does not prove who is asking. Any client can send a GPTBot user agent. OpenAI, Anthropic, Perplexity and Common Crawl each publish IP ranges to check a request against, and Common Crawl warns it is "aware of crawlers falsely identifying themselves as CCBot".

When a crawler must not get in, that is a firewall or CDN rule, not a robots.txt line.

Block training, stay in AI search

1. 1

   Put the training names in one group

   One User-agent line per name, then a single Disallow: / for all of them. This file refuses the training crawlers and the two control tokens in this post:

   # Training and training controls: no
   User-agent: GPTBot
   User-agent: ClaudeBot
   User-agent: CCBot
   User-agent: Bytespider
   User-agent: Google-Extended
   User-agent: Applebot-Extended
   Disallow: /
   
   # Everyone else, AI search and user fetchers included
   User-agent: *
   Disallow: /admin/

2. 2

   Leave search crawlers to the general group

   OAI-SearchBot, Claude-SearchBot, PerplexityBot and the user fetchers have no group of their own here, so they follow *: the whole site except /admin/.

3. 3

   Check what each crawler matched

   An audit lists every crawler with the rule it matched. We ran this file through geo.new's own robots matcher (rules v23): on the homepage ACC-04 and ACC-05 pass, and ACC-06 and ACC-07 list the six refused tokens without touching the score.

Content-Signal is a second way to say the same thing. Cloudflare's Content Signals Policy adds a line that names a use instead of a crawler, Content-Signal: search=yes, ai-train=no, so it covers bots your file doesn't list. Like every robots.txt line, it states a preference; each operator decides whether to read it.

Where they overlap

One robots.txt is read by all of them, and one crawl can serve more than one job. When GPTBot and OAI-SearchBot are both allowed, OpenAI "may use the results from just one crawl for both use cases". Applebot's crawl feeds Apple's search and can also train Apple's models unless Applebot-Extended is disallowed. Google-Extended applies to pages Google's ordinary crawlers fetched. A visit in your logs doesn't tell you what the copy will be used for; the tokens in your robots.txt do.

Which one you need

flowchart LR
  Q("Keep it out of?") --> T["Training: training group"]
  Q --> S["ChatGPT: OAI-SearchBot"]
  Q --> O["AI Overviews: nosnippet"]
  Q --> U["Pasted links: fetchers"]
  Q --> F["Any crawler: firewall"]

• Keep pages out of model training, stay in AI search. Use the block above: training crawlers and the two control tokens disallowed, everything else left alone.
• Keep a page out of ChatGPT search answers. Disallow OAI-SearchBot. OpenAI says the page can still appear as a navigational link.
• Keep one page out of Google's AI Overviews. That isn't a crawler question. It's nosnippet on that page, the guide linked above.
• Stop assistants opening a page when someone pastes its link. Disallow the user fetchers, knowing two vendors say robots.txt may not stop theirs. A server rule is the reliable route.
• Make sure a crawler cannot get in. Use the firewall or CDN, and check requests against the vendor's IP list.
• Allow everything. Do nothing. With no group for a crawler and no * group, robots.txt leaves it free to fetch, and so does a site with no robots.txt.

How geo.new reads each

An audit reads robots.txt for the exact address you give it and sorts 22 AI crawler tokens into four groups, under rules v23 (31 October 2026). All four rows sit in AI Access and Rendering, and only the first two can move the score:

• ACC-04, AI search index crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot, Applebot, DuckAssistBot), tier 3. The catalog reads "pass none disallowed · warn some disallowed · fail all disallowed; evidence lists each agent". It also feeds the ChatGPT search and Perplexity lines in Platform Readiness.
• ACC-05, User-initiated AI fetchers (ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, MistralAI-User), tier 1: "pass none disallowed · warn any disallowed".
• ACC-06, Grounding and ecosystem controls (Google-Extended, GoogleOther, Applebot-Extended, Amazonbot, FacebookBot): "status per agent; informational".
• ACC-07, Training-only crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, cohere-ai, anthropic-ai, Meta-ExternalAgent): "status per agent; informational, never penalized".

ACC-06 and ACC-07 are tier 0, which our scoring reference defines as "Informational: shown, never scored, never an action". So a training opt-out never lowers the GEO score. That is a product decision of ours: refusing training is the publisher's call, and the report shows it without judging it.

The agent observation A05, Crawler purposes, puts the three OpenAI names side by side. On the sample report, a fixture page whose robots.txt blocks only OAI-SearchBot, it reads:

User-agent: *
Allow: /

User-agent: OAI-SearchBot
Disallow: /

OAI-SearchBot (search): disallowed by robots policy; GPTBot (training): allowed by robots policy; ChatGPT-User (user initiated): allowed by robots policy. Training restrictions do not reduce scores.

That page is the reverse of the block above: out of ChatGPT search answers, still open to training. ACC-04 warns on it, because one of its five search crawlers is disallowed.

ACC-13, Content signals, surfaces the observation A04 inside the same category and reads Content-Signal from robots.txt or a response header. It is tier 0 too. When a page declares ai-input=no, or a signal we can't interpret, geo.new's own AI summary does not run on it; the deterministic checks still do.

GeoNewBot follows the rules it reports. It checks robots.txt before it fetches the page, at every redirect and before each extra file it probes, such as a sitemap. When the file disallows it, the audit stops with "This site’s robots policy blocks this audit." Every crawler row is the robots policy for that URL, not observed reachability: a firewall that turns GPTBot away while robots.txt allows it still reads as allowed.