SEO glossary
GPTBot
GPTBot is OpenAI's web crawler, the user agent it uses to collect pages that may be used to train and ground its models.
It obeys robots.txt, and blocking it is two lines: User-agent: GPTBot followed by Disallow: /. Allowing it is the default — no entry means it may crawl. OpenAI also runs separate agents for other jobs, OAI-SearchBot for search indexing and ChatGPT-User for pages fetched during a live conversation, and blocking one does not block the others.
That separation is the decision people get wrong. Blocking GPTBot is often framed as opting out of training, but the same content is what grounds answers, so a full block also makes you less citable inside ChatGPT. If your goal is to be named in AI answers, blocking the crawler that reads you works against it.
Whether to allow it is a business judgement, not a technical one, and reasonable sites go both ways. Publishers whose product is the text itself have a real case for blocking; a company that wants to be recommended does not. Decide deliberately and write down why, because this is the kind of line that gets copied between sites without anyone rereading it.
Common question
- Does blocking GPTBot remove content already used?
- No. Robots.txt affects future crawling only — it cannot reach back into a model that has already been trained. It stops further collection, nothing more.
Check this on your own site
- Site Audit
Crawl your site and group what is broken
- Brand Lookup
How often models mention and cite your brand
Definitions are the easy part
The free plan opens every tool with 500 credits and no card, so you can see these numbers on your own site rather than in an example.
Start free