GPTBot: OpenAI releases new web crawler
source link: https://searchengineland.com/gptbot-openais-new-web-crawler-430360
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
You can now prevent OpenAI's ChatGPT from accessing your website, or parts of it, using robots.txt.
OpenAI has published information about GPTBot, its new web crawler.
What is GPTBot. GPTBot is OpenAI’s web crawler. OpenAI uses it to crawl the web, consume knowledge for its AI features (e.g., ChatGPT) and provide AI-generated answers to questions (or prompts).
Useragent. GPTBot’s User agent token is “GPTBot”. Its full user-agent string is: “Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)”.
Robots.txt. You can use robots.txt to block GPTBot from accessing your website, or parts of it. To disallow GPTBot to access your site you can add GPTBot to your site’s robots.txt:
User-agent: GPTBot
Disallow: /To allow GPTBot to access only parts of your site, you can add the GPTBot token to your site’s robots.txt like this:
User-agent: GPTBot
Allow: /directory-1/
Disallow: /directory-2/GPTBot documentation. You can read the documentation on GPTBot.
GPTBot IP ranges. OpenAI also published the IP ranges that GPTBot uses. It only lists one, but I suspect they will add more over time.
Why we care. You can disallow GPTBot from crawling your site if you don’t want OpenAI using your content in any way. This is the same protocol you would use to block GoogleBot, BingBot or other web crawlers. These companies are also looking for an alternative to robots.txt for these purposes.
Dig deeper. Should you block ChatGPT’s web browser plugin from accessing your website?
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK