List of Googlebot IP addresses released
source link: https://searchengineland.com/google-publishes-the-list-of-googlebot-ip-addresses-375935
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
List of Googlebot IP addresses released
Knowing which bot is really Google and which is not can help you decide which rogue bots to block from your site.
Google has published the full list of IP addresses it uses for crawling and accessing your website under the Googlebot user agents. Google said if you do not want to use reverse DNS other those types of methods for verifying Googlebot, you “can identify Googlebot by IP address by matching the crawler’s IP address to the list of Googlebot IP addresses.”
Why do I need this. If you believe someone is crawling your site, pretending to be Googlebot, and you want to verify that blocking this fake Googlebot crawler is not really Google, then you can use either the command line method or automated IP list method.
IP list. You can access the full list of Googlebot IP addresses over here in this JSON file, note that Google may update this list, so you should check the file probably daily. You can also access all other Google crawlers that match the crawler’s IP address against the complete list of Google IP addresses. Again, these IP addresses may change over time, so you should build into your mechanisms a way to update the IP addresses on your end.
Why we care. Often sites can be slowed down and potentially even go offline because of fake bots crawling and spidering the site. You rarely want to block Google from crawling your site because that can lead to indexing and ranking issues in Google Search. So knowing which bot is really Google and which is not, can help you decide which rogue bots to block from your site.
There are also third party services, like Cloudflare and others that help you manage this.
About The Author
Related Topics
Recommend
-
66
-
86
One of my favourite new features in JavaScript is Proxy. As a library author, it’s exciting to think of the possibilities it opens for new APIs. In Pose, I use it to make a simple syntax for defining…
-
76
Googlebot is now using an evergreen Chromium rendering engine to render websites - improving Google Search's JavaScript compatibility even more
-
36
即将更新 Googlebot 用户代理 2019-11-22adminWebMaster
-
22
Categories ProgrammingRate Limiting Googlebot Across Many WebsitesGooglebot is one of the good guys. It will check your rob...
-
10
DuckDuckGo Hits New Record: 100 Million Searches Per DayDuckDuckGo continues its steady growth, reaching 100 million searches per day for the first time.
-
7
SEO Microsoft list of Bingbot IP addresses released Knowing which bot is really from Microsoft Bing and which is not can help you...
-
8
V2EX › Google 请教一下， GoogleBot 抓取的方式跟控制台调试有关吗？ ab · 47 分钟前 · 58 次点...
-
9
What Happens When Google Picks The Wrong Canonical URL? Google may select a different URL to display in search results from the one listed in a canonical ta...
-
5
Introduction to Googlebot spoofing In this article, I'll describe how and why to use Google Chrome (or Chrome Canary) to view a website as Googlebot. We'll set up a web browser specifically for Googlebot browsing. Using a user-agen...
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK