在 Hacker News 上看到的討論，裡面剛好有 Reddit 第一個雇用的工程師，Jeremy Edberg 的留言：「Reddit is taking over Google (businessinsider.com)」。

在 id=40068381 這邊提到了不少東西，首先是把 title 放進 url 裡面的作法：

To this day, my most public contribution to reddit is that I wrote the code to put the title of the post in the URL. That was done specifically for SEO purposes.

這個在 Google Webmasters (現在叫做 Google Search Console) 也針對 Reddit 處理，將速率強制設為 Custom，不讓 Reddit 的人改：

It was pretty much the only SEO optimization we ever did (along with a few DOM changes), because shortly after that, Google basically dedicated engineering effort specifically to crawling reddit. So much so that we lost the "crawl rate" button in our SEO admin page on Google, it was just set to "Custom".

後續還要針對 Google 的抓取在 load balancer 上把流量拆開處理，不然 crawling pattern 與一般使用情境很不一樣，會造成 cache 的效率極度低落：

I had to stand up a fleet of app servers and caches and databases, and change the load balancers so that Google basically had their own infrastructure (although we would shunt all crawlers there). Crawler traffic was very different than regular traffic -- it looked at pages more than two days old, something humans rarely did at the time. It would blow out every cache (memory, database, disk, etc.). So we put them on their own infra to keep them from killing the rest of the site.

這些算是頗有趣的經驗？

To respond on your own website, enter the URL of your response which should contain a link to this post's permalink URL. Your response will then appear (possibly after moderation) on this page. Want to update or remove your response? Update or delete your post and re-enter your post's URL again. (Learn More)