Google's robots.txt parser is now open source

6 years ago

source link: https://www.tuicool.com/articles/Z3iuErU
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.

For 25 years, the Robots Exclusion Protocol (REP) was only a de-facto standard. This had frustrating implications sometimes. On one hand, for webmasters, it meant uncertainty in corner cases, like when their text editor included BOM characters in their robots.txt files. On the other hand, for crawler and tool developers, it also brought uncertainty; for example, how should they deal with robots.txt files that are hundreds of megabytes large?

Ebe2eiQ.png!web

Today,we announced that we're spearheading the effort to make the REP an internet standard. While this is an important step, it means extra work for developers who parse robots.txt files.

We're here to help: we open sourced the C++ library that our production systems use for parsing and matching rules in robots.txt files. This library has been around for 20 years and it contains pieces of code that were written in the 90's. Since then, the library evolved; we learned a lot about how webmasters write robots.txt files and corner cases that we had to cover for, and added what we learned over the years also to the internet draft when it made sense.

We also included a testing tool in the open source package to help you test a few rules. Once built, the usage is very straightforward:

robots_main <robots.txt content> <user_agent> <url>

If you want to check out the library, head over to our GitHub repository for the robots.txt parser . We'd love to see what you can build using it! If you built something using the library, drop us a comment on Twitter , and if you have comments or questions about the library, find us on GitHub .

Posted by Edu Pereda , Lode Vandevenne , and Gary

, Search Open Sourcing team

Recommend

chinagdg.org 6 years ago
Cache

Google’s robots.txt Parser is Now Open Source

Google’s robots.txt Parser is Now Open Source 2019-07-01adminGoogleDevFeeds

github.blog 4 years ago
Cache

An open source parser for GitHub Actions

February 7, 2019 Engineering An open source parser for GitHub Actions

www.java-source.net 3 years ago
Cache

Open Source Parser Generators in Java

Open Source Parser Generators in JavaANTLRANother Tool for Language Recognition, (formerly PCCTS) is a language tool that provides a framework for co...